Thermodynamic Gibbs Formalism and Information Theory
Links between Information Theory and Thermodynamics are well known. The concept of entropy, introduced by C. Shannon in 1948 in his groundbreaking work which gave birth to Information Theory, originates in Statistical Mechanics. In the past 60 years multiple links have been found, which led to new results. Information Theory has been incredibly successful in utilization of probabilistic methods in problems like data compression, prediction, classification, and coding of information sources. Most approaches and algorithms can be classified as causal, or omni-directional: the data are processed in a directed sequential fashion; distribution of the present with respect to the past is used for prediction or classification purposes. However, recently some novel approaches have been proposed in Information Theory. It turns out that the non-causal (bi-directional) approaches, i.e., when the influences of the past as well as of their future are taken into account, lead to very interesting and often superior solutions in problems like denoising and classification. The theory of Gibbs states in Statistical Mechanics—the so-called Gibbs formalism—provides the right framework for treatment of stochastic processes in a non-causal way. We will discuss specific information-theoretic algorithms based on the Gibbs formalism.
KeywordsInformation theory Gibbs states Thermodynamic formalism
- 3.Fernández, F., Viola, A., Weinberger, M.J.: Efficient algorithms for constructing optimal bi-directional context sets. In: IEEE Data Compression Conference, pp. 179–188 (2010)Google Scholar
- 4.Fernández, R.: Gibbsianness and non-Gibbsianness in lattice random fields. Math. Stat. Phys. Session LXXXIII, 731–799 (2006) (Elsevier)Google Scholar
- 10.Ordentlich, E., Weinberger, M.J., Weissman, T.: Multi-directional context sets with applications to universal denoising and compression, In: Proceedings of International Symposium on Information Theory, 2005. ISIT 2005, pp. 1270–1274 (2005)Google Scholar
- 12.Verbitskiy, E.: Thermodynamics of hidden Markov processes. In: Entropy of Hidden Markov Processes and Connections to Dynamical Systems. London Mathematical Society, Lecture note series, vol. 385, pp. 258–272. Cambridge University Press, Cambridge (2011)Google Scholar