On Entropy, Information Inequalities, and Groups
There has been significant progress in the study of entropy functions and information inequalities in the past 10 years. The set-theoretic structure of Shannon’s information measures has been established, and machineproving of most information inequalities known to date (Shannon-type inequalities) has become possible. Most importantly, the recent discovery of a few so-called non-Shannon-type inequalities reveals the existence of information inequalities which cannot be proved by techniques known during the first 50 years of information theory. In this expository paper, the essence of this fundamental subject is explained, a number of applications of the results are given, and their implications in information theory, probability theory, and group theory are discussed.
KeywordsMutual Information Conditional Independence Markov Random Field Network Security Entropy Function
Unable to display preview. Download preview PDF.
- F. M. Reza, An Introduction to Information Theory, MaGraw-Hill, 1961.Google Scholar
- G.-D. Hu, “On the amount of Information,” Teor. Veroyatnost. i Primenen. vol. 4, pp. 447–455, 1962 (in Russian).Google Scholar
- N. M. Abramson, Information Theory and Coding, MaGraw-Hill, 1963.Google Scholar
- N. Pippenger, “What are the laws of information theory?” 1986 Special Problems on Communication and Computation Conference, Palo Alto, California, Sept. 3–5, 1986.Google Scholar
- F. Matúš and M. Studen, “Conditional independences among four random variables I,” Combinatorics, Probability and Computing, vol. 4, no. 3, pp. 267–278, 1995.Google Scholar
- T. H. Chan and R. W. Yeung, “On a relation between information inequalities and group theory,” to appear in IEEE Trans. Inform. Theory. Google Scholar
- F.-W.Fu and R. W. Yeung, “On the rate-distortion region for multiple descriptions,” to appear in IEEE Trans. Inform. Theory. Google Scholar
- R. W. Yeung, T. T. Lee and Z. Ye, “An information-theoretic characterization of Markov random fields and its applications,” to appear in IEEE Trans. Inform. Theory. Google Scholar
- I. Sason, “Identification of new enormous classes of non-Shannon type constrained information inequalities and related inequalities for finite groups,” submitted to 2002 IEEE International Symposium on Information Theory.Google Scholar
- K. Makarychev, Y. Makarychev, A. Romashchenko, and N. Vereshchagin, “A new class of non-Shannon-type inequalities for entropies,” submitted to Communications in Information and Systems. Google Scholar