On Entropy, Information Inequalities, and Groups

  • Raymond W. Yeung
Part of the The Springer International Series in Engineering and Computer Science book series (SECS, volume 712)


There has been significant progress in the study of entropy functions and information inequalities in the past 10 years. The set-theoretic structure of Shannon’s information measures has been established, and machineproving of most information inequalities known to date (Shannon-type inequalities) has become possible. Most importantly, the recent discovery of a few so-called non-Shannon-type inequalities reveals the existence of information inequalities which cannot be proved by techniques known during the first 50 years of information theory. In this expository paper, the essence of this fundamental subject is explained, a number of applications of the results are given, and their implications in information theory, probability theory, and group theory are discussed.


Mutual Information Conditional Independence Markov Random Field Network Security Entropy Function 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. [1]
    C. E. Shannon, “Communication theory of Secrecy systems,” Bell System Technical Journal, vol. 28, pp. 656–715, Oct 1949.MathSciNetMATHGoogle Scholar
  2. [2]
    F. M. Reza, An Introduction to Information Theory, MaGraw-Hill, 1961.Google Scholar
  3. [3]
    G.-D. Hu, “On the amount of Information,” Teor. Veroyatnost. i Primenen. vol. 4, pp. 447–455, 1962 (in Russian).Google Scholar
  4. [4]
    N. M. Abramson, Information Theory and Coding, MaGraw-Hill, 1963.Google Scholar
  5. [5]
    T. S. Han, “Linear dependence structure of the entropy space,” Inform. form. Contr, vol. 29, pp. 337–368, 1975.MATHCrossRefGoogle Scholar
  6. [6]
    T. S. Han, “Nonnegative entropy measures of multivariate symmetric correlations,” Inform. Contr., vol. 36, pp. 133–156, 1978.MATHCrossRefGoogle Scholar
  7. [7]
    T. S. Han, “A uniqueness of Shannon’s information distance and related nonnegativity problems,” Journal of Combinatorics, Information System Sciences, vol. 6, no. 4, pp. 320–331, 1981.MathSciNetMATHGoogle Scholar
  8. [8]
    I. Csiszár and J. Körner, “Information Theory: Coding Theorem for Discrete Memoryless Systems,” New York: Academic Press, and Budapest: Akademiai Kiado, 1981.MATHGoogle Scholar
  9. [9]
    A. Papoulis, Probability, Random Variables and Stochastic Processes, 2nd ed., McGraw-Hill, 1984.MATHGoogle Scholar
  10. [10]
    N. Pippenger, “What are the laws of information theory?” 1986 Special Problems on Communication and Computation Conference, Palo Alto, California, Sept. 3–5, 1986.Google Scholar
  11. [11]
    T. M. Cover and J. A. Thomas, Elements of Information Theory. New York: Wiley, 1991.MATHCrossRefGoogle Scholar
  12. [12]
    R. W. Yeung, “A new outlook on Shannon’s information measures,” IEEE Trans. Inform. Theory, vol. 37, pp. 466–474, May 1991.MathSciNetMATHCrossRefGoogle Scholar
  13. [13]
    T. Kawabata and R. W. Yeung, “The structure of the I-Measure of a Markov chain,” IEEE Trans. Inform. Theory, vol. 38, pp. 1146–1149, 1992.MathSciNetMATHCrossRefGoogle Scholar
  14. [14]
    R. M. Capocelli, A. De Santis, L. Gargano and U. Vaccaro, “On the size of shares for secret sharing schemes,” J. Cryptography, vol. 6, pp. 157–167, 1993.MATHGoogle Scholar
  15. [15]
    D. R. Stinson, “New general lower bounds on the information rate of secret sharing schemes,” Lecture Notes in Comp. Sci. 740, pp 168–182, 1993.MathSciNetCrossRefGoogle Scholar
  16. [16]
    F. Matúš, “Probabilistic conditional independence structures and matroid theory: Background,” Int. J. General Systems, vol. 22, pp. 185–196, 1994.MATHGoogle Scholar
  17. [17]
    F. Matúš and M. Studen, “Conditional independences among four random variables I,” Combinatorics, Probability and Computing, vol. 4, no. 3, pp. 267–278, 1995.Google Scholar
  18. [18]
    F. Matúš, “Conditional independences among four random variables II,” Combinatorics, Probability Computing, vol. 4, pp. 407–417, 1995.MATHCrossRefGoogle Scholar
  19. [19]
    R. W. Yeung, “Multilevel diversity coding with distortion,” IEEE Trans. Inform. Theory, vol. IT-41, pp. 412–422, Mar 1995.CrossRefGoogle Scholar
  20. [20]
    R. W. Yeung, “A framework for linear information inequalities,” IEEE Trans. Inform. Theory, vol. 43, pp. 1924–1934, Nov 1997.MathSciNetMATHCrossRefGoogle Scholar
  21. [21]
    Z. Zhang and R. W. Yeung, “A non-Shannon-type conditional information inequality,” IEEE Trans. Inform. Theory, vol. 43, pp. 1982–1986, Nov 1997.MathSciNetMATHCrossRefGoogle Scholar
  22. [22]
    Z. Zhang and R. W. Yeung, “On Characterization of entropy function via information inequalities,” IEEE Trans. Inform. Theory, vol. 44, pp. 1440–1452, Jul 1998.MathSciNetMATHCrossRefGoogle Scholar
  23. [23]
    F. Matúš, “Conditional independences among four random variables III: Final conclusion,” Combinatorics, Probability Computing, vol. 8, pp. 269–276, 1999.MATHCrossRefGoogle Scholar
  24. [24]
    R. W. Yeung and Z. Zhang, “Distributed source coding for satellite communications,” IEEE Trans. Inform. Theory, vol. 45, pp. 1111–1120, May 1999.MathSciNetMATHCrossRefGoogle Scholar
  25. [25]
    R. W. Yeung and Z. Zhang, “A class of non-Shannon-type information inequalities and their applications,” Communications in Information and Systems, vol. 1, no. 1, pp. 87–100, 2001 (͂is).MathSciNetMATHGoogle Scholar
  26. [26]
    T. H. Chan and R. W. Yeung, “On a relation between information inequalities and group theory,” to appear in IEEE Trans. Inform. Theory. Google Scholar
  27. [27]
    R. W. Yeung, A First Course in Information Theory, Kluwer Academic/Plenum Publishers, 2002.CrossRefGoogle Scholar
  28. [28]
    F.-W.Fu and R. W. Yeung, “On the rate-distortion region for multiple descriptions,” to appear in IEEE Trans. Inform. Theory. Google Scholar
  29. [29]
    R. W. Yeung, T. T. Lee and Z. Ye, “An information-theoretic characterization of Markov random fields and its applications,” to appear in IEEE Trans. Inform. Theory. Google Scholar
  30. [30]
    I. Sason, “Identification of new enormous classes of non-Shannon type constrained information inequalities and related inequalities for finite groups,” submitted to 2002 IEEE International Symposium on Information Theory.Google Scholar
  31. [31]
    K. Makarychev, Y. Makarychev, A. Romashchenko, and N. Vereshchagin, “A new class of non-Shannon-type inequalities for entropies,” submitted to Communications in Information and Systems. Google Scholar

Copyright information

© Springer Science+Business Media New York 2003

Authors and Affiliations

  • Raymond W. Yeung
    • 1
  1. 1.Department of Information EngineeringThe Chinese University of Hong KongN. T., Hong KongChina

Personalised recommendations