Skip to main content

A new theorem of information theory


Consider a random experiment whose possible outcomes arez 1,z 2,...,z n. Let the prior probabilities be p1 0, ...,pn 0, and let the posterior probabilities bep 1,...,p n. It is shown that, subject to certain prescribed and intuitively reasonable conditions, the expressionI =k ⌆ p i In (p i/p i 0), wherek is a positive constant, is the unique expression for the information contained in a message which alters the probabilities from thep i 0 to thep i.

This is a preview of subscription content, access via your institution.


  1. C.E. Shannon,Bell System Tech. J. 27, 379 and 623 (1948). Reprinted in C. E. Shannon and W. Weaver,The Mathematical Theory of Communication (University of Illinois Press, Urbana, Illinois, 1949),

    Google Scholar 

  2. A. Feinstein,Foundations of Information Theory (McGraw-Hill Book Company, Inc., New York, 1958).

    Google Scholar 

  3. A. I. Khinchin,Mathematical Foundations of Information Theory (Dover Publications, Inc., New York, 1957).

    Google Scholar 

  4. F. M. Reza,An Introduction to Information Theory (McGraw-Hill Book Company, Inc., New York, 1961).

    Google Scholar 

  5. A. Katz,Principles of Statistical Mechanics: The Information Theory Approach (W. H. Freeman and Company, San Francisco, 1967).

    Google Scholar 

  6. S. Kullback,Information Theory and Statistics (John Wiley & Sons, Inc., New York, 1959).

    Google Scholar 

  7. A. Renyi,Wahrscheinlichkeitrechnung (VEB Deutscher Verlag der Wissenschaften, Berlin, 1966).

    Google Scholar 

  8. M. S. Pinsker,Information and Information Stability of Random Variables and Processes, translated and edited by A. Feinstein (Holden-Day, Inc., San Francisco, 1964).

    Google Scholar 

  9. F. Schlögl, inStatistical Mechanics (Proc. of the IUPAP Meeting, Copenhagen, 1966), edited by T. A. Bak (W. A. Benjamin, Inc., New York, 1967); inProc. of the International Conf. on Statistical Mechanics (Kyoto, 1968) (supplement toJ. Phys. Soc. Japan 26, 1969).

    Google Scholar 

  10. A. Hobson,Concepts in Statistical Mechanics (Gordon and Breach, Science Publishers, Inc., 1970).

    Google Scholar 

Download references

Author information

Authors and Affiliations


Rights and permissions

Reprints and Permissions

About this article

Cite this article

Hobson, A. A new theorem of information theory. J Stat Phys 1, 383–391 (1969).

Download citation

  • Received:

  • Issue Date:

  • DOI:

Key words

  • information theory
  • mathematics
  • probability theory
  • quantitative measure of uncertainty
  • communication theory
  • entropy
  • foundations of statistical mechanics