Journal of Statistical Physics

, Volume 3, Issue 2, pp 191–210 | Cite as

Statistical thermodynamic formalism in the solution of information theory problems

  • H. Reiss
  • C. Huang
Articles

Abstract

This is the first of several papers dealing with the application of statistical thermodynamic methodology to the solution of coding and communication theory problems. Emphasis is placed on the various “ensemble techniques” of statistical mechanics, the words or “samples” of a message taking the place of molecules in the prototype physical system. Analogs of temperature, internal energy, pressure, chemical potential, volume, entropy, etc., are developed. The isomorphism with thermodynamics is complete and these quantities transform (for example, by partial differentiation) in exactly the same way as the prototype physical quantities. The methods are nicely applicable to coding cases involving sources with memory, in which case, correlation can be discussed in terms of analog “coupling energies” between signals or words so that the store of “many-body-problem” techniques can be used. In addition, the manipulative freedom stemming from the possibility of choosing from a multiplicity of ensembles constrained by “intensive” parameters proves a distinct advantage. A concrete example dealing with the choice of a compact code for a nonextended source with memory is presented. The compact code is derived, and some discussion is given concerning the breadth of its power spectrum. In a following paper, its autocorrelation function within the framework of “pulse code modulation” is derived and transformed by Wiener theory so that the power spectrum is directly exhibited (along with the spectra for several other cases).

Key words

Information theory coding statistical mechanics ensembles thermodynamics 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    H. Reiss,J. Stat. Phys. 1:107 (1969).Google Scholar
  2. 2.
    C. E. Shannon,Proc. IRE 37:10 (1949).Google Scholar
  3. 3.
    T. L. Hill,Introduction to Statistical Thermodynamics (Addison-Wesley, New York, 1960), Chapter 1.Google Scholar
  4. 4.
    D. Sakrison,Communication Theory: Transmission of Wave Forms and Digital Information (John Wiley and Sons, New York, 1968), p. 59.Google Scholar
  5. 5.
    A. M. Rosie,Information and Communication Theory (Blackie and Son, London, 1966), p. 148.Google Scholar
  6. 6.
    R. G. Gallager,Information Theory and Reliable Communication (John Wiley and Sons, New York, 1968), Chapter 6.Google Scholar
  7. 7.
    N. Abramson,Information Theory and Coding (McGraw-Hill, New York, 1963), p. 77.Google Scholar
  8. 8.
    A. M. Rosie,Information and Communication Theory (Blackie and Son, London 1966), p. 81.Google Scholar
  9. 9.
    Ibid., p. 44.Google Scholar
  10. 10.
    Ibid., p. 41.Google Scholar
  11. 11.
    N. Abramson,Information and Communication Theory (McGraw-Hill, New York, 1963), p. 29.Google Scholar
  12. 12.
    R. H. Fowler,Statistical Mechanics (Cambridge, New York, 1929), p. 27.Google Scholar
  13. 13.
    R. G. Gallager,Information Theory and Reliable Communication (John Wiley and Sons, New York, 1968), Chapter 2.Google Scholar
  14. 14.
    T. L. Hill,Introduction to Statistical Mechanics (Addison-Wesley, Reading, Massachussets, 1960), p. 478.Google Scholar
  15. 15.
    T. L. Hill,Introduction to Statistical Thermodynamics (Addison-Wesley, Reading, Massachussets, 1960), p. 6.Google Scholar
  16. 16.
    N. Abramson,Information Theory and Coding (McGraw-Hill, New York, 1963), Chapter 2.Google Scholar
  17. 17.
    H. Reiss,Methods of Thermodynamics (Blaisdell, New York, 1965), p. 79.Google Scholar
  18. 18.
    N. Abramson,Information Theory and Coding (McGraw-Hill, New York, 1963), p. 46.Google Scholar
  19. 19.
    Ibid., p. 59.Google Scholar
  20. 20.
    Ibid., p. 68.Google Scholar
  21. 21a.
    H. Reiss,Methods of Thermodynamics (Blaisdell, New York, 1965), p. 113.Google Scholar
  22. 21b.
    T. L. Hill,Introduction to Statistical Thermodynamics (Addison-Wesley, Reading, Massachusetts, 1960).Google Scholar
  23. 22.
    Ibid., p. 25.Google Scholar
  24. 23.
    Ibid. Chapter 14.Google Scholar
  25. 24.
    N. Davidson,Statistical Mechanics (McGraw-Hill, New York, 1962), pp. 378–393.Google Scholar

Copyright information

© Plenum Publishing Corporation 1971

Authors and Affiliations

  • H. Reiss
    • 1
  • C. Huang
    • 1
  1. 1.Department of ChemistryUniversity of CaliforniaLos Angeles

Personalised recommendations