Abstract
This is the first of several papers dealing with the application of statistical thermodynamic methodology to the solution of coding and communication theory problems. Emphasis is placed on the various “ensemble techniques” of statistical mechanics, the words or “samples” of a message taking the place of molecules in the prototype physical system. Analogs of temperature, internal energy, pressure, chemical potential, volume, entropy, etc., are developed. The isomorphism with thermodynamics is complete and these quantities transform (for example, by partial differentiation) in exactly the same way as the prototype physical quantities. The methods are nicely applicable to coding cases involving sources with memory, in which case, correlation can be discussed in terms of analog “coupling energies” between signals or words so that the store of “many-body-problem” techniques can be used. In addition, the manipulative freedom stemming from the possibility of choosing from a multiplicity of ensembles constrained by “intensive” parameters proves a distinct advantage. A concrete example dealing with the choice of a compact code for a nonextended source with memory is presented. The compact code is derived, and some discussion is given concerning the breadth of its power spectrum. In a following paper, its autocorrelation function within the framework of “pulse code modulation” is derived and transformed by Wiener theory so that the power spectrum is directly exhibited (along with the spectra for several other cases).
Similar content being viewed by others
References
H. Reiss,J. Stat. Phys. 1:107 (1969).
C. E. Shannon,Proc. IRE 37:10 (1949).
T. L. Hill,Introduction to Statistical Thermodynamics (Addison-Wesley, New York, 1960), Chapter 1.
D. Sakrison,Communication Theory: Transmission of Wave Forms and Digital Information (John Wiley and Sons, New York, 1968), p. 59.
A. M. Rosie,Information and Communication Theory (Blackie and Son, London, 1966), p. 148.
R. G. Gallager,Information Theory and Reliable Communication (John Wiley and Sons, New York, 1968), Chapter 6.
N. Abramson,Information Theory and Coding (McGraw-Hill, New York, 1963), p. 77.
A. M. Rosie,Information and Communication Theory (Blackie and Son, London 1966), p. 81.
Ibid., p. 44.
Ibid., p. 41.
N. Abramson,Information and Communication Theory (McGraw-Hill, New York, 1963), p. 29.
R. H. Fowler,Statistical Mechanics (Cambridge, New York, 1929), p. 27.
R. G. Gallager,Information Theory and Reliable Communication (John Wiley and Sons, New York, 1968), Chapter 2.
T. L. Hill,Introduction to Statistical Mechanics (Addison-Wesley, Reading, Massachussets, 1960), p. 478.
T. L. Hill,Introduction to Statistical Thermodynamics (Addison-Wesley, Reading, Massachussets, 1960), p. 6.
N. Abramson,Information Theory and Coding (McGraw-Hill, New York, 1963), Chapter 2.
H. Reiss,Methods of Thermodynamics (Blaisdell, New York, 1965), p. 79.
N. Abramson,Information Theory and Coding (McGraw-Hill, New York, 1963), p. 46.
Ibid., p. 59.
Ibid., p. 68.
H. Reiss,Methods of Thermodynamics (Blaisdell, New York, 1965), p. 113.
T. L. Hill,Introduction to Statistical Thermodynamics (Addison-Wesley, Reading, Massachusetts, 1960).
Ibid., p. 25.
Ibid. Chapter 14.
N. Davidson,Statistical Mechanics (McGraw-Hill, New York, 1962), pp. 378–393.
Author information
Authors and Affiliations
Additional information
Research supported under AFOSR Grant No. 70-1877. The present work is contribution No. 2643 of the Department of Chemistry, University of California-Los Angeles.
Rights and permissions
About this article
Cite this article
Reiss, H., Huang, C. Statistical thermodynamic formalism in the solution of information theory problems. J Stat Phys 3, 191–210 (1971). https://doi.org/10.1007/BF01019850
Received:
Issue Date:
DOI: https://doi.org/10.1007/BF01019850