Journal of Statistical Physics

, Volume 60, Issue 5–6, pp 823–837 | Cite as

Mutual information functions versus correlation functions

  • Wentian Li


This paper studies one application of mutual information to symbolic sequences: the mutual information functionM(d). This function is compared with the more frequently used correlation functionΓ(d). An exact relation betweenM(d) andΓ(d) is derived for binary sequences. For sequences with more than two symbols, no such general relation exists; in particular,Γ(d)=0 may or may not lead toM(d)=0. This linear, but not general, independence between symbols separated by a distance is studied for ternary sequences. Also included is the estimation of the finite-size effect on calculating mutual information. Finally, the concept of “symbolic noise” is discussed.

Key words

Mutual information function correlation functions linear and general dependence symbolic noise 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    C. E. Shannon, The mathematical theory of communication,Bell Syst. Tech. J. 27:379–423 (1948).Google Scholar
  2. 2.
    A. Fraser and H. Swinney, Independent coordinates for strange attractors from mutual information,Phys. Rev. A 33:1134–1140 (1986).Google Scholar
  3. 3.
    A. Fraser, Reconstructing attractors from scalar time series: A comparison of singular system and redundancy criteria,Physics D 34:391–404 (1989).Google Scholar
  4. 4.
    C. E. Shannon, Prediction and entropy of printed English,Bell Syst. Tech. J. 1951:50–64.Google Scholar
  5. 5.
    B. Hayes, A progress report on the fine art of turning literature into drivel, Computer Recreations,Sci. Am. 249(5):18–28 (1983).Google Scholar
  6. 6.
    L. Gatlin,Information Theory and the Living System (Columbia University Press, 1972).Google Scholar
  7. 7.
    S. Wolfram, ed.,Theory and Application of Cellular Automata (World Scientific, 1986).Google Scholar
  8. 8.
    G. J. Chaitin, Toward a mathematical definition of “life,” inThe Maximum Entropy Formalism, Levine and Tribus, eds. (MIT Press, 1979).Google Scholar
  9. 9.
    R. Shaw,The Dripping Faucet as a Model Chaotic System (Aerial Press, 1984).Google Scholar
  10. 10.
    P. Grassberger, Towards a quantitative theory of self-organized complexity,Int. J. Theor. Phys. 25:907–938 (1986).Google Scholar
  11. 11.
    S. Karlin and H. Taylor,A Second Course in Stochastic Processes (Academic Press, 1981); S. Karlin,A First Course in Stochastic Processes (Academic Press, 1968).Google Scholar
  12. 12.
    J. E. Hopcroft and J. D. Ullman,Introduction to Automata Theory, Languages, and Computation (Addison-Welsey, 1979).Google Scholar
  13. 13.
    W. Li, Power spectra of regular languages and cellular automata,Complex Syst. 1(1):107–130 (1987).Google Scholar
  14. 14.
    H. Herzel, Complexity of symbolic sequences,Syst. Anal. Model. Simul. 5(5):435–444 (1988).Google Scholar
  15. 15.
    M. Gardner, Mathematical Games: White and brown music, fractal curves and 1/f fluctuations,Sci. Am. 238(4):16–32 (1978).Google Scholar
  16. 16.
    W. Li, Mutual information functions of natural language texts, Santa Fe Institute preprint, SFI-89-008 (1989).Google Scholar
  17. 17.
    V. M. Alekseev and M. V. Yacobson, Symbolic dynamics and hyperbolic dynamical systems,Phys. Rep. 75:287–325 (1981).Google Scholar
  18. 18.
    A. Lindenmayer, Mathematical models for cellular interactions in development I. Filaments with one-sided inputs,J. Theor. Biol. 18:280–299 (1968).Google Scholar
  19. 19.
    W. Li, Spatial 1/f spectra in open dynamical systems,Europhys. Lett. 10(5):395–400 (1989).Google Scholar
  20. 20.
    W. Li, Expansion-modification systems: Another model for 1/f spectra, preprint (1990).Google Scholar

Copyright information

© Plenum Publishing Corporation 1990

Authors and Affiliations

  • Wentian Li
    • 1
    • 2
  1. 1.Center for Complex Systems Research, Physics Department, Beckman InstituteUniversity of IllinoisUrbana
  2. 2.Department of PhysicsColumbia UniversityNew York

Personalised recommendations