Non-sufficient Memories That Are Sufficient for Prediction

  • Wolfgang Löhr
  • Nihat Ay
Part of the Lecture Notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering book series (LNICST, volume 4)


The causal states of computational mechanics define the minimal sufficient (prescient) memory for a given stationary stochastic process. They induce the ε-machine which is a hidden Markov model (HMM) generating the process. The ε-machine is, however, not the minimal generative HMM and minimal internal state entropy of a generative HMM is a tighter upper bound for excess entropy than provided by statistical complexity. We propose a notion of prediction that does not require sufficiency. The corresponding models can be substantially smaller than the ε-machine and are closely related to generative HMMs.


hidden Markov models HMM computational mechanics causal states ε-machine prediction 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Crutchfield, J.P., Young, K.: Inferring statistical complexity. Phys. Rev. Let. 63, 105–108 (1989)MathSciNetCrossRefGoogle Scholar
  2. 2.
    Shalizi, C.R., Crutchfield, J.P.: Computational mechanics: Pattern and prediction, structure and simplicity. Journal of Statistical Physics 104, 817–879 (2001)MathSciNetCrossRefzbMATHGoogle Scholar
  3. 3.
    Löhr, W., Ay, N.: On the generative nature of prediction. Accepted for publication in Advances in Complex Systems (preprint, 2008),
  4. 4.
    Still, S., Crutchfield, J.P.: Optimal causal inference. Informal publication (2007),
  5. 5.
    Grassberger, P.: Toward a quantitative theory of self-generated complexity. Int. J. Theor. Phys. 25, 907–938 (1986)MathSciNetCrossRefzbMATHGoogle Scholar
  6. 6.
    Bialek, W., Nemenman, I., Tishby, N.: Predictability, complexity, and learning. Neural Computation 13, 2409–2463 (2001)CrossRefzbMATHGoogle Scholar
  7. 7.
    Heller, A.: On stochastic processes derived from Markov chains. Annals of Mathematical Statistics 36, 1286–1291 (1965)MathSciNetCrossRefzbMATHGoogle Scholar
  8. 8.
    Crutchfield, J.P.: The calculi of emergence: Computation, dynamics and induction. Physica D 75, 11–54 (1994)CrossRefzbMATHGoogle Scholar

Copyright information

© ICST Institute for Computer Science, Social Informatics and Telecommunications Engineering 2009

Authors and Affiliations

  • Wolfgang Löhr
    • 1
  • Nihat Ay
    • 1
    • 2
  1. 1.Max Planck Institute for Mathematics in the SciencesLeipzigGermany
  2. 2.Santa Fe InstituteNew MexicoUSA

Personalised recommendations