In all but special circumstances, measurements of time-dependent processes reflect internal structures and correlations only indirectly. Building predictive models of such hidden information sources requires discovering, in some way, the internal states and mechanisms. Unfortunately, there are often many possible models that are observationally equivalent. Here we show that the situation is not as arbitrary as one would think. We show that generators of hidden stochastic processes can be reduced to a minimal form and compare this reduced representation to that provided by computational mechanics – the ε-machine. On the way to developing deeper, measure-theoretic foundations for the latter, we introduce a new two-step reduction process. The first step (internal-event reduction) produces the smallest observationally equivalent σ-algebra and the second (internal-state reduction) removes σ-algebra components that are redundant for optimal prediction. For several classes of stochastic dynamical systems these reductions produce representations that are equivalent to ε-machines.
Similar content being viewed by others
References
J. P. Crutchfield K. Young (1989) ArticleTitleInferring statistical complexity Phys. Rev. Lett. 63 105–108 Occurrence Handle10.1103/PhysRevLett.63.105 Occurrence Handle10040781
J. P. Crutchfield C. R. Shalizi (1999) ArticleTitleThermodynamic depth of causal states: objective complexity via minimal representations Phys. Rev. E 59 IssueID1 275–283 Occurrence Handle10.1103/PhysRevE.59.275
C. R. Shalizi J. P. Crutchfield (2001) ArticleTitleComputational mechanics: Pattern and prediction, structure and simplicity J. Stat. Phys. 104 817–879 Occurrence Handle10.1023/A:1010388907793
James E. Hansen (1993) Computational Mechanics of Cellular Automata University of California Berkeley
James P. Crutchfield and Karl Young, Computation at the onset of chaos, in W. H. Zurek, ed. Complexity, Entropy, and the Physics of Information, volume VIII of Santa Fe Institute Studies in the Sciences of Complexity (Addison-Wesley, 1990), pp. 223–269.
David P Feldman (1998) Computational Mechanics of Classical Spin Systems University of California Davis
James P. Crutchfield David P. Feldman (1997) ArticleTitleStatistical complexity of simple one-dimensional spin systems Phys. Rev. E 55 R1239–R1242 Occurrence Handle10.1103/PhysRevE.55.R1239
W. M. Gonçalves R. D. Pinto J. C. Sartorelli M. J. Oliveira Particlede (1998) ArticleTitleInferring statistical complexity in the dripping faucet experiment Physica A 257 385–389
A. J. Palmer C. W. Fairall W. A. Brewer (2000) ArticleTitleComplexity in the atmosphere IEEE Trans. Geosci. Remote Sens 38 2056–2063 Occurrence Handle10.1109/36.851786
Richard W. Clarke, Mervyn P. Freeman, and Nicholas W. Watkins, The application of computational mechanics to the analysis of geomagnetic data, Phys. Rev. E 67:016203 (2003)
Varn Dowman Parks (2001) Language Extraction from ZnS University of Tennessee Knoxville
Dowman P. Varn Geoffrey S. Canright James P. Crutchfield (2002) ArticleTitleDiscovering planar disorder in close-packed structures from x-ray diffraction: Beyond the fault model Phys. Rev. B. 66 IssueID17 174110 Occurrence Handle10.1103/PhysRevB.66.174110
Dowman P. Varn, Geoffrey S. Canright, and James P. Crutchfield, Discovering planar disorder in close-packed structures from x-ray diffraction: Beyond the fault model, Phys. Rev. B. 66(17):174110 (2002)
D. Nerukh G. Karvounis R. C. Glen (2002) ArticleTitleComplexity of classical dynamics of molecular systems. II. Finite statistical complexity of water-Na+ system J. Chem. Phys 117 9618–9622 Occurrence Handle10.1063/1.1518011
D. Blackwell L. Koopmans (1957) ArticleTitleOn the identifiability problem for functions of Markov chains Ann. Math. Statist 28 1011
L. R. Rabiner (1989) ArticleTitleA tutorial on hidden Markov models and selected applications IEEE Proc 77 257 Occurrence Handle10.1109/5.18626
R. J. Elliot L. Aggoun J. B. Moore (1995) Hidden Markov Models: Estimation and Control Volume 29 of Applications of Mathematics Springer New York
C. E. Shannon W. Weaver (1962) The Mathematical Theory of Communication University of Illinois Press Champaign-Urbana
H. Ito S.-I. Amari K. Kobayashi (1962) ArticleTitleIdentifiability of hidden Markov information sources and their minimum degrees of freedom IEEE Info. Th 38 324 Occurrence Handle10.1109/18.119690
J. P. Crutchfield (1994) ArticleTitleThe calculi of emergence: computation, dynamics, and induction Physica D 75 11–54
D. R. Upper, Theory and Algorithms for Hidden Markov Models and Generalized Hidden Markov Models, PhD thesis (University of California, Berkeley, 1997). Published by University Microfilms Intl, Ann Arbor, Michigan.
C. Glymour G. F. Cooper (Eds) (1999) Computation, Causation, and Discovery AAAI Press Menlo Park, California
M. I. Jordan (Eds) (1999) Learning in Graphical Models MIT Press Cambridge, Massachusetts
M. Casdagli S. Eubank (Eds) (1992) Nonlinear Modeling SFI Studies in the Sciences of Complexity Addison-Wesley Reading, Massachusetts
J. E. Hopcroft J. D. Ullman (1969) Introduction to Automata Theory, Languages, and Computation Addison-Wesley Reading
A. Paz (1971) Introduction to Probabilistic Automata Academic Press New York
H. Bauer (1972) Probability Theory and Elements of Measure Theory, International Series in Decision Processes Holt, Reinhardt and Winston, Inc New York
O. Penrose (1970) Foundations of Statistical Mechanics; A Deductive Treatment Pergamon Press Oxford
T. M. Cover J. A. Thomas (1991) Elements of Information Theory Wiley-Interscience New York
J. P Crutchfield (1992) Semantics and thermodynamics M. Casdagli S. Eubank (Eds) Nonlinear Modeling and Forecasting, volume XII of Santa Fe Institute Studies in the Sciences of Complexity Addison-Wesley Reading, Massachusetts 317–359
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Ay, N., Crutchfield, J.P. Reductions of Hidden Information Sources. J Stat Phys 120, 659–684 (2005). https://doi.org/10.1007/s10955-005-6797-4
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10955-005-6797-4