Skip to main content
SpringerLink
Log in

Informational and Causal Architecture of Continuous-time Renewal Processes

  • Published:
Journal of Statistical Physics Aims and scope Submit manuscript

Abstract

We introduce the minimal maximally predictive models (\(\epsilon \text{-machines }\)) of processes generated by certain hidden semi-Markov models. Their causal states are either discrete, mixed, or continuous random variables and causal-state transitions are described by partial differential equations. As an application, we present a complete analysis of the \(\epsilon \text{-machines }\) of continuous-time renewal processes. This leads to closed-form expressions for their entropy rate, statistical complexity, excess entropy, and differential information anatomy rates.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

References

  1. Shalizi, C.R., Crutchfield, J.P.: Computational mechanics: Pattern and prediction structure and simplicity. J. Stat. Phys. 104, 817–879 (2001)

    Article  MathSciNet  MATH  Google Scholar 

  2. Lohr, W.: Properties of the statistical complexity functional and partially deterministic HMMs. Entropy 11(3), 385–401 (2009)

    Article  ADS  MathSciNet  MATH  Google Scholar 

  3. Crutchfield, J.P., Riechers, P., Ellison, C.J.: Exact complexity: Spectral decomposition of intrinsic computation. Phys. Lett. A 380(9–10), 998–1002 (2016)

    Article  ADS  Google Scholar 

  4. Kelly, D., Dillingham, M., Hudson, A., Wiesner, K.: A new method for inferring hidden Markov models from noisy time sequences. PLoS ONE 7(1), e29703 (2012)

    Article  ADS  Google Scholar 

  5. Marzen, S., Crutchfield, J.P.: Information anatomy of stochastic equilibria. Entropy 16, 4713–4748 (2014)

    Article  ADS  MathSciNet  MATH  Google Scholar 

  6. Riechers, P.M., Crutchfield, J.P.: Beyond the spectral theorem: Decomposing arbitrary functions of nondiagonalizable operators. arXiv:1607.06526

  7. James, R.G., Burke, K., Crutchfield, J.P.: Chaos forgets and remembers: Measuring information creation, destruction, and storage. Phys. Lett. A 378, 2124–2127 (2014)

    Article  ADS  Google Scholar 

  8. Crutchfield, J.P., Feldman, D.P.: Regularities unseen, randomness observed: Levels of entropy convergence. CHAOS 13(1), 25–54 (2003)

    Article  ADS  MathSciNet  MATH  Google Scholar 

  9. Nair, C., Prabhakar, B., Shah, D.: On entropy for mixtures of discrete and continuous variables. arXiv:cs/0607075

  10. Ellison, C.J., Mahoney, J.R., Crutchfield, J.P.: Prediction, retrodiction, and the amount of information stored in the present. J. Stat. Phys. 136(6), 1005–1034 (2009)

    Article  ADS  MathSciNet  MATH  Google Scholar 

  11. Crutchfield, J.P., Ellison, C.J., Mahoney, J.R.: Time’s barbed arrow: Irreversibility, crypticity, and stored information. Phys. Rev. Lett. 103(9), 094101 (2009)

    Article  ADS  Google Scholar 

  12. Gaspard, P., Wang, X.-J.: Noise, chaos, and (\(\epsilon,\tau \))-entropy per unit time. Phys. Rep 235(6), 291–343 (1993)

    Article  ADS  MathSciNet  Google Scholar 

  13. Marzen, S., DeWeese, M.R., Crutchfield, J.P.: Time resolution dependence of information measures for spiking neurons: Scaling and universality. Front. Comput. Neurosci. 9, 109 (2015)

    Article  Google Scholar 

  14. Cover, T.M., Thomas, J.A.: Elements of Information Theory. Wiley-Interscience, New York (1991)

    Book  MATH  Google Scholar 

  15. Rao, M., Chen, Y., Vemuri, B.C., Wang, F.: Cumulative residual entropy: a new measure of information. IEEE Trans. Inform. Theory 50(6), 1220–1228 (2004)

    Article  MathSciNet  MATH  Google Scholar 

  16. Marzen, S., Crutchfield, J.P.: Informational and causal architecture of discrete-time renewal processes. Entropy 17(7), 4891–4917 (2015)

    Article  ADS  Google Scholar 

  17. Lowen, S.B., Teich, M.C.: Fractal renewal processes generate 1/f noise. Phys. Rev. E 47(2), 992–1001 (1993)

    Article  ADS  Google Scholar 

  18. Marzen, S.E., Crutchfield, J.P.: Statistical signatures of structural organization: The case of long memory in renewal processes. Phys. Lett. A 380(17), 1517–1525 (2016)

    Article  ADS  MathSciNet  Google Scholar 

  19. Risken, H.: The Fokker–Planck Equation: Methods of Solution and Applications. Springer Series in Synergetics. Springer, New York (2012)

    MATH  Google Scholar 

  20. James, R.G., Ellison, C.J., Crutchfield, J.P.: Anatomy of a bit: Information in a time series observation. CHAOS 21(3), 037109 (2011)

    Article  ADS  MathSciNet  MATH  Google Scholar 

  21. Yeung, R.W.: A new outlook on shannon’s information measures. IEEE Trans. Inform. Theory 37(3), 466–474 (1991)

    Article  MathSciNet  MATH  Google Scholar 

  22. Pinsker, M.S.: Information and Information Stability of Random Variables and Processes. Holden-Day Series in Time Series Analysis. Holden-Day, San Francisco (1964)

    MATH  Google Scholar 

  23. Girardin, V.: On the different extensions of the ergodic theorem of information theory. In: Baeza-Yates, R., Glaz, J., Gzyl, H., Husler, J., Palacios, J.L. (eds.) Recent Advances in Applied Probability Theory, pp. 163–179. Springer, New York (2005)

    Chapter  Google Scholar 

  24. Victor, J.D.: Binless strategies for estimation of information from neural data. Phys. Rev. E 66(5), 051903 (2002)

    Article  ADS  Google Scholar 

  25. Girardin, V., Limnios, N.: On the entropy for semi-Markov processes. J. Appl. Prob. 40(4), 1060–1068 (2003)

    Article  MathSciNet  MATH  Google Scholar 

  26. Marzen, S., Crutchfield, J.P.: Structure and randomness of continuous-time discrete-event processes. In preparation (2017)

Download references

Acknowledgements

The authors thank the Santa Fe Institute for its hospitality during visits. JPC is an SFI External Faculty member. This material is based upon work supported by, or in part by, the U. S. Army Research Laboratory and the U. S. Army Research Office under contract number W911NF-13-1-0390. SM was funded by a National Science Foundation Graduate Student Research Fellowship, a U.C. Berkeley Chancellor’s Fellowship, and the MIT Physics of Living Systems Fellowship.

Author information

Authors and Affiliations

  1. Physics of Living Systems Group, Department of Physics, Massachusetts Institute of Technology, Cambridge, MA, 02139, USA

    Sarah Marzen

  2. Department of Physics, University of California at Berkeley, Berkeley, CA, 94720, USA

    Sarah Marzen

  3. Complexity Sciences Center and Department of Physics, University of California at Davis, One Shields Avenue, Davis, CA, 95616, USA

    James P. Crutchfield

Authors
  1. Sarah Marzen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. James P. Crutchfield
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to James P. Crutchfield.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Marzen, S., Crutchfield, J.P. Informational and Causal Architecture of Continuous-time Renewal Processes. J Stat Phys 168, 109–127 (2017). https://doi.org/10.1007/s10955-017-1793-z

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10955-017-1793-z

Keywords

Search

Navigation