Journal of Statistical Physics

, Volume 138, Issue 4–5, pp 767–779 | Cite as

On the Entropy Production of Time Series with Unidirectional Linearity

  • Dominik JanzingEmail author
Open Access


There are non-Gaussian time series that admit a causal linear autoregressive moving average (ARMA) model when regressing the future on the past, but not when regressing the past on the future. The reason is that, in the latter case, the regression residuals are not statistically independent of the regressor. In previous work, we have experimentally verified that many empirical time series indeed show such a time inversion asymmetry.

For various physical systems, it is known that time-inversion asymmetries are linked to the thermodynamic entropy production in non-equilibrium states. Here we argue that unidirectional linearity is also accompanied by entropy generation.

To this end, we study the dynamical evolution of a physical toy system with linear coupling to an infinite environment and show that the linearity of the dynamics is inherited by the forward-time conditional probabilities, but not by the backward-time conditionals. The reason is that the environment permanently provides particles that are in a product state before they interact with the system, but show statistical dependence afterwards. From a coarse-grained perspective, the interaction thus generates entropy. We quantitatively relate the strength of the non-linearity of the backward process to the minimal amount of entropy generation.

The paper thus shows that unidirectional linearity is an indirect implication of the thermodynamic arrow of time, given that the joint dynamics of the system and its environment is linear.

Arrow of time Entropy production Irreversible processes Time series ARMA models 


  1. 1.
    Reichenbach, H.: The Direction of Time. University of California Press, Berkley (1956). Dover, New York (1999) Google Scholar
  2. 2.
    Penrose, O., Percival, I.: The direction of time. Proc. Phys. Soc. 79, 605–616 (1962) zbMATHCrossRefMathSciNetGoogle Scholar
  3. 3.
    Balian, R.: From Microphysics to Macrophysics. Springer, Berlin (1992) zbMATHGoogle Scholar
  4. 4.
    Gibbs, J.W.: Elementary Principles in Statistical Mechanics. Ox Bow Press, Woodbridge (1902) zbMATHGoogle Scholar
  5. 5.
    Jaynes, E.T.: Gibbs vs. Boltzmann entropies. Am. J. Phys. 33, 391 (1965) zbMATHCrossRefADSGoogle Scholar
  6. 6.
    Lebowitz, J.: Macroscopic dynamics, time’s arrow and Boltzmann entropy. Physica A 194, 1–27 (1993) CrossRefMathSciNetADSGoogle Scholar
  7. 7.
    Wallace, C.: Statistical and Inductive Inference by Minimum Message Length. Springer, Berlin (2005) zbMATHGoogle Scholar
  8. 8.
    Peters, J., Janzing, D., Gretton, A., Schölkopf, B.: Detecting the direction of causal time series. In: Proceedings of the International Conference on Machine Learning, Montreal. ACM International Conference Proceeding Series, vol. 382, pp. 801–808. ACM, New York (2009). and Google Scholar
  9. 9.
    Peters, J., Janzing, D., Gretton, A., Schölkopf, B.: Kernel methods for detecting the direction of time series. In: Proceedings of the 32nd Annual Conference of the German Classification Society (GfCKI 2008), pp. 1–10. Springer, Berlin (2009) Google Scholar
  10. 10.
    Maes, C., Redig, F., Van Moffaert, A.: On the definition of entropy production via examples. J. Math. Phys. 41, 1528–1554 (2000) zbMATHCrossRefMathSciNetADSGoogle Scholar
  11. 11.
    Gallavotti, G., Cohen, E.: Dynamical ensembles and nonequilibrium statistical mechanics. Phys. Rev. Lett. 74, 2694–2697 (1995) CrossRefADSGoogle Scholar
  12. 12.
    Horowitz, E., Sahni, S.: Fundamentals of Data Structures. Computer Science Press, New York (1976) zbMATHGoogle Scholar
  13. 13.
    Maes, C., Netocný, K.: Time reversal and entropy. J. Stat. Phys. 110(1–2), 269–309 (2003) zbMATHCrossRefGoogle Scholar
  14. 14.
    Chazottes, J.-R., Redig, F.: Testing the irreversibility of a Gibbsian process via hitting and return times. Nonlinearity 18(18), 2477–2489 (2005) zbMATHCrossRefMathSciNetADSGoogle Scholar
  15. 15.
    Darmois, G.: Analyse générale des liaisons stochastiques. Rev. Inst. Int. Stat. 21, 2–8 (1953) CrossRefMathSciNetGoogle Scholar
  16. 16.
    Skitovic, V.: Linear combinations of independent random variables and the normal distribution law. Sel. Transl. Math. Stat. Probab. 2, 211–228 (1962) zbMATHMathSciNetGoogle Scholar
  17. 17.
    Kano, Y., Shimizu, S.: Causal inference using nonnormality. In: Proceedings of the International Symposium on Science of Modeling, the 30th Anniversary of the Information Criterion, Tokyo, Japan, pp. 261–270 (2003) Google Scholar
  18. 18.
    Shimizu, S., Hyvärinen, A., Kano, Y., Hoyer, P.O.: Discovery of non-Gaussian linear causal models using ICA. In: Proceedings of the 21st Conference on Uncertainty in Artificial Intelligence, Edinburgh, UK, pp. 526–533 (2005) Google Scholar
  19. 19.
    Brockwell, P., Davis, R.: Time Series: Theory and Methods. Springer, Berlin (1991) CrossRefGoogle Scholar
  20. 20.
    Cover, T., Thomas, J.: Elements of Information Theory. Wileys Series in Telecommunications. Wiley, New York (1991) Google Scholar
  21. 21.
    Tolman, R.: The Principles of Statistical Mechanics. Oxford University Press, Oxford (1938) Google Scholar
  22. 22.
    Lemeire, J., Dirkx, E.: Causal models as minimal descriptions of multivariate systems. (2006)
  23. 23.
    Janzing, D., Schölkopf, B.: Causal inference using the algorithmic Markov condition. (2008)
  24. 24.
    Zurek, W.: Algorithmic randomness and physical entropy. Phys Rev A 40(8), 4731–4751 (1989) CrossRefMathSciNetADSGoogle Scholar
  25. 25.
    Hoyer, P., Janzing, D., Mooij, J., Peters, J., Schölkopf, B.: Nonlinear causal discovery with additive noise models. In: Koller, D., Schuurmans, D., Bengio, Y., Bottou, L. (eds.) Proceedings of the Conference on Neural Information Processing Systems (NIPS), Vancouver, Canada, 2008. MIT Press, Cambridge (2009). Google Scholar
  26. 26.
    Mooij, J., Janzing, D., Peters, J., Schölkopf, B.: Regression by dependence minimization and its application to causal inference. In: Proceedings of the International Conference on Machine Learning, Montreal. ACM International Conference Proceeding Series, vol. 382, pp. 745–752. ACM, New York (2009) and Google Scholar
  27. 27.
    Zhang, K., Hyvärinen, A.: On the identifiability of the post-nonlinear causal model. In: 25th Conference on Uncertainty in Artificial Intelligence, Montreal, Canada (2009) Google Scholar
  28. 28.
    Janzing, D.: On causally asymmetric versions of Occam’s Razor and their relation to thermodynamics. (2008)
  29. 29.
    Allahverdyan, A., Janzing, D.: Relating the thermodynamic arrow of time to the causal arrow. J. Stat. Mech. P04001 (2008) Google Scholar

Copyright information

© The Author(s) 2009

Authors and Affiliations

  1. 1.Max Planck Institute for Biological CyberneticsTübingenGermany

Personalised recommendations