Skip to main content
SpringerLink
Log in

Whitening as a Tool for Estimating Mutual Information in Spatiotemporal Data Sets

  • Published:
Journal of Statistical Physics Aims and scope Submit manuscript

Abstract

We address the issue of inferring the connectivity structure of spatially extended dynamical systems by estimation of mutual information between pairs of sites. The well-known problems resulting from correlations within and between the time series are addressed by explicit temporal and spatial modelling steps which aim at approximately removing all spatial and temporal correlations, i.e. at whitening the data, such that it is replaced by spatiotemporal innovations; this approach provides a link to the maximum-likelihood method and, for appropriately chosen models, removes the problem of estimating probability distributions of unknown, possibly complicated shape. A parsimonious multivariate autoregressive model based on nearest-neighbour interactions is employed. Mutual information can be reinterpreted in the framework of dynamical model comparison (i.e. likelihood ratio testing), since it is shown to be equivalent to the difference of the log-likelihoods of coupled and uncoupled models for a pair of sites, and a parametric estimator of mutual information can be derived. We also discuss, within the framework of model comparison, the relationship between the coefficient of linear correlation and mutual information. The practical application of this methodology is demonstrated for simulated multivariate time series generated by a stochastic coupled-map lattice. The parsimonious modelling approach is compared to general multivariate autoregressive modelling and to Independent Component Analysis (ICA).

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. P. E. Cladis and P. Palffy-Muhoray (eds.), Spatio-Temporal Patterns in Nonequilibrium Complex Systems. SFI Studies in the Sciences of Complexity (Addison Wesley Longman, Reading, MA, 1995).

  2. L. B. Almeida, F. Lopes da Silva and J. C. Principe (eds.), Spatiotemporal Models in Biological and Artificial Systems, volume 37 of Frontiers in Artificial Intelligence and Applications (IOS Press, Amsterdam, 1997).

  3. D. Walgraef, Spatio-temporal Pattern Formation (Springer, Berlin, Heidelberg, New York, 1997).

    MATH  Google Scholar 

  4. E. Gehrig and O. Hess, Spatio-Temporal Dynamics and Quantum Fluctuations in Semiconductor Lasers. Springer Tracts in Modern Physics, Vol. 189 (Springer, Berlin, Heidelberg, New York, 2003).

    Google Scholar 

  5. G. Christakos, Modern Spatiotemporal Geostatistics. Studies in Mathematical Geology, Vol. 6 (Oxford University Press, Oxford, 2000).

    Google Scholar 

  6. J. Bascompte and R. V. Solé (eds.), Modelling Spatiotemporal Dynamics in Ecology (Springer, Berlin, Heidelberg, New York, 1998).

    Google Scholar 

  7. E. Yago, C. Escera, K. Alho, M. H. Giard and J. M. Serra-Grabulosa, Spatiotemporal dynamics of the auditory novelty-P3 event-related brain potential. Cogn. Brain Res. 16:383–390, 2003.

    Article  Google Scholar 

  8. C. Lukas, J. Falck, J. Bartkova, J. Bartek and J. Lukas, Distinct spatiotemporal dynamics of mammalian checkpoint regulators induced by DNA damage. Nature Cell Biol. 5:255–260, 2003.

    Article  Google Scholar 

  9. M. Mori, M. Kaino, S. Kanemoto, M. Enomoto, S. Ebata and S. Tsunoyama, Development of advanced core noise monitoring system for BWRS. Prog. Nucl. Energy 43:201–207, 2003.

    Article  Google Scholar 

  10. W. Li, Mutual information functions versus correlation functions. J. Stat. Phys. 60:823–837, 1990.

    Article  MATH  Google Scholar 

  11. P. Hall and S. C. Morton, On the estimation of entropy. Ann. Inst. Statist. Math. 45:69–88, 1993.

    Article  MATH  MathSciNet  Google Scholar 

  12. D. R. Brillinger, Second-order moments and mutual information in the analysis of time series. In Y. P. Chaubey (ed.), Recent Advances in Statistical Methods, pp. 64–76 (Imperial College Press, London, 2002).

    Google Scholar 

  13. D. R. Brillinger, Some data analyses using mutual information. Brazilian J. Prob. Statist. 18:163–183, 2004.

    Google Scholar 

  14. K. Kaneko, Spatiotemporal chaos in one- and two-dimensional coupled map lattices. Physica D 37:60–82, 1989.

    Article  ADS  MathSciNet  Google Scholar 

  15. C. E. Shannon, A mathematical theory of communication. Bell Syst. Tech. J. 27:379–423, 623–656, 1948.

  16. L. Boltzmann, Über die Beziehung zwischen dem zweiten Hauptsatze der mechanischen Wärmetheorie und der Wahrscheinlichkeitsrechnung respektive den Sätzen über das Wärmegleichgewicht. Wiener Berichte 76:373–435, 1877.

    Google Scholar 

  17. H. Akaike, On the likelihood of a time series model. The Statistician 27:217–235, 1978.

    Article  MathSciNet  Google Scholar 

  18. T. M. Cover and J. A. Thomas, Elements of Information Theory (Wiley, New York, 1991).

    MATH  Google Scholar 

  19. Y. Moon, B. Rajagopalan and U. Lall, Estimation of mutual information using kernel density estimators. Phys. Rev. E 52:2318–2321, 1995.

    Article  ADS  Google Scholar 

  20. D. Prichard and J. Theiler, Generalized redundancies for time series analysis. Physica D 84:476–493, 1995.

    Article  MATH  Google Scholar 

  21. A. Kraskov, Harald Stögbauer and P. Grassberger, Estimating mutual information. Phys. Rev. E 69:066138, 2004.

    Google Scholar 

  22. R. Moddemeijer, A statistic to estimate the variance of the histogram based mutual information estimator based on dependent pairs of observations. Signal Proc. 75:51–63, 1999.

    Google Scholar 

  23. H. Stögbauer, A. Kraskov, S. A. Astakhov and P. Grassberger, Least-dependent-component analysis based on mutual information. Phys. Rev. E 70:066123, 2004.

    Google Scholar 

  24. K. Ito, On stochastic differential equations. Mem. Am. Math. Soc. 4:1–51, 1951.

    Google Scholar 

  25. J. L. Doob, Stochastic Processes (Wiley, New York, 1953).

  26. A. H. Jazwinski, Stochastic Processes and Filtering Theory (Academic Press, San Diego, 1970).

  27. T. Kailath, A view of three decades of linear filtering theory. IEEE Trans. Inf. Theory 20:146–181, 1974.

    Article  MATH  MathSciNet  Google Scholar 

  28. P. Lévy, Sur une classe de courbes de l'espace de Hilbert et sur une équation intégrale non linéaire. Ann. Sci. École Norm. Sup. 73:121–156, 1956.

    MATH  Google Scholar 

  29. P. Protter, Stochastic Integration and Differential Equations (Springer-Verlag, Berlin, 1990).

    MATH  Google Scholar 

  30. P. A. Frost and T. Kailath, An innovation approach to least squares estimation—part III: Nonlinear estimation in white gaussian noise. IEEE Trans. Autom. Contr. 16:217–226, 1971.

    Article  Google Scholar 

  31. B. V. Gnedenko, The Theory of Probability (Mir Publishers, Moscow, 1969).

    Google Scholar 

  32. G. E. P. Box and G. M. Jenkins, Time Series Analysis, Forecasting and Control 2nd edition (Holden-Day, San Francisco, 1976).

    Google Scholar 

  33. R. Moddemeijer, On estimation of entropy and mutual information of continuous distributions. Signal Proc. 16:233–248, 1989.

    Google Scholar 

  34. A. Galka and T. Ozaki, Testing for nonlinearity in high-dimensional time series from continuous dynamics. Physica D 158:32–44, 2001.

    Article  MATH  ADS  Google Scholar 

  35. H. Akaike, Prediction and entropy. In A. C. Atkinson and S. E. Fienberg (eds.), A Celebration of Statistics, pp. 1–24 (Springer, Berlin, Heidelberg, New York, 1985).

    Google Scholar 

  36. J. Kuha, AIC and BIC: Comparisons of assumptions and performance. Sociol. Methods Res. 33:188–229, 2004.

    Article  MathSciNet  Google Scholar 

  37. M. Stone, An asymptotics equivalence of choice of models by cross-validation and Akaike's criterion. J. Roy. Statist. Soc. B 39:44–47, 1977.

    Google Scholar 

  38. N. Levinson, The Wiener rms error criterion in filter design and prediction. J. Math. Phys. 25:261–278, 1947.

    MathSciNet  Google Scholar 

  39. P. Whittle, On the fitting of multivariate autoregressions, and the approximate canonical factorization of a spectral density matrix. Biometrika 50:129–134, 1963.

    Article  MATH  MathSciNet  Google Scholar 

  40. A. Neumaier and T. Schneider, Estimation of parameters and eigenmodes of multivariate autoregressive models. ACM Trans. Math. Softw. 27:27–57, 2001.

    Article  MATH  Google Scholar 

  41. C. Penland, Random forcing and forecasting using principal oscillation pattern analysis. Monthly Weather Rev. 117:2165–2185, 1989.

    Article  Google Scholar 

  42. A. Galka, O. Yamashita, T. Ozaki, R. Biscay and P. A. Valdés-Sosa, A solution to the dynamical inverse problem of EEG generation using spatiotemporal Kalman filtering. NeuroImage 23:435–453, 2004.

    Article  Google Scholar 

  43. A. Galka, O. Yamashita and T. Ozaki, GARCH modelling of covariance in dynamical estimation of inverse solutions. Phys. Lett. A 333:261–268, 2004.

    Article  ADS  Google Scholar 

  44. J. Geweke, Inference and causality in economic time series models. In Z. Grilliches and M. Intriligator (eds.), Handbook of Econometrics, pp. 1101–1144 (North-Holland, Amsterdam, 1984).

    Google Scholar 

  45. A. Buse, The likelihood ratio, Wald, and Lagrange multiplier test: An expository note. Am. Stat. 36:153–157, 1982.

    Article  Google Scholar 

  46. A. Hyvärinen, J. Karhunen and E. Oja, Independent Component Analysis (Wiley, New York, 2001).

    MATH  Google Scholar 

  47. L. Molgedey and H. G. Schuster, Separation of a mixture of independent signals using time delayed correlations. Phys. Rev. Lett. 72:3634–3637, 1994.

    Article  ADS  Google Scholar 

  48. T. Schreiber, Spatio-temporal structure in coupled map lattices: two-point correlations versus mutual information. J. Phys. A: Math. Gen. 23:L393–L398, 1990.

    Article  ADS  MathSciNet  Google Scholar 

  49. H. Wu and J. Principe, Generalized anti-Hebbian learning for source separation. In Proceedings of ICASSP'99, pp. 1073–1076 (IEEE Press, Piscataway, NJ, 1999).

  50. S. Choi, A. Cichocki and S. Amari, Flexible independent component analysis. J. VLSI Signal Process 26:25–38, 2000.

    Google Scholar 

  51. K. G. Jöreskog, Some contributions to maximum likelihood Factor Analysis. Psychometrika 32:443–482, 1967.

    Article  MATH  MathSciNet  Google Scholar 

  52. B. Flury, A First Course in Multivariate Statistics (Springer, New York, Berlin, Heidelberg, 1997).

    MATH  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Institute of Statistical Mathematics (ISM), Minami-Azabu 4-6-7, Tokyo, 106-8569, Japan

    Andreas Galka & Tohru Ozaki

  2. Institute of Experimental and Applied Physics, University of Kiel, 24098, Kiel, Germany

    Andreas Galka

  3. Cuban Neuroscience Center, Ave 25 No. 5202 esquina 158 Cubanacán, POB 6880, 6990, Ciudad Habana, Cuba

    Jorge Bosch Bayard

  4. ATR Computational Neuroscience Laboratories, Hikaridai 2-2-2, Kyoto, 619-0288, Japan

    Okito Yamashita

Authors
  1. Andreas Galka
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Tohru Ozaki
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jorge Bosch Bayard
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Okito Yamashita
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Andreas Galka.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Galka, A., Ozaki, T., Bayard, J.B. et al. Whitening as a Tool for Estimating Mutual Information in Spatiotemporal Data Sets. J Stat Phys 124, 1275–1315 (2006). https://doi.org/10.1007/s10955-006-9131-x

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10955-006-9131-x

Keywords

Search

Navigation