Abstract
We address the issue of inferring the connectivity structure of spatially extended dynamical systems by estimation of mutual information between pairs of sites. The well-known problems resulting from correlations within and between the time series are addressed by explicit temporal and spatial modelling steps which aim at approximately removing all spatial and temporal correlations, i.e. at whitening the data, such that it is replaced by spatiotemporal innovations; this approach provides a link to the maximum-likelihood method and, for appropriately chosen models, removes the problem of estimating probability distributions of unknown, possibly complicated shape. A parsimonious multivariate autoregressive model based on nearest-neighbour interactions is employed. Mutual information can be reinterpreted in the framework of dynamical model comparison (i.e. likelihood ratio testing), since it is shown to be equivalent to the difference of the log-likelihoods of coupled and uncoupled models for a pair of sites, and a parametric estimator of mutual information can be derived. We also discuss, within the framework of model comparison, the relationship between the coefficient of linear correlation and mutual information. The practical application of this methodology is demonstrated for simulated multivariate time series generated by a stochastic coupled-map lattice. The parsimonious modelling approach is compared to general multivariate autoregressive modelling and to Independent Component Analysis (ICA).
Similar content being viewed by others
References
P. E. Cladis and P. Palffy-Muhoray (eds.), Spatio-Temporal Patterns in Nonequilibrium Complex Systems. SFI Studies in the Sciences of Complexity (Addison Wesley Longman, Reading, MA, 1995).
L. B. Almeida, F. Lopes da Silva and J. C. Principe (eds.), Spatiotemporal Models in Biological and Artificial Systems, volume 37 of Frontiers in Artificial Intelligence and Applications (IOS Press, Amsterdam, 1997).
D. Walgraef, Spatio-temporal Pattern Formation (Springer, Berlin, Heidelberg, New York, 1997).
E. Gehrig and O. Hess, Spatio-Temporal Dynamics and Quantum Fluctuations in Semiconductor Lasers. Springer Tracts in Modern Physics, Vol. 189 (Springer, Berlin, Heidelberg, New York, 2003).
G. Christakos, Modern Spatiotemporal Geostatistics. Studies in Mathematical Geology, Vol. 6 (Oxford University Press, Oxford, 2000).
J. Bascompte and R. V. Solé (eds.), Modelling Spatiotemporal Dynamics in Ecology (Springer, Berlin, Heidelberg, New York, 1998).
E. Yago, C. Escera, K. Alho, M. H. Giard and J. M. Serra-Grabulosa, Spatiotemporal dynamics of the auditory novelty-P3 event-related brain potential. Cogn. Brain Res. 16:383–390, 2003.
C. Lukas, J. Falck, J. Bartkova, J. Bartek and J. Lukas, Distinct spatiotemporal dynamics of mammalian checkpoint regulators induced by DNA damage. Nature Cell Biol. 5:255–260, 2003.
M. Mori, M. Kaino, S. Kanemoto, M. Enomoto, S. Ebata and S. Tsunoyama, Development of advanced core noise monitoring system for BWRS. Prog. Nucl. Energy 43:201–207, 2003.
W. Li, Mutual information functions versus correlation functions. J. Stat. Phys. 60:823–837, 1990.
P. Hall and S. C. Morton, On the estimation of entropy. Ann. Inst. Statist. Math. 45:69–88, 1993.
D. R. Brillinger, Second-order moments and mutual information in the analysis of time series. In Y. P. Chaubey (ed.), Recent Advances in Statistical Methods, pp. 64–76 (Imperial College Press, London, 2002).
D. R. Brillinger, Some data analyses using mutual information. Brazilian J. Prob. Statist. 18:163–183, 2004.
K. Kaneko, Spatiotemporal chaos in one- and two-dimensional coupled map lattices. Physica D 37:60–82, 1989.
C. E. Shannon, A mathematical theory of communication. Bell Syst. Tech. J. 27:379–423, 623–656, 1948.
L. Boltzmann, Über die Beziehung zwischen dem zweiten Hauptsatze der mechanischen Wärmetheorie und der Wahrscheinlichkeitsrechnung respektive den Sätzen über das Wärmegleichgewicht. Wiener Berichte 76:373–435, 1877.
H. Akaike, On the likelihood of a time series model. The Statistician 27:217–235, 1978.
T. M. Cover and J. A. Thomas, Elements of Information Theory (Wiley, New York, 1991).
Y. Moon, B. Rajagopalan and U. Lall, Estimation of mutual information using kernel density estimators. Phys. Rev. E 52:2318–2321, 1995.
D. Prichard and J. Theiler, Generalized redundancies for time series analysis. Physica D 84:476–493, 1995.
A. Kraskov, Harald Stögbauer and P. Grassberger, Estimating mutual information. Phys. Rev. E 69:066138, 2004.
R. Moddemeijer, A statistic to estimate the variance of the histogram based mutual information estimator based on dependent pairs of observations. Signal Proc. 75:51–63, 1999.
H. Stögbauer, A. Kraskov, S. A. Astakhov and P. Grassberger, Least-dependent-component analysis based on mutual information. Phys. Rev. E 70:066123, 2004.
K. Ito, On stochastic differential equations. Mem. Am. Math. Soc. 4:1–51, 1951.
J. L. Doob, Stochastic Processes (Wiley, New York, 1953).
A. H. Jazwinski, Stochastic Processes and Filtering Theory (Academic Press, San Diego, 1970).
T. Kailath, A view of three decades of linear filtering theory. IEEE Trans. Inf. Theory 20:146–181, 1974.
P. Lévy, Sur une classe de courbes de l'espace de Hilbert et sur une équation intégrale non linéaire. Ann. Sci. École Norm. Sup. 73:121–156, 1956.
P. Protter, Stochastic Integration and Differential Equations (Springer-Verlag, Berlin, 1990).
P. A. Frost and T. Kailath, An innovation approach to least squares estimation—part III: Nonlinear estimation in white gaussian noise. IEEE Trans. Autom. Contr. 16:217–226, 1971.
B. V. Gnedenko, The Theory of Probability (Mir Publishers, Moscow, 1969).
G. E. P. Box and G. M. Jenkins, Time Series Analysis, Forecasting and Control 2nd edition (Holden-Day, San Francisco, 1976).
R. Moddemeijer, On estimation of entropy and mutual information of continuous distributions. Signal Proc. 16:233–248, 1989.
A. Galka and T. Ozaki, Testing for nonlinearity in high-dimensional time series from continuous dynamics. Physica D 158:32–44, 2001.
H. Akaike, Prediction and entropy. In A. C. Atkinson and S. E. Fienberg (eds.), A Celebration of Statistics, pp. 1–24 (Springer, Berlin, Heidelberg, New York, 1985).
J. Kuha, AIC and BIC: Comparisons of assumptions and performance. Sociol. Methods Res. 33:188–229, 2004.
M. Stone, An asymptotics equivalence of choice of models by cross-validation and Akaike's criterion. J. Roy. Statist. Soc. B 39:44–47, 1977.
N. Levinson, The Wiener rms error criterion in filter design and prediction. J. Math. Phys. 25:261–278, 1947.
P. Whittle, On the fitting of multivariate autoregressions, and the approximate canonical factorization of a spectral density matrix. Biometrika 50:129–134, 1963.
A. Neumaier and T. Schneider, Estimation of parameters and eigenmodes of multivariate autoregressive models. ACM Trans. Math. Softw. 27:27–57, 2001.
C. Penland, Random forcing and forecasting using principal oscillation pattern analysis. Monthly Weather Rev. 117:2165–2185, 1989.
A. Galka, O. Yamashita, T. Ozaki, R. Biscay and P. A. Valdés-Sosa, A solution to the dynamical inverse problem of EEG generation using spatiotemporal Kalman filtering. NeuroImage 23:435–453, 2004.
A. Galka, O. Yamashita and T. Ozaki, GARCH modelling of covariance in dynamical estimation of inverse solutions. Phys. Lett. A 333:261–268, 2004.
J. Geweke, Inference and causality in economic time series models. In Z. Grilliches and M. Intriligator (eds.), Handbook of Econometrics, pp. 1101–1144 (North-Holland, Amsterdam, 1984).
A. Buse, The likelihood ratio, Wald, and Lagrange multiplier test: An expository note. Am. Stat. 36:153–157, 1982.
A. Hyvärinen, J. Karhunen and E. Oja, Independent Component Analysis (Wiley, New York, 2001).
L. Molgedey and H. G. Schuster, Separation of a mixture of independent signals using time delayed correlations. Phys. Rev. Lett. 72:3634–3637, 1994.
T. Schreiber, Spatio-temporal structure in coupled map lattices: two-point correlations versus mutual information. J. Phys. A: Math. Gen. 23:L393–L398, 1990.
H. Wu and J. Principe, Generalized anti-Hebbian learning for source separation. In Proceedings of ICASSP'99, pp. 1073–1076 (IEEE Press, Piscataway, NJ, 1999).
S. Choi, A. Cichocki and S. Amari, Flexible independent component analysis. J. VLSI Signal Process 26:25–38, 2000.
K. G. Jöreskog, Some contributions to maximum likelihood Factor Analysis. Psychometrika 32:443–482, 1967.
B. Flury, A First Course in Multivariate Statistics (Springer, New York, Berlin, Heidelberg, 1997).
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Galka, A., Ozaki, T., Bayard, J.B. et al. Whitening as a Tool for Estimating Mutual Information in Spatiotemporal Data Sets. J Stat Phys 124, 1275–1315 (2006). https://doi.org/10.1007/s10955-006-9131-x
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10955-006-9131-x