Efficient Estimation of Information Transfer

Part of the Understanding Complex Systems book series (UCS)


Any measure of interdependence can lose much of its appeal due to a poor choice of its numerical estimator. Information theoretic functionals are particularly sensitive to this problem, especially when applied to noisy signals of only a few thousand data points or less. Unfortunately, this is a common scenario in applications to electrophysiology data sets. In this chapter, we will review the stateof- the-art estimators based on nearest-neighbor statistics for information transfer measures. Nearest neighbors techniques are more data-efficient than naive partition or histogram estimators and rely on milder assumptions than parametric approaches. However, they also come with limitations and several parameter choices that influence the numerical estimation of information theoretic functionals.We will describe step by step the efficient estimation of transfer entropy for a typical electrophysiology data set, and how the multi-trial structure of such data sets can be used to partially alleviate the problem of non-stationarity.


Mutual Information Spike Train Information Transfer Granger Causality Shannon Entropy 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Hubel, D.H., Wiesel, T.N.: Receptive fields and functional architecture of monkey striate cortex. The Journal of Physiology 195(1), 215–243 (1968)Google Scholar
  2. 2.
    Gray, C.M., Knig, P., Engel, A.K., Singer, W.: Oscillatory responses in cat visual cortex exhibit inter-columnar synchronization which reflects global stimulus properties. Nature 338(6213), 334–337 (1989)CrossRefGoogle Scholar
  3. 3.
    Canolty, R.T., Knight, R.T.: The functional role of cross-frequency coupling. Trends in Cognitive Sciences 14(11), 506–515 (2010)CrossRefGoogle Scholar
  4. 4.
    Victor, J.D.: Approaches to information-theoretic analysis of neural activity. Biological Theory 1(3), 302 (2006)CrossRefGoogle Scholar
  5. 5.
    Lizier, J.T.: The Local Information Dynamics of Distributed Computation in Complex Systems. Springer theses. Springer (2013)Google Scholar
  6. 6.
    Lehmann, E.L., Casella, G.: Theory of point estimation, vol. 31. Springer (1998)Google Scholar
  7. 7.
    Niso, G., Brua, R., Pereda, E., Gutirrez, R., Bajo, R., Maest, F., Del-Pozo, F.: Hermes: Towards an integrated toolbox to characterize functional and effective brain connectivity. Neuroinformatics 11, 405–434 (2013)CrossRefGoogle Scholar
  8. 8.
    Pereda, E., Quiroga, R.Q., Bhattacharya, J.: Nonlinear multivariate analysis of neurophysiological signals. Progress in Neurobiology 77(1), 1–37 (2005)Google Scholar
  9. 9.
    Cover, T.M., Thomas, J.A.: Elements of information theory. Wiley-Interscience, New York (1991)CrossRefMATHGoogle Scholar
  10. 10.
    Latham, P.E., Nirenberg, S.: Synergy, redundancy, and independence in population codes, revisited. J. Neurosci. 25(21), 5195–5206 (2005)CrossRefGoogle Scholar
  11. 11.
    Williams, P.L., Beer, R.D.: Nonnegative decomposition of multivariate information. arXiv preprint arXiv:1004.2515 (2010)Google Scholar
  12. 12.
    Rieke, F., Warland, D., Deruytervansteveninck, R., Bialek, W.: Spikes: exploring the neural code (computational neuroscience). MIT Press (1999)Google Scholar
  13. 13.
    Shannon, C.E.: The bell technical journal. A Mathematical Theory of Communication 27(4), 379–423 (1948)MATHMathSciNetGoogle Scholar
  14. 14.
    Shannon, C.E., Weaver, W.: The mathematical theory of communication, urbana, il, vol. 19(7), p. 1. University of Illinois Press (1949)Google Scholar
  15. 15.
    Barlow, H.B.: Possible principles underlying the transformation of sensory messages. Sensory Communication, 217–234 (1961)Google Scholar
  16. 16.
    de Ruyter van Steveninck, R.R., Laughlin, S.B.: The rate of information transfer at graded-potential synapses. Nature 379(6566), 642–645 (1996)CrossRefGoogle Scholar
  17. 17.
    Lewicki, M.S.: Efficient coding of natural sounds. Nature Neuroscience 5(4), 356–363 (2002)CrossRefGoogle Scholar
  18. 18.
    Olshausen, B.A., Field, D.J.: Sparse coding of sensory inputs. Current Opinion in Neurobiology 14(4), 481–487 (2004)CrossRefGoogle Scholar
  19. 19.
    Johnson, D.H.: Information theory and neuroscience: Why is the intersection so small? In: IEEE Information Theory Workshop, ITW 2008, pp. 104–108 (2008)Google Scholar
  20. 20.
    Shannon, C.E.: The bandwagon. IRE Transactions on Information Theory 2(1), 3 (1956)CrossRefGoogle Scholar
  21. 21.
    Nirenberg, S.H., Victor, J.D.: Analyzing the activity of large populations of neurons: how tractable is the problem? Current Opinion in Neurobiology 17(4), 397–400 (2007)CrossRefGoogle Scholar
  22. 22.
    Johnson, D.H.: Information theory and neural information processing. IEEE Transactions on Information Theory 56(2), 653–666 (2010)CrossRefGoogle Scholar
  23. 23.
    Schreiber, T.: Measuring information transfer. Phys. Rev. Lett. 85(2), 461–464 (2000)CrossRefGoogle Scholar
  24. 24.
    Wiener, N.: The theory of prediction. In: Beckmann, E.F. (ed.) Modern Mathematics for the Engineer. McGraw-Hill, New York (1956)Google Scholar
  25. 25.
    Vicente, R., Wibral, M., Lindner, M., Pipa, G.: Transfer entropy – a model-free measure of effective connectivity for the neurosciences. J. Comput. Neurosci. 30(1), 45–67 (2011)CrossRefMathSciNetGoogle Scholar
  26. 26.
    Ay, N., Polani, D.: Information flows in causal networks. Adv. Complex Syst. 11, 17 (2008)CrossRefMATHMathSciNetGoogle Scholar
  27. 27.
    Kaiser, A., Schreiber, T.: Information transfer in continuous processes. Physica D 166, 43 (2002)CrossRefMATHMathSciNetGoogle Scholar
  28. 28.
    Chicharro, D., Ledberg, A.: When two become one: the limits of causality analysis of brain dynamics. PLoS One 7(3), e32466 (2012)Google Scholar
  29. 29.
    Chávez, M., Martinerie, J., Le Van Quyen, M.: Statistical assessment of nonlinear causality: application to epileptic EEG signals. J. Neurosci. Methods 124(2), 113–128 (2003)CrossRefGoogle Scholar
  30. 30.
    Wibral, M., Rahm, B., Rieder, M., Lindner, M., Vicente, R., Kaiser, J.: Transfer entropy in magnetoencephalographic data: Quantifying information flow in cortical and cerebellar networks. Prog. Biophys. Mol. Biol. 105(1-2), 80–97 (2011)CrossRefGoogle Scholar
  31. 31.
    Vicente, R., Gollo, L.L., Mirasso, C.R., Fischer, I., Pipa, G.: Dynamical relaying can yield zero time lag neuronal synchrony despite long conduction delays. Proceedings of the National Academy of Sciences 105(44), 17157–17162 (2008)CrossRefGoogle Scholar
  32. 32.
    Kay, S.M.: Fundamentals of statistical signal processing. In: Estimation Theory, vol. 1 (1993)Google Scholar
  33. 33.
    Hlaváčková-Schindler, K., Paluš, M., Vejmelka, M., Bhattacharya, J.: Causality detection based on information-theoretic approaches in time series analysis. Physics Reports 441(1), 1–46 (2007)CrossRefGoogle Scholar
  34. 34.
    Gourevitch, B., Eggermont, J.J.: Evaluating information transfer between auditory cortical neurons. J. Neurophysiol. 97(3), 2533–2543 (2007)CrossRefGoogle Scholar
  35. 35.
    Ito, S., Hansen, M.E., Heiland, R., Lumsdaine, A., Litke, A.M., Beggs, J.M.: Extending transfer entropy improves identification of effective connectivity in a spiking cortical network model. PLoS One 6(11), e27431 (2011)Google Scholar
  36. 36.
    Li, Z., Li, X.: Estimating temporal causal interaction between spike trains with permutation and transfer entropy. PLoS One 8(8), e70894 (2013)Google Scholar
  37. 37.
    Barnett, L., Barrett, A.B., Seth, A.K.: Granger causality and transfer entropy are equivalent for Gaussian variables. Phys. Rev. Lett. 103(23), 238701 (2009)CrossRefGoogle Scholar
  38. 38.
    Hlavácková-Schindler, K.: Equivalence of Granger causality and transfer entropy: A generalization. Applied Mathematical Sciences 5(73), 3637–3648 (2011)MATHMathSciNetGoogle Scholar
  39. 39.
    Nichols, J.M., Seaver, M., Trickey, S.T., Todd, M.D., Olson, C., Overbey, L.: Detecting nonlinearity in structural systems using the transfer entropy. Phys. Rev. E Stat. Nonlin. Soft Matter Phys. 72(4 Pt. 2), 046217 (2005)Google Scholar
  40. 40.
    Hahs, D.W., Pethel, S.D.: Transfer entropy for coupled autoregressive processes. Entropy 15(3), 767–788 (2013)CrossRefMathSciNetGoogle Scholar
  41. 41.
    Barnett, L., Bossomaier, T.: Transfer entropy as a log-likelihood ratio. Physical Review Letters 109(13), 138105 (2012)CrossRefGoogle Scholar
  42. 42.
    Miller, G.A.: Note on the bias of information estimates. Information Theory in Psychology: Problems and Methods 2, 95–100 (1955)Google Scholar
  43. 43.
    Efron, B., Stein, C.: The jackknife estimate of variance. The Annals of Statistics, 586–596 (1981)Google Scholar
  44. 44.
    Pompe, B., Runge, J.: Momentary information transfer as a coupling measure of time series. Phys. Rev. E Stat. Nonlin. Soft Matter Phys. 83(5 Pt. 1), 051122 (2011)Google Scholar
  45. 45.
    Wibral, M., Pampu, N., Priesemann, V., Siebenhhner, F., Seiwert, H., Lindner, M., Lizier, J.T., Vicente, R.: Measuring information-transfer delays. PLoS One 8(2), e55809 (2013)Google Scholar
  46. 46.
    Paluš, M.: Testing for nonlinearity using redundancies: Quantitative and qualitative aspects. Physica D: Nonlinear Phenomena 80(1), 186–205 (1995)MATHGoogle Scholar
  47. 47.
    Fraser, A.M., Swinney, H.L.: Independent coordinates for strange attractors from mutual information. Phys. Rev. A 33, 1134 (1986)CrossRefMATHMathSciNetGoogle Scholar
  48. 48.
    Darbellay, G.A., Vajda, I.: Estimation of the information by an adaptive partitioning of the observation space. IEEE Transactions on Information Theory 45(4), 1315–1321 (1999)CrossRefMATHMathSciNetGoogle Scholar
  49. 49.
    Cellucci, C.J., Albano, A.M., Rapp, P.E.: Statistical validation of mutual information calculations: Comparison of alternative numerical algorithms. Physical Review E 71(6), 066208 (2005)Google Scholar
  50. 50.
    Daub, C.O., Steuer, R., Selbig, J., Kloska, S.: Estimating mutual information using b-spline functions–an improved similarity measure for analysing gene expression data. BMC Bioinformatics 5(1), 118 (2004)CrossRefGoogle Scholar
  51. 51.
    Victor, J.: Binless strategies for estimation of information from neural data. Phys. Rev. E 72, 051903 (2005)Google Scholar
  52. 52.
    Silverman, B.W.: Density estimation for statistics and data analysis, vol. 26. CRC Press (1986)Google Scholar
  53. 53.
    Young-Il, M., Rajagopalan, B., Lall, U.: Estimation of mutual information using kernel density estimators. Physical Review E 52(3), 2318 (1995)CrossRefGoogle Scholar
  54. 54.
    Steuer, R., Kurths, J., Daub, C.O., Weise, J., Selbig, J.: The mutual information: detecting and evaluating dependencies between variables. Bioinformatics 18(suppl. 2), S231–S240 (2002)Google Scholar
  55. 55.
    Kozachenko, L.F., Leonenko, N.N.: Sample estimate of entropy of a random vector. Probl. Inform. Transm. 23, 95–100 (1987)MATHMathSciNetGoogle Scholar
  56. 56.
    Knuth, D.E.: The art of computer programming. In: Sorting and Searching, vol. 3 (1973)Google Scholar
  57. 57.
    Vaidya, P.M.: An O(n logn) algorithm for the all-nearest-neighbors problem. Discrete & Computational Geometry 4(1), 101–115 (1989)CrossRefMATHMathSciNetGoogle Scholar
  58. 58.
    Zezula, P., Amato, G., Dohnal, V., Batko, M.: Similarity search: The metric space approach. Advances in Database Systems, vol. 32. Springer, Secaucus (2005)Google Scholar
  59. 59.
    Heineman, G.T., Pollice, G., Selkow, S.: Algorithms in a Nutshell. O’Reilly Media, Inc. (2009)Google Scholar
  60. 60.
    Merkwirth, P., Lauterborn, W.: Fast nearest-neighbor searching for nonlinear signal processing. Phys. Rev. E Stat. Phys. Plasmas Fluids Relat. Interdiscip. Topics 62(2 Pt. A), 2089–2097 (2000)Google Scholar
  61. 61.
    Wollstadt, P., Martinez-Zarzuela, M., Vicente, R., Wibral, M.: Efficient transfer entropy analysis of nonstationary neural time series. arXiv preprint arXiv:1401.4068 (2014)Google Scholar
  62. 62.
    Kraskov, A., Stoegbauer, H., Grassberger, P.: Estimating mutual information. Phys. Rev. E Stat. Nonlin. Soft Matter Phys. 69(6 Pt. 2), 066138 (2004)Google Scholar
  63. 63.
    Kraskov, A.: Synchronization and Interdependence measures and their application to the electroencephalogram of epilepsy patients and clustering of data. PhD thesis, University of Wuppertal (February 2004)Google Scholar
  64. 64.
    Gomez-Herrero, G., Wu, W., Rutanen, K., Soriano, M.C., Pipa, G., Vicente, R.: Assessing coupling dynamics from an ensemble of time series. arXiv preprint arXiv:1008.0539 (2010)Google Scholar
  65. 65.
    Takens, F.: Detecting Strange Attractors in Turbulence. In: Dynamical Systems and Turbulence, Warwick, 1980. Lecture Notes in Mathematics, vol. 898, pp. 366–381. Springer (1981)Google Scholar
  66. 66.
    Kantz, H., Schreiber, T.: Nonlinear Time Series Analysis, 2nd edn. Cambridge University Press (November 2003)Google Scholar
  67. 67.
    Cao, L.Y.: Practical method for determining the minimum embedding dimension of a scalar time series. Physica A 110, 43–50 (1997)MATHGoogle Scholar
  68. 68.
    Ragwitz, M., Kantz, H.: Markov models from data by simple nonlinear time series predictors in delay embedding spaces. Phys. Rev. E Stat. Nonlin. Soft Matter Phys. 65(5 Pt. 2), 056201 (2002)Google Scholar
  69. 69.
    Theiler, J.: Spurious dimension from correlation algorithms applied to limited time-series data. Physical Review A 34(3), 2427 (1986)CrossRefGoogle Scholar
  70. 70.
    Vejmelka, M., Hlaváčková-Schindler, K.: Mutual information estimation in higher dimensions: A speed-up of a k-nearest neighbor based estimator. In: Beliczynski, B., Dzielinski, A., Iwanowski, M., Ribeiro, B. (eds.) ICANNGA 2007, Part I. LNCS, vol. 4431, pp. 790–797. Springer, Heidelberg (2007)CrossRefGoogle Scholar
  71. 71.
    Lindner, M., Vicente, R., Priesemann, V., Wibral, M.: Trentool: A Matlab open source toolbox to analyse information flow in time series data with transfer entropy. BMC Neurosci. 12(119), 1–22 (2011)Google Scholar
  72. 72.
    Lindner, M., Vicente, R., Wibral, M., Pampu, N., Wollstadt, P., Martinez-Zarzuela, M.: TRENTOOL, http://www.trentool.de
  73. 73.
  74. 74.
    Lizier, J.: Java Information Dynamics Toolkit, http://code.google.com/p/information-dynamics-toolkit/
  75. 75.
    Faes, L., Nollo, G., Porta, A.: Non-uniform multivariate embedding to assess the information transfer in cardiovascular and cardiorespiratory variability series. Comput. Biol. Med. 42(3), 290–297 (2012)CrossRefGoogle Scholar
  76. 76.
    Lizier, J.T., Rubinov, M.: Inferring effective computational connectivity using incrementally conditioned multivariate transfer entropy. BMC Neuroscience 14(suppl. 1), P337 (2013)Google Scholar
  77. 77.
    Wiener, N.: Cybernetics. Hermann, Paris (1948)Google Scholar
  78. 78.
    Davies, P.C.W., Gregersen, N.H.: Information and the Nature of Reality, vol. 3. Cambridge University Press, Cambridge (2010)CrossRefGoogle Scholar
  79. 79.
    Barnett, L., Lizier, J.T., Harré, M., Seth, A.K., Bossomaier, T.: Information flow in a kinetic Ising model peaks in the disordered phase. Physical Review Letters 111(17), 177203 (2013)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2014

Authors and Affiliations

  1. 1.Max-Planck Institute for Brain ResearchFrankfurt am MainGermany
  2. 2.MEG Unit, Brain Imaging CenterGoethe UniversityFrankfurt am MainGermany

Personalised recommendations