Transfer Entropy in Neuroscience

Abstract

Information transfer is a key component of information processing, next to information storage and modification. Information transfer can be measured by a variety of directed informationmeasures of which transfer entropy is themost popular, andmost principled one. This chapter presents the basic concepts behind transfer entropy in an intuitive fashion, including graphical depictions of the key concepts. It also includes a special section devoted to the correct interpretation of the measure, especially with respect to concepts of causality. The chapter also provides an overview of estimation techniques for transfer entropy and pointers to popular open source toolboxes. It also introduces recent extensions of transfer entropy that serve to estimate delays involved in information transfer in a network. By touching upon alternative measures of information transfer, such as Massey’s directed information transfer and Runge’s momentary information transfer, it may serve as a frame of reference for more specialised treatments and as an overview over the field of studies in information transfer in general.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Amblard, P.O., Michel, O.J.J.: On directed information theory and granger causality graphs. J. Comput. Neurosci. 30(1), 7–16 (2011)CrossRefMathSciNetGoogle Scholar
  2. 2.
    Amblard, P.O., Michel, O.J.J.: The relation between granger causality and directed information theory: A review. Entropy 15(1), 113–143 (2012)CrossRefMathSciNetGoogle Scholar
  3. 3.
    Ay, N., Polani, D.: Information flows in causal networks. Adv. Complex Syst. 11, 17 (2008)CrossRefMATHMathSciNetGoogle Scholar
  4. 4.
    Barnett, L., Barrett, A.B., Seth, A.K.: Granger causality and transfer entropy are equivalent for Gaussian variables. Phys. Rev. Lett. 103(23), 238–701 (2009)CrossRefGoogle Scholar
  5. 5.
    Battaglia, D., Witt, A., Wolf, F., Geisel, T.: Dynamic effective connectivity of inter-areal brain circuits. PLoS Comput. Biol. 8(3), e1002438 (2012)Google Scholar
  6. 6.
    Besserve, M., Schlkopf, B., Logothetis, N.K., Panzeri, S.: Causal relationships between frequency bands of extracellular signals in visual cortex revealed by an information theoretic analysis. J. Comput. Neurosci. 29(3), 547–566 (2010)CrossRefGoogle Scholar
  7. 7.
    Bühlmann, A., Deco, G.: Optimal information transfer in the cortex through synchronization. PLoS Comput. Biol. 6(9), e1000934 (2010)Google Scholar
  8. 8.
    Chávez, M., Martinerie, J., Le Van Quyen, M.: Statistical assessment of nonlinear causality: application to epileptic EEG signals. J. Neurosci. Methods 124(2), 113–128 (2003)CrossRefGoogle Scholar
  9. 9.
    Chicharro, D., Ledberg, A.: When two become one: the limits of causality analysis of brain dynamics. PLoS One 7(3), e32466 (2012)Google Scholar
  10. 10.
    Cover, T.M., Thomas, J.A.: Elements of information theory. Wiley-Interscience, New York (1991)CrossRefMATHGoogle Scholar
  11. 11.
    Faes, L., Nollo, G.: Bivariate nonlinear prediction to quantify the strength of complex dynamical interactions in short-term cardiovascular variability. Med. Biol. Eng. Comput. 44(5), 383–392 (2006)CrossRefGoogle Scholar
  12. 12.
    Faes, L., Nollo, G., Porta, A.: Information-based detection of nonlinear granger causality in multivariate processes via a nonuniform embedding technique. Phys. Rev. E Stat. Nonlin. Soft. Matter Phys. 83(5 Pt. 1), 051112 (2011)Google Scholar
  13. 13.
    Faes, L., Nollo, G., Porta, A.: Non-uniform multivariate embedding to assess the information transfer in cardiovascular and cardiorespiratory variability series. Comput. Biol. Med. 42(3), 290–297 (2012)CrossRefGoogle Scholar
  14. 14.
    Faes, L., Nollo, G., Porta, A.: Compensated transfer entropy as a tool for reliably estimating information transfer in physiological time series. Entropy 15(1), 198–219 (2013)CrossRefMathSciNetGoogle Scholar
  15. 15.
    Felts, P.A., Baker, T.A., Smith, K.J.: Conduction in segmentally demyelinated mammalian central axons. J. Neurosci. 17(19), 7267–7277 (1997)Google Scholar
  16. 16.
    Freiwald, W.A., Valdes, P., Bosch, J., Biscay, R., Jimenez, J.C., Rodriguez, L.M., Rodriguez, V., Kreiter, A.K., Singer, W.: Testing non-linearity and directedness of interactions between neural groups in the macaque inferotemporal cortex. J. Neurosci. Methods 94(1), 105–119 (1999)CrossRefGoogle Scholar
  17. 17.
    Garofalo, M., Nieus, T., Massobrio, P., Martinoia, S.: Evaluation of the performance of information theory-based methods and cross-correlation to estimate the functional connectivity in cortical networks. PLoS One 4(8), e6482 (2009)Google Scholar
  18. 18.
    Gomez-Herrero, G., Wu, W., Rutanen, K., Soriano, M.C., Pipa, G., Vicente, R.: Assessing coupling dynamics from an ensemble of time series. arXiv preprint arXiv:1008.0539 (2010)Google Scholar
  19. 19.
    Gourevitch, B., Eggermont, J.J.: Evaluating information transfer between auditory cortical neurons. J. Neurophysiol. 97(3), 2533–2543 (2007)CrossRefGoogle Scholar
  20. 20.
    Gray, C.M., Knig, P., Engel, A.K., Singer, W.: Oscillatory responses in cat visual cortex exhibit inter-columnar synchronization which reflects global stimulus properties. Nature 338(6213), 334–337 (1989)CrossRefGoogle Scholar
  21. 21.
    Griffith, V., Koch, C.: Quantifying synergistic mutual information. In: Prokopenko, M. (ed.) Guided Self-Organization: Inception, pp. 159–190. Springer, Heidelberg (2014)CrossRefGoogle Scholar
  22. 22.
    Hadjipapas, A., Hillebrand, A., Holliday, I.E., Singh, K.D., Barnes, G.R.: Assessing interactions of linear and nonlinear neuronal sources using MEG beamformers: a proof of concept. Clin. Neurophysiol. 116(6), 1300–1313 (2005)CrossRefGoogle Scholar
  23. 23.
    Hahs, D.W., Pethel, S.D.: Distinguishing anticipation from causality: anticipatory bias in the estimation of information flow. Phys. Rev. Lett. 107(12), 128701 (2011)CrossRefGoogle Scholar
  24. 24.
    Hahs, D.W., Pethel, S.D.: Transfer entropy for coupled autoregressive processes. Entropy 15(3), 767–788 (2013)CrossRefMathSciNetGoogle Scholar
  25. 25.
    Harder, M., Salge, C., Polani, D.: Bivariate measure of redundant information. Phys. Rev. E Stat. Nonlin. Soft Matter Phys. 87(1), 012130 (2013)Google Scholar
  26. 26.
    Hebb, D.O.: The organization of behavior: A neuropsychological theory. Wiley, New York (1949)Google Scholar
  27. 27.
    Ito, S., Hansen, M.E., Heiland, R., Lumsdaine, A., Litke, A.M., Beggs, J.M.: Extending transfer entropy improves identification of effective connectivity in a spiking cortical network model. PLoS One 6(11), e27431 (2011)Google Scholar
  28. 28.
    Kaiser, A., Schreiber, T.: Information transfer in continuous processes. Physica D 166, 43 (2002)CrossRefMATHMathSciNetGoogle Scholar
  29. 29.
    Kim, J., Kim, G., An, S., Kwon, Y.K., Yoon, S.: Entropy-based analysis and bioinformatics-inspired integration of global economic information transfer. PLoS One 8(1), e51986 (2013)Google Scholar
  30. 30.
    Kozachenko, L., Leonenko, N.: Sample estimate of entropy of a random vector. Probl. Inform. Transm. 23, 95–100 (1987)MATHMathSciNetGoogle Scholar
  31. 31.
    Kraskov, A., Stoegbauer, H., Grassberger, P.: Estimating mutual information. Phys. Rev. E Stat. Nonlin. Soft Matter Phys. 69(6 Pt. 2), 066138 (2004)Google Scholar
  32. 32.
    Kwon, O., Yang, J.S.: Information flow between stock indices. EPL (Europhysics Letters) 82(6), 68003 (2008)CrossRefGoogle Scholar
  33. 33.
    Lapidoth, A., Pete, G.: On the entropy of the sum and of the difference of independent random variables. In: IEEE 25th Convention of Electrical and Electronics Engineers in Israel, IEEEI 2008, pp. 623–625. IEEE (2008)Google Scholar
  34. 34.
    Leistritz, L., Hesse, W., Arnold, M., Witte, H.: Development of interaction measures based on adaptive non-linear time series analysis of biomedical signals. Biomed. Tech (Berl.) 51(2), 64–69 (2006)CrossRefGoogle Scholar
  35. 35.
    Li, X., Ouyang, G.: Estimating coupling direction between neuronal populations with permutation conditional mutual information. NeuroImage 52(2), 497–507 (2010)CrossRefGoogle Scholar
  36. 36.
    Lindner, M., Vicente, R., Priesemann, V., Wibral, M.: Trentool: A Matlab open source toolbox to analyse information flow in time series data with transfer entropy. BMC Neurosci. 12(119), 1–22 (2011)Google Scholar
  37. 37.
    Lizier, J.: The Local Information Dynamics of Distributed Computation in Complex Systems. Springer theses. Springer (2013)Google Scholar
  38. 38.
    Lizier, J.T., Atay, F.M., Jost, J.: Information storage, loop motifs, and clustered structure in complex networks. Phys. Rev. E Stat. Nonlin. Soft Matter Phys. 86(2 Pt. 2), 026110 (2012)Google Scholar
  39. 39.
    Lizier, J.T., Flecker, B., Williams, P.L.: Towards a synergy-based approach to measuring information modification. In: Proceedings of the 2013 IEEE Symposium on Artificial Life (ALIFE), pp. 43–51. IEEE (2013)Google Scholar
  40. 40.
    Lizier, J.T., Heinzle, J., Horstmann, A., Haynes, J.D., Prokopenko, M.: Multivariate information-theoretic measures reveal directed information structure and task relevant changes in fMRI connectivity. J. Comput. Neurosci. 30(1), 85–107 (2011)CrossRefMathSciNetGoogle Scholar
  41. 41.
    Lizier, J.T., Mahoney, J.R.: Moving frames of reference, relativity and invariance in transfer entropy and information dynamics. Entropy 15(1), 177–197 (2013)CrossRefMathSciNetGoogle Scholar
  42. 42.
    Lizier, J.T., Pritam, S., Prokopenko, M.: Information dynamics in small-world Boolean networks. Artif. Life 17(4), 293–314 (2011)CrossRefGoogle Scholar
  43. 43.
    Lizier, J.T., Prokopenko, M.: Differentiating information transfer and causal effect. Eur. Phys. J. B 73, 605–615 (2010)CrossRefGoogle Scholar
  44. 44.
    Lizier, J.T., Prokopenko, M., Zomaya, A.Y.: Local information transfer as a spatiotemporal filter for complex systems. Phys. Rev. E 77(2 Pt. 2), 026110 (2008)Google Scholar
  45. 45.
    Lizier, J.T., Prokopenko, M., Zomaya, A.Y.: Information modification and particle collisions in distributed computation. Chaos 20(3), 037109 (2010)Google Scholar
  46. 46.
    Lizier, J.T., Rubinov, M.: Multivariate construction of effective computational networks from observational data. Max Planck Preprint 25/2012. Max Planck Institute for Mathematics in the Sciences (2012)Google Scholar
  47. 47.
    Lüdtke, N., Logothetis, N.K., Panzeri, S.: Testing methodologies for the nonlinear analysis of causal relationships in neurovascular coupling. Magn. Reson. Imaging 28(8), 1113–1119 (2010)CrossRefGoogle Scholar
  48. 48.
    Marko, H.: The bidirectional communication theory–a generalization of information theory. IEEE Transactions on Communications 21(12), 1345–1351 (1973)CrossRefGoogle Scholar
  49. 49.
    Marr, D.: Vision: A Computational Investigation into the Human Representation and Processing of Visual Information. Henry Holt and Co. Inc., New York (1982)Google Scholar
  50. 50.
    Massey, J.: Causality, feedback and directed information. In: Proc. Int. Symp. Information Theory Application (ISITA 1990), pp. 303–305 (1990)Google Scholar
  51. 51.
    Merkwirth, C., Parlitz, U., Lauterborn, W.: Fast nearest-neighbor searching for nonlinear signal processing. Phys. Rev. E Stat. Phys. Plasmas. Fluids Relat. Interdiscip. Topics 62(2 Pt. A), 2089–2097 (2000)Google Scholar
  52. 52.
    Neymotin, S.A., Jacobs, K.M., Fenton, A.A., Lytton, W.W.: Synaptic information transfer in computer models of neocortical columns. J. Comput. Neurosci. 30(1), 69–84 (2011)CrossRefMathSciNetGoogle Scholar
  53. 53.
    Nolte, G., Ziehe, A., Nikulin, V.V., Schlogl, A., Kramer, N., Brismar, T., Muller, K.R.: Robustly estimating the flow direction of information in complex physical systems. Phys. Rev. Lett. 100(23), 234101 (2008)CrossRefGoogle Scholar
  54. 54.
    Oostenveld, R., Fries, P., Maris, E., Schoffelen, J.M.: Fieldtrip: Open source software for advanced analysis of MEG, EEG, and invasive electrophysiological data. Comput. Intell. Neurosci. 2011, 156869 (2011)CrossRefGoogle Scholar
  55. 55.
    Paluš, M.: Synchronization as adjustment of information rates: detection from bivariate time series. Phys. Rev. E 63, 046211 (2001)Google Scholar
  56. 56.
    Pearl, J.: Causality: models, reasoning, and inference. Cambridge University Press (2000)Google Scholar
  57. 57.
    Pompe, B., Runge, J.: Momentary information transfer as a coupling measure of time series. Phys. Rev. E Stat. Nonlin. Soft Matter Phys. 83(5 Pt. 1), 051122 (2011)Google Scholar
  58. 58.
    Ragwitz, M., Kantz, H.: Markov models from data by simple nonlinear time series predictors in delay embedding spaces. Phys. Rev. E Stat. Nonlin. Soft Matter Phys. 65(5 Pt. 2), 056201 (2002)Google Scholar
  59. 59.
    Sabesan, S., Good, L.B., Tsakalis, K.S., Spanias, A., Treiman, D.M., Iasemidis, L.D.: Information flow and application to epileptogenic focus localization from intracranial EEG. IEEE Trans. Neural. Syst. Rehabil. Eng. 17(3), 244–253 (2009)CrossRefGoogle Scholar
  60. 60.
    Schreiber, T.: Measuring information transfer. Phys. Rev. Lett. 85(2), 461–464 (2000)CrossRefGoogle Scholar
  61. 61.
    Small, M., Tse, C.: Optimal embedding parameters: a modelling paradigm. Physica D: Nonlinear Phenomena 194, 283–296 (2004)CrossRefMATHMathSciNetGoogle Scholar
  62. 62.
    Staniek, M., Lehnertz, K.: Symbolic transfer entropy: inferring directionality in biosignals. Biomed. Tech (Berl.) 54(6), 323–328 (2009)CrossRefGoogle Scholar
  63. 63.
    Stetter, O., Battaglia, D., Soriano, J., Geisel, T.: Model-free reconstruction of excitatory neuronal connectivity from calcium imaging signals. PLoS Comput. Biol. 8(8), e1002653 (2012)Google Scholar
  64. 64.
    Sun, L., Grützner, C., Bölte, S., Wibral, M., Tozman, T., Schlitt, S., Poustka, F., Singer, W., Freitag, C.M., Uhlhaas, P.J.: Impaired gamma-band activity during perceptual organization in adults with autism spectrum disorders: evidence for dysfunctional network activity in frontal-posterior cortices. J. Neurosci. 32(28), 9563–9573 (2012)CrossRefGoogle Scholar
  65. 65.
    Takens, F.: Detecting Strange Attractors in Turbulence. In: Dynamical Systems and Turbulence, Warwick. Lecture Notes in Mathematics, vol. 898, pp. 366–381. Springer (1980)Google Scholar
  66. 66.
    Vakorin, V.A., Kovacevic, N., McIntosh, A.R.: Exploring transient transfer entropy based on a group-wise ica decomposition of EEG data. Neuroimage 49(2), 1593–1600 (2010)CrossRefGoogle Scholar
  67. 67.
    Vakorin, V.A., Krakovska, O.A., McIntosh, A.R.: Confounding effects of indirect connections on causality estimation. J. Neurosci. Methods 184(1), 152–160 (2009)CrossRefGoogle Scholar
  68. 68.
    Vakorin, V.A., Mii, B., Krakovska, O., McIntosh, A.R.: Empirical and theoretical aspects of generation and transfer of information in a neuromagnetic source network. Front Syst. Neurosci. 5, 96 (2011)CrossRefGoogle Scholar
  69. 69.
    Vicente, R., Wibral, M., Lindner, M., Pipa, G.: Transfer entropy – a model-free measure of effective connectivity for the neurosciences. J. Comput. Neurosci. 30(1), 45–67 (2011)CrossRefMathSciNetGoogle Scholar
  70. 70.
    Victor, J.: Binless strategies for estimation of information from neural data. Phys. Rev. E 72, 051903 (2005)Google Scholar
  71. 71.
    Whitford, T.J., Ford, J.M., Mathalon, D.H., Kubicki, M., Shenton, M.E.: Schizophrenia, myelination, and delayed corollary discharges: a hypothesis. Schizophr Bull. 38(3), 486–494 (2012)Google Scholar
  72. 72.
    Wibral, M., Pampu, N., Priesemann, V., Siebenhhner, F., Seiwert, H., Lindner, M., Lizier, J.T., Vicente, R.: Measuring information-transfer delays. PLoS One 8(2), e55809 (2013)Google Scholar
  73. 73.
    Wibral, M., Rahm, B., Rieder, M., Lindner, M., Vicente, R., Kaiser, J.: Transfer entropy in magnetoencephalographic data: Quantifying information flow in cortical and cerebellar networks. Prog. Biophys. Mol. Biol. 105(1-2), 80–97 (2011)CrossRefGoogle Scholar
  74. 74.
    Wibral, M., Wollstadt, P., Meyer, U., Pampu, N., Priesemann, V., Vicente, R.: Revisiting wiener’s principle of causality – interaction-delay reconstruction using transfer entropy and multivariate analysis on delay-weighted graphs. Conf. Proc. IEEE Eng. Med. Biol. Soc. 2012, 3676–3679 (2012)Google Scholar
  75. 75.
    Wollstadt, P., Martinéz-Zarzuela, M., Vicente, R., Díaz-Pernas, F., Wibral, M.: Efficient transfer entropy analysis of non-stationary neural time series. arXiv preprint arXiv:1401.4068 (2014)Google Scholar
  76. 76.
    Wiener, N.: The theory of prediction. In: Beckmann, E.F. (ed.) In Modern Mathematics for the Engineer. McGraw-Hill, New York (1956)Google Scholar
  77. 77.
    Williams, P.L., Beer, R.D.: Nonnegative decomposition of multivariate information. arXiv preprint arXiv:1004.2515 (2010)Google Scholar
  78. 78.
    Williams, P.L., Beer, R.D.: Generalized measures of information transfer. arXiv preprint arXiv:1102.1507 (2011)Google Scholar
  79. 79.
    Wolfram, S.: A new kind of science. Wolfram Media, Champaign (2002)MATHGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2014

Authors and Affiliations

  1. 1.MEG Unit, Brain Imaging CenterGoethe UniversityFrankfurt am MainGermany
  2. 2.Max-Planck Institute for Brain ResearchFrankfurt am MainGermany
  3. 3.School of Psychology and Clinical Language ScienceUniversity of ReadingReadingUK

Personalised recommendations