Journal of Computational Neuroscience

, Volume 36, Issue 2, pp 119–140 | Cite as

Synergy, redundancy, and multivariate information measures: an experimentalist’s perspective

  • Nicholas Timme
  • Wesley Alford
  • Benjamin Flecker
  • John M. Beggs
Article

Abstract

Information theory has long been used to quantify interactions between two variables. With the rise of complex systems research, multivariate information measures have been increasingly used to investigate interactions between groups of three or more variables, often with an emphasis on so called synergistic and redundant interactions. While bivariate information measures are commonly agreed upon, the multivariate information measures in use today have been developed by many different groups, and differ in subtle, yet significant ways. Here, we will review these multivariate information measures with special emphasis paid to their relationship to synergy and redundancy, as well as examine the differences between these measures by applying them to several simple model systems. In addition to these systems, we will illustrate the usefulness of the information measures by analyzing neural spiking data from a dissociated culture through early stages of its development. Our aim is that this work will aid other researchers as they seek the best multivariate information measure for their specific research goals and system. Finally, we have made software available online which allows the user to calculate all of the information measures discussedwithin this paper.

Keywords

Information theory Multivariate information measures Complex systems Neural coding Dissociated neuronal cultures Multielectrode array 

References

  1. Abdallah, S.A., & Plumbley, M.D. (2010). A measure of statistical complexity based on predictive information. arXiv:1012.1890v1.
  2. Amari, S.I. (1995). Information geometry of the EM and em algorithms for neural networks. Neural Networks, 8(9), 1379.CrossRefGoogle Scholar
  3. Amari, S. (2001). IEEE Transactions on Information Theory, 47, 1701.CrossRefGoogle Scholar
  4. Anastassiou, D. (2007). Molecular Systems Biology, 3, 83.PubMedCentralPubMedCrossRefGoogle Scholar
  5. Averbeck, B.B., Latham, P.E., Pouget, A. (2006). Nature Reviews Neuroscience, 7, 358.PubMedCrossRefGoogle Scholar
  6. Beggs, J.M., & Plenz, D. (2004). Neuronal avalanches are diverse and precise activity patterns that are stable for many hours in cortical slice cultures. Journal of Neuroscience, 24(22), 5216.PubMedCrossRefGoogle Scholar
  7. Bell, A.J. (2003). International workshop on independent component analysis and blind signal separation, (p. 921).Google Scholar
  8. Berrou, C., Glavieux, A., Thitimajshima, P. (1993). In Proceedings of IEEE International Conference on Communications (Vol. 2, p. 1064).Google Scholar
  9. Bettencourt, L.M.A., Stephens, G.J., Ham, M.I., Gross, G.W. (2007). Physical Review E, 75, 021915.CrossRefGoogle Scholar
  10. Bettencourt, L.M.A., Gintautas, V., Ham, M.I. (2008). Physical Review Letters, 100, 238701.PubMedCrossRefGoogle Scholar
  11. Bialek, W., Rieke, F., de Ruyter van Steveninck, R.R., Warland, D. (1991). Science, 252, 1854.PubMedCrossRefGoogle Scholar
  12. Borst, A., & Theunissen, F.E. (1999). Nature Neuorscience, 2, 947.CrossRefGoogle Scholar
  13. Brenner, N., Strong, S.P., Koberle, R., Bialek, W., de Ruyter van Steveninck, R.R. (2000). Neural Computation, 12, 1531.PubMedCrossRefGoogle Scholar
  14. Butte, A.J., & Kohane, I.S. (2000). In Pacific Symposium on Biocomputing (Vol. 5, p. 415).Google Scholar
  15. Butts, D.A., & Rokhsar, D.S. (2001). Journal of Neuroscience, 21, 961.PubMedGoogle Scholar
  16. Butts, D.A., Weng, C., Jin, J., Yeh, C.I., Lesica, N.A., Alonso, J.M., Stanley, G.B. (2007). Nature Letters, 449, 92.CrossRefGoogle Scholar
  17. Cerf, N.J., & Adami, C. (1997). Physical Review A, 55, 3371.CrossRefGoogle Scholar
  18. Chanda, P., Zhang, A., Brazeau, D., Sucheston, L., Freudenheim, J.L., Ambrosone, C., Ramanathan, M. (2007). American Journal of Human Genetics, 81, 939.PubMedCentralPubMedCrossRefGoogle Scholar
  19. Chechik, G., Globerson, A., Tishby, N., Anderson, M.J., Young, E.D., Nelken, I. (2001). In T.G. Dietterich, S. Becker, Z. Ghahramani (Eds.), Neaural information processing systems 14 (Vol. 1, p. 173) MIT Press.Google Scholar
  20. Cover, T.M., & Thomas, J.A. (2006). Elements of information theory, 2nd edn. Wiley-Interscience.Google Scholar
  21. DeWeese, M.R., & Meister, M. (1999). Network: Computation in Neural Systems, 10, 325.CrossRefGoogle Scholar
  22. Fairhall, A., Shea-Brown, E., Barreiro, A. (2012). Current Opinion in Neurobiology, 22, 653.PubMedCrossRefGoogle Scholar
  23. Flecker, B., Alford, W., Beggs, J.M., Williams, P.L., Beer, R.D. (2011). Chaos, 21, 037104.PubMedCrossRefGoogle Scholar
  24. Fraser, A.M., & Swinney, H.L. (1986). Phys. Rev. A, 33, 1134.PubMedCrossRefGoogle Scholar
  25. Fujisawa, S., Amarasingham, A., Harrison, M.T., G. Buzsáki (2008). Nature Neuroscience, 11, 823.PubMedCentralPubMedCrossRefGoogle Scholar
  26. Garofalo, M., Nieus, T., Massobrio, P., Martinoia, S. (2009). PLoS One, 4, e6482.PubMedCentralPubMedCrossRefGoogle Scholar
  27. Gat, I., & Tishby, N. (1999). In M.S. Kearns, S.A. Solla, D.A. Cohn (Eds.), Neural information processing systems 11 (p. 111). MIT Press.Google Scholar
  28. Globerson, A., Stark, E., Vaadia, E., Tishby, N. (2009). PNAS, 106, 3490.PubMedCentralPubMedCrossRefGoogle Scholar
  29. Gollisch, T., & Meister, M. (2008). Science, 319, 1108.PubMedCrossRefGoogle Scholar
  30. Griffith, V., & Koch, C. (2012). Quantifying synergistic mutual information. arXiv:12054265v2.
  31. Han, T.S. (1975). Information and Control, 29, 337.CrossRefGoogle Scholar
  32. Han, T.S. (1978). Information and Control, 36, 133.CrossRefGoogle Scholar
  33. Hatsopoulos, N., Geman, S., Amarasingham, A., Bienenstock, E. (2003). Neurocomputing, 52, 25.CrossRefGoogle Scholar
  34. Hlaváčková-Schindler, K., Paluš, M., Vejmelka, M., Bhattacharya, J. (2007). Physics Reports, 441, 1.CrossRefGoogle Scholar
  35. Honey, C.J., Kotter, R., Breakspear, M., Sporns, O. (2007). PNAS, 104, 10240.PubMedCentralPubMedCrossRefGoogle Scholar
  36. Ikegaya, Y., Aaron, G., Cossart, R., Aronov, D., Lampl, I., Ferster, D., Yuste, R. (2004). Science, 304, 559.PubMedCrossRefGoogle Scholar
  37. Ito, S., Hansen, M.E., Heiland, R., Lumsdaine, A., Litke, A.M., Beggs, J.M. (2011). PLoS One, 6(21), e27431.PubMedCentralPubMedCrossRefGoogle Scholar
  38. Jakulin, A., & Bratko, I. (2008). Quantifying and visualizing attribute interactions. arXiv:cs/0308002v3.
  39. James, R.G., Ellison, C.J., Crutchfield, J.P. (2011). Chaos, 21, 037109.PubMedCrossRefGoogle Scholar
  40. Kamioka, H., Maeda, E., Jimbo, Y., Robinson, H.P.C., Kawana, A. (1996). Neuroscience Letters, 206, 109.PubMedCrossRefGoogle Scholar
  41. Kennel, M.B., Shlens, J., Abarbanel, H.D.I., Chichilnisky, E.J. (2005). Neural Computation, 17, 1531.PubMedCrossRefGoogle Scholar
  42. Latham, P.E., & Nirenberg, S. (2005). Journal of Neuroscience, 25, 5195.PubMedCrossRefGoogle Scholar
  43. Lizier, J.T., Heinzle, J., Horstmann, A., Haynes, J.D., Prokopenko, M. (2011). Journal of Computational Neuroscience, 30, 85.PubMedCrossRefGoogle Scholar
  44. Lizier, J.T., Flecker, B., Williams, P.L. (2013). Towards a synergy-based approach to measuring information modification. arXiv:1303.3440.
  45. Louie, K., & Wilson, M.A. (2001). Neuron, 29, 145.PubMedCrossRefGoogle Scholar
  46. Lungarella, M., & Sporn, O. (2006). PLoS One, 2, e144.Google Scholar
  47. Madhavan, R., Chao, Z.C., Potter, S.M. (2007). Physical Biology, 4, 181.PubMedCentralPubMedCrossRefGoogle Scholar
  48. Marschinski, R., & Kantz, H. (2002). European Physical Journal B, 30, 275.CrossRefGoogle Scholar
  49. Martignon, L., Deco, G., Laskey, K., Diamond, M., Freiwald, W., Vaadia, E. (2000). Neural Computation, 12, 2621.PubMedCrossRefGoogle Scholar
  50. Matsuda, H. (2000). Physical Review E, 62, 3096.CrossRefGoogle Scholar
  51. McGill, W.J. (1954). Psychometrika, 19, 97.CrossRefGoogle Scholar
  52. Nemenman, I., Bialek, W., de Ruyter van Steveninck, R.R. (2004). Physical Review E, 69, 056111.CrossRefGoogle Scholar
  53. Nirenberg, S., Carcieri, S.M., Jacobs, A.L., Latham, P.E. (2001). Nature, 411, 698.PubMedCrossRefGoogle Scholar
  54. Ohiorhenuan, I.E., & Victor, J.D. (2011). Journal of Computational Neuroscience, 30, 125.PubMedCentralPubMedCrossRefGoogle Scholar
  55. Ohiorhenuan, I.E., Mechlar, F., Purpura, K.P., Schmid, A.M., Hiu, Q., Victor, J.D. (2010). Nature Letters, 466, 617.CrossRefGoogle Scholar
  56. Olbrich, E., Bertschinger, N., Ay, N., Jost, J. (2008). European Physical Journal B, 63, 407.CrossRefGoogle Scholar
  57. Optican, L.M., & Richmond, B.J. (1987). Journal of Neurophysiology, 57, 162.PubMedGoogle Scholar
  58. Paiva, A.R.C., Park, I., Principe, J.C. (2010). Neural Computation and Application, 19, 405.CrossRefGoogle Scholar
  59. Paninski, L. (2003). Neural Computation, 15, 1191.CrossRefGoogle Scholar
  60. Panzeri, S., & Treves, A. (1996). Network: Computation in Neural Systems, 7, 87.CrossRefGoogle Scholar
  61. Panzeri, S., Petersen, R.S., Schultz, S.R., Lebedev, M., Diamond, M.E. (2001). Neuron, 29, 769.PubMedCrossRefGoogle Scholar
  62. Panzeri, S., Senatore, R., Montemurro, M.A., Petersen, R.S. (2007). Journal of Neurophysiology, 98, 1064.PubMedCrossRefGoogle Scholar
  63. Pasquale, V., Massobrio, P., Bologna, L.L., Chiappalonea, M., Martinoia, S. (2008). Neuroscience, 153, 1354.PubMedCrossRefGoogle Scholar
  64. Pazienti, A., Maldonado, P.E., Diesmann, M., Grun, S. (2008). Brain Research, 1225, 39.PubMedCrossRefGoogle Scholar
  65. Pillow, J.W., Shlens, J., Paninski, L., Sher, A., Litke, A.M., Chichilnisky, E.J., Simoncelli, E.P. (2008). Nature, 454, 995.PubMedCentralPubMedCrossRefGoogle Scholar
  66. Quiroga, R.Q., & Panzeri, S. (2013). Nature Reviews Neuroscience, 10, 173.CrossRefGoogle Scholar
  67. Quiroga R.Q., & Panzeri S.(Eds.) (2013). Principles of Neural Coding. CRC Press LLC.Google Scholar
  68. Rieke, F., Warland, D., de Ruyter van Steveninck, R.R., Bialek, W. (1997). Spikes: exploring the neural code. MIT Press.Google Scholar
  69. Rivlin-Etzion, M., Ritov, Y., Heimer, G., Bergman, H., Bar-Gad, I. (2006). Journal of Neurophysiology, 95, 3245.PubMedCrossRefGoogle Scholar
  70. Rokem, A., Watzl, S., Gollisch, T., Stemmler, M., Herz, A.V.M., Samengo, I. (2006). Journal of Neurophysiology, 95, 2541.PubMedCrossRefGoogle Scholar
  71. Rolston, J.D., Wagenaar, D.A., Potter, S.M. (2007). Neuroscience, 148, 294.PubMedCentralPubMedCrossRefGoogle Scholar
  72. Schreiber, T. (2000). Physical Review Letters, 85, 461.PubMedCrossRefGoogle Scholar
  73. Schneidman, E., Bialek, W., Berry II, M.J. (2003a). Journal of Neuroscience, 23, 11539.PubMedGoogle Scholar
  74. Schneidman, E., Still, S., Berry II, M.J., Bialek, W. (2003b). Physical Review Letters, 91, 238701.PubMedCrossRefGoogle Scholar
  75. Schneidman, E., Berry II, M.J., Segev, R., Bialek, W. (2006). Nature, 440, 1007.PubMedCentralPubMedCrossRefGoogle Scholar
  76. Shannon, C.E. (1948). The Bell System Technical Journal, 27, 379.CrossRefGoogle Scholar
  77. Shimazaki, H., Amari, S., Brown, E.N., Grun, S. (2012). PLoS Computational Biology, 8(3), e1002385.PubMedCentralPubMedCrossRefGoogle Scholar
  78. Shlens, J., Field, G.D., Gauthier, J.L., Grivich, M.I., Petrusca, D., Sher, A., Litke, A.M., Chichilnisky, E.J. (2006). Journal of Neuroscience, 26, 8254.PubMedCrossRefGoogle Scholar
  79. Shlens, J., Kennel, M.B., Abarbanel, H.D.I., Chichilnisky, E.J. (2007). Neural Computation, 19, 1683.PubMedCrossRefGoogle Scholar
  80. Sporns, O., Tononi, G., Edelman, G.E. (2000). Cerebral Cortex, 10, 127.PubMedCrossRefGoogle Scholar
  81. Strong, S.P., Koberle, R., de Ruyter van Steveninck, R.R., Bialek, W. (1997). Physical Review Letters, 80, 197.CrossRefGoogle Scholar
  82. Tang, A., Jackson, D., Hobss, J., Chen, W., Smith, J.L., Patel, H., Prieto, A., Petrusca, D., Grivich, M.I., Sher, A., Hottowy, P., Dabrowski, W., Litke, A.M., Beggs, J.M. (2008). Journal of Neuroscience, 28, 505.PubMedCrossRefGoogle Scholar
  83. Tetzlaff, C., Okujeni, S., Egert, U., Worgotter, F., Butz, M. (2010). PLoS Computational Biology, 6, e1001013.PubMedCentralPubMedCrossRefGoogle Scholar
  84. Timme, N., Alford, W., Flecker, B., Beggs, J.M. (2011). Multivariate information measures: an experimentalist’s perspective. arXiv:1111.6857.
  85. Tononi, G., Sporns, O., Edelman, G.M. (1994). Proceedings of the National Academy of Sciences, 91, 5033.CrossRefGoogle Scholar
  86. Treves, A., & Panzeri, S. (1995). Neural Computation, 7, 399.CrossRefGoogle Scholar
  87. Varadan, V., Miller III, D.M., Anastassiou, D. (2006). Bioinformatics, 22, e497.PubMedCrossRefGoogle Scholar
  88. Vicente, R., Wibral, M., Lindner, M., Pipa, G. (2011). Journal of Computational Neuroscience, 30, 45.PubMedCentralPubMedCrossRefGoogle Scholar
  89. Victor, J.D. (2002). Physical Review E, 66, 051902.CrossRefGoogle Scholar
  90. Victor, J.D. (2006). Biological Theory, 1, 302.PubMedCentralPubMedCrossRefGoogle Scholar
  91. Wagenaar, D.A., Pine, J., Potter, S.M. (2006a). BMC Neuroscience, 7.Google Scholar
  92. Wagenaar, D.A., Nadasdy, Z., Potter, S.M. (2006b). Physical Review E, 73, 051907.CrossRefGoogle Scholar
  93. Wang, L., Narayan, R., na, G.G., Shamir, M., Sen, K. (2007). Journal of Neuroscience, 27(3), 582.PubMedCrossRefGoogle Scholar
  94. Warland, D.K., Reinagel, P., Meister, M. (1997). Journal of Neurophysiology, 78, 2336.PubMedGoogle Scholar
  95. Watanabe, S. (1960). IBM Journal of Research and Development, 4, 66.CrossRefGoogle Scholar
  96. Wennekers, T., & Ay, N. (2003). Theory in Bioscience, 122, 5.CrossRefGoogle Scholar
  97. Williams, P.L., & Beer, R.D. (2010). Decomposing multivariate information. arXiv:1004.2515v1.
  98. Williams, P.L., & Beer, R.D. (2011). Generalized measures of information transfer. arXiv:1102.1507v1.
  99. Yeh, F.C., Tang, A., Hobbs, J.P., Hottowy, P., Dabrowski, W., Sher, A., Litke, A., Beggs, J.M. (2010). Entropy, 12, 89.CrossRefGoogle Scholar
  100. Yu, S., Yang, H., Nakahara, H., Santos, G.S., Nikolic, D., Plenz, D. (2011). Higher-order interactions characterized in cortical activity. Journal of Neuroscience, 31, 17514.PubMedCrossRefGoogle Scholar
  101. Ziv, J., & Lempel, A. (1977). IEEE Transactions on Information Theory, 23, 337.CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media New York 2013

Authors and Affiliations

  • Nicholas Timme
    • 1
  • Wesley Alford
    • 1
  • Benjamin Flecker
    • 1
  • John M. Beggs
    • 1
  1. 1.Department of PhysicsIndiana University,BloomingtonUSA

Personalised recommendations