A Framework for the Local Information Dynamics of Distributed Computation in Complex Systems

  • Joseph T. Lizier
  • Mikhail Prokopenko
  • Albert Y. Zomaya
Part of the Emergence, Complexity and Computation book series (ECC, volume 9)


The nature of distributed computation has long been a topic of interest in complex systems science, physics, artificial life and bioinformatics. In particular, emergent complex behavior has often been described from the perspective of computation within the system (Mitchell 1998b,a) and has been postulated to be associated with the capability to support universal computation (Langton 1990; Wolfram 1984c; Casti 1991).


Cellular Automaton Information Transfer Cellular Automaton Excess Entropy Transfer Entropy 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. Adamatzky, A. (ed.): Collision-Based Computing. Springer, Berlin (2002)zbMATHGoogle Scholar
  2. Atick, J.J.: Could information theory provide an ecological theory of sensory processing? Network: Computation in Neural Systems 3(2), 213 (1992)zbMATHCrossRefGoogle Scholar
  3. Badii, R., Politi, A.: Thermodynamics and Complexity of Cellular Automata. Physical Review Letters 78(3), 444 (1997)CrossRefGoogle Scholar
  4. Bialek, W., Nemenman, I., Tishby, N.: Complexity through nonextensivity. Physica A: Statistical Mechanics and its Applications 302(1-4), 89–99 (2001)zbMATHMathSciNetCrossRefGoogle Scholar
  5. Boccara, N., Nasser, J., Roger, M.: Particlelike structures and their interactions in spatiotemporal patterns generated by one-dimensional deterministic cellular-Automaton rules. Physical Review A 44(2), 866–875 (1991)CrossRefGoogle Scholar
  6. Boedecker, J., Obst, O., Lizier, J.T., Mayer, N.M., Asada, M.: Information processing in echo state networks at the edge of chaos. Theory in Biosciences 131(3), 205–213 (2012)CrossRefGoogle Scholar
  7. Brown, J.A., Tuszynski, J.A.: A review of the ferroelectric model of microtubules. Ferroelectrics 220, 141–156 (1999)CrossRefGoogle Scholar
  8. Casti, J.L.: Chaos, Gödel and truth. In: Casti, J.L., Karlqvist, A. (eds.) Beyond Belief: Randomness, Prediction and Explanation in Science, pp. 280–327. CRC Press, Boca Raton (1991)Google Scholar
  9. Ceguerra, R.V., Lizier, J.T., Zomaya, A.Y.: Information storage and transfer in the synchronization process in locally-connected networks. In: Proceedings of the 2011 IEEE Symposium on Artificial Life (ALIFE), pp. 54–61. IEEE (2011)Google Scholar
  10. Cliff, O.M., Lizier, J.T., Wang, X.R., Wang, P., Obst, O., Prokopenko, M.: Towards quantifying interaction networks in a football match. In: Proceedings of the RoboCup 2013 Symposium (to be published, 2013)Google Scholar
  11. Conway, J.H.: What is Life? In: Berlekamp, E., Conway, J.H., Guy, R. (eds.) Winning Ways for Your Mathematical Plays, vol. 2, ch. 25, pp. 927–962. Academic Press, New York (1982)Google Scholar
  12. Cook, M.: Universality in Elementary Cellular Automata. Complex Systems 15(1), 1–40 (2004)zbMATHMathSciNetGoogle Scholar
  13. Couzin, I.D., James, R., Croft, D.P., Krause, J.: Social Organization and Information Transfer in Schooling Fishes. In: Brown, C., Laland, K.N., Krause, J. (eds.) Fish Cognition and Behavior, Fish and Aquatic Resources, pp. 166–185. Blackwell Publishing (2006)Google Scholar
  14. Cover, T.M., Thomas, J.A.: Elements of Information Theory. Wiley-Interscience, New York (1991)Google Scholar
  15. Crutchfield, J.P.: Personal communication (2009)Google Scholar
  16. Crutchfield, J.P., Ellison, C.J., Riechers, P.M.: Exact complexity: The spectral decomposition of intrinsic computation. arXiv:1309.3792 (2013)Google Scholar
  17. Crutchfield, J.P., Feldman, D.P.: Regularities Unseen, Randomness Observed: Levels of Entropy Convergence. Chaos 13(1), 25–54 (2003)zbMATHMathSciNetCrossRefGoogle Scholar
  18. Crutchfield, J.P., Young, K.: Inferring statistical complexity. Physical Review Letters 63(2), 105–108 (1989)MathSciNetCrossRefGoogle Scholar
  19. Edmundson, D.E., Enns, R.H.: Fully 3-dimensional collisions of bistable light bullets. Optics Letters 18, 1609–1611 (1993)CrossRefGoogle Scholar
  20. Eppstein, D.: Searching for spaceships. In: Nowakowski, R.J. (ed.) More Games of No Chance. MSRI Publications, vol. 42, pp. 433–453. Cambridge Univ. Press (2002)Google Scholar
  21. Fano, R.M.: Transmission of information: a statistical theory of communications. M.I.T. Press, Cambridge (1961)Google Scholar
  22. Feldman, D.P., McTague, C.S., Crutchfield, J.P.: The organization of intrinsic computation: Complexity-entropy diagrams and the diversity of natural information processing. Chaos 18(4), 43106 (2008)MathSciNetCrossRefGoogle Scholar
  23. Flecker, B., Alford, W., Beggs, J.M., Williams, P.L., Beer, R.D.: Partial information decomposition as a spatiotemporal filter. Chaos 21(3), 037104+ (2011)Google Scholar
  24. Goh, K.I., Barabási, A.L.: Burstiness and memory in complex systems. Europhysics Letters 81(4), 48002 (2008)MathSciNetCrossRefGoogle Scholar
  25. Grassberger, P.: New mechanism for deterministic diffusion. Physical Review A 28(6), 3666 (1983)CrossRefGoogle Scholar
  26. Grassberger, P.: Long-range effects in an elementary cellular automaton. Journal of Statistical Physics 45(1-2), 27–39 (1986a)MathSciNetCrossRefGoogle Scholar
  27. Grassberger, P.: Toward a quantitative theory of self-generated complexity. International Journal of Theoretical Physics 25(9), 907–938 (1986b)zbMATHMathSciNetCrossRefGoogle Scholar
  28. Grassberger, P.: Information content and predictability of lumped and distributed dynamical systems. Physica Scripta 40(3), 346 (1989)MathSciNetCrossRefGoogle Scholar
  29. Gray, L.: A Mathematician Looks at Wolfram’s New Kind of Science. Notices of the American Mathematical Society 50(2), 200–211 (2003)zbMATHMathSciNetGoogle Scholar
  30. Gutowitz, H., Domain, C.: The Topological Skeleton of Cellular Automaton Dynamics. Physica D 103(1-4), 155–168 (1997)zbMATHMathSciNetCrossRefGoogle Scholar
  31. Hanson, J.E., Crutchfield, J.P.: The Attractor-Basin Portait of a Cellular Automaton. Journal of Statistical Physics 66, 1415–1462 (1992)zbMATHMathSciNetCrossRefGoogle Scholar
  32. Hanson, J.E., Crutchfield, J.P.: Computational mechanics of cellular automata: An example. Physica D 103(1-4), 169–189 (1997)zbMATHMathSciNetCrossRefGoogle Scholar
  33. Helvik, T., Lindgren, K., Nordahl, M.G.: Local information in one-dimensional cellular automata. In: Sloot, P.M.A., Chopard, B., Hoekstra, A.G. (eds.) ACRI 2004. LNCS, vol. 3305, pp. 121–130. Springer, Heidelberg (2004)CrossRefGoogle Scholar
  34. Hordijk, W., Shalizi, C.R., Crutchfield, J.P.: Upper bound on the products of particle interactions in cellular automata. Physica D 154(3-4), 240–258 (2001)zbMATHMathSciNetCrossRefGoogle Scholar
  35. Jakubowski, M.H., Steiglitz, K., Squier, R.: Information transfer between solitary waves in the saturable Schrödinger equation. Physical Review E 56(6), 7267 (1997)CrossRefGoogle Scholar
  36. Jakubowski, M.H., Steiglitz, K., Squier, R.K.: Computing with solitons: A review and prospectus. Multiple-Valued Logic 6(5-6), 439–462 (2001)zbMATHMathSciNetGoogle Scholar
  37. Kinouchi, O., Copelli, M.: Optimal dynamical range of excitable networks at criticality. Nature Physics 2(5), 348–351 (2006)CrossRefGoogle Scholar
  38. Klyubin, A.S., Polani, D., Nehaniv, C.L.: Tracking Information Flow through the Environment: Simple Cases of Stigmergy. In: Pollack, J., Bedau, M., Husbands, P., Ikegami, T., Watson, R.A. (eds.) Proceedings of the Ninth International Conference on the Simulation and Synthesis of Living Systems (ALife IX), Boston, USA, pp. 563–568. MIT Press, Cambridge (2004)Google Scholar
  39. Klyubin, A.S., Polani, D., Nehaniv, C.L.: All Else Being Equal Be Empowered. In: Capcarrère, M.S., Freitas, A.A., Bentley, P.J., Johnson, C.G., Timmis, J. (eds.) ECAL 2005. LNCS (LNAI), vol. 3630, pp. 744–753. Springer, Heidelberg (2005)CrossRefGoogle Scholar
  40. Lafusa, A., Bossomaier, T.: Hyperplane Localisation of Self-Replicating and Other Complex Cellular Automata Rules. In: Proceedings of the the 2005 IEEE Congress on Evolutionary Computation, Edinburgh, vol. 1, pp. 844–849. IEEE Press (2005)Google Scholar
  41. Langton, C.G.: Computation at the edge of chaos: phase transitions and emergent computation. Physica D 42(1-3), 12–37 (1990)MathSciNetCrossRefGoogle Scholar
  42. Lindgren, K., Nordahl, M.G.: Complexity Measures and Cellular Automata. Complex Systems 2(4), 409–440 (1988)zbMATHMathSciNetGoogle Scholar
  43. Lindgren, K., Nordahl, M.G.: Universal computation in simple one-dimensional cellular automata. Complex Systems 4, 299–318 (1990)zbMATHMathSciNetGoogle Scholar
  44. Lindner, M., Vicente, R., Priesemann, V., Wibral, M.: TRENTOOL: A Matlab open source toolbox to analyse information flow in time series data with transfer entropy. BMC Neuroscience 12(1), 119+ (2011)Google Scholar
  45. Lizier, J.T.: JIDT: An information-theoretic toolkit for studying the dynamics of complex systems (2012),
  46. Lizier, J.T.: The Local Information Dynamics of Distributed Computation in Complex Systems. Springer Theses. Springer, Heidelberg (2013)zbMATHCrossRefGoogle Scholar
  47. Lizier, J.T., Atay, F.M., Jost, J.: Information storage, loop motifs, and clustered structure in complex networks. Physical Review E 86(2), 026110+ (2012a)Google Scholar
  48. Lizier, J.T., Flecker, B., Williams, P.L.: Towards a synergy-based approach to measuring information modification. In: Proceedings of the 2013 IEEE Symposium on Artificial Life (ALIFE), pp. 43–51. IEEE (2013)Google Scholar
  49. Lizier, J.T., Heinzle, J., Horstmann, A., Haynes, J.-D., Prokopenko, M.: Multivariate information-theoretic measures reveal directed information structure and task relevant changes in fMRI connectivity. Journal of Computational Neuroscience 30(1), 85–107 (2011a)MathSciNetCrossRefGoogle Scholar
  50. Lizier, J.T., Mahoney, J.R.: Moving frames of reference, relativity and invariance in transfer entropy and information dynamics. Entropy 15(1), 177–197 (2013)MathSciNetCrossRefGoogle Scholar
  51. Lizier, J.T., Pritam, S., Prokopenko, M.: Information dynamics in small-world Boolean networks. Artificial Life 17(4), 293–314 (2011b)CrossRefGoogle Scholar
  52. Lizier, J.T., Prokopenko, M.: Differentiating information transfer and causal effect. European Physical Journal B 73(4), 605–615 (2010)CrossRefGoogle Scholar
  53. Lizier, J.T., Prokopenko, M., Tanev, I., Zomaya, A.Y.: Emergence of Glider-like Structures in a Modular Robotic System. In: Bullock, S., Noble, J., Watson, R., Bedau, M.A. (eds.) Proceedings of the Eleventh International Conference on the Simulation and Synthesis of Living Systems (ALife XI), Winchester, UK, pp. 366–373. MIT Press, Cambridge (2008a)Google Scholar
  54. Lizier, J.T., Prokopenko, M., Zomaya, A.Y.: Detecting Non-trivial Computation in Complex Dynamics. In: Almeida e Costa, F., Rocha, L.M., Costa, E., Harvey, I., Coutinho, A. (eds.) ECAL 2007. LNCS (LNAI), vol. 4648, pp. 895–904. Springer, Heidelberg (2007)CrossRefGoogle Scholar
  55. Lizier, J.T., Prokopenko, M., Zomaya, A.Y.: The information dynamics of phase transitions in random Boolean networks. In: Bullock, S., Noble, J., Watson, R., Bedau, M.A. (eds.) Proceedings of the Eleventh International Conference on the Simulation and Synthesis of Living Systems (ALife XI), Winchester, UK, pp. 374–381. MIT Press, Cambridge (2008b)Google Scholar
  56. Lizier, J.T., Prokopenko, M., Zomaya, A.Y.: Local information transfer as a spatiotemporal filter for complex systems. Physical Review E 77(2), 026110+ (2008c)Google Scholar
  57. Lizier, J.T., Prokopenko, M., Zomaya, A.Y.: Information modification and particle collisions in distributed computation. Chaos 20(3), 037109+ (2010)Google Scholar
  58. Lizier, J.T., Prokopenko, M., Zomaya, A.Y.: Coherent information structure in complex computation. Theory in Biosciences 131(3), 193–203 (2012b)CrossRefGoogle Scholar
  59. Lizier, J.T., Prokopenko, M., Zomaya, A.Y.: Local measures of information storage in complex distributed computation. Information Sciences 208, 39–54 (2012c)CrossRefGoogle Scholar
  60. Lungarella, M., Sporns, O.: Mapping information flow in sensorimotor networks. PLoS Computational Biology 2(10), e144+ (2006)Google Scholar
  61. MacKay, D.J.C.: Information Theory, Inference, and Learning Algorithms. Cambridge University Press, Cambridge (2003)zbMATHGoogle Scholar
  62. Marinazzo, D., Wu, G., Pellicoro, M., Angelini, L., Stramaglia, S.: Information flow in networks and the law of diminishing marginal returns: evidence from modeling and human electroencephalographic recordings. PloS ONE 7(9), e45026+ (2012)Google Scholar
  63. Martinez, G.J., Adamatzky, A., McIntosh, H.V.: Phenomenology of glider collisions in cellular automaton Rule 54 and associated logical gates. Chaos, Solitons and Fractals 28(1), 100–111 (2006)zbMATHMathSciNetCrossRefGoogle Scholar
  64. McIntosh, H.V.: Linear Cellular Automata. Universidad Autónoma de Puebla, Puebla, Mexico (1990)Google Scholar
  65. Mitchell, M.: A Complex-Systems Perspective on the “Computation vs. Dynamics” Debate in Cognitive Science. In: Gernsbacher, M.A., Derry, S.J. (eds.) Proceedings of the 20th Annual Conference of the Cognitive Science Society (Cogsci 1998), Madison, Wisconsin, pp. 710–715 (1998a)Google Scholar
  66. Mitchell, M.: Computation in Cellular Automata: A Selected Review. In: Gramss, T., Bornholdt, S., Gross, M., Mitchell, M., Pellizzari, T. (eds.) Non-Standard Computation, pp. 95–140. VCH Verlagsgesellschaft, Weinheim (1998b)Google Scholar
  67. Mitchell, M., Crutchfield, J.P., Das, R.: Evolving Cellular Automata with Genetic Algorithms: A Review of Recent Work. In: Goodman, E.D., Punch, W., Uskov, V. (eds.) Proceedings of the First International Conference on Evolutionary Computation and Its Applications, Moscow, Russia, Russian Academy of Sciences (1996)Google Scholar
  68. Mitchell, M., Crutchfield, J.P., Hraber, P.T.: Evolving Cellular Automata to Perform Computations: Mechanisms and Impediments. Physica D 75, 361–391 (1994)zbMATHCrossRefGoogle Scholar
  69. Morgado, R., Cieśla, M., Longa, L., Oliveira, F.A.: Synchronization in the presence of memory. Europhysics Letters 79(1), 10002 (2007)MathSciNetCrossRefGoogle Scholar
  70. Obst, O., Boedecker, J., Schmidt, B., Asada, M.: On active information storage in input-driven systems. arXiv:1303.5526 (2013)Google Scholar
  71. Oxford English Dictionary (2008), (accessed August 5, 2008)
  72. Pahle, J., Green, A.K., Dixon, C.J., Kummer, U.: Information transfer in signaling pathways: a study using coupled simulated and experimental data. BMC Bioinformatics 9, 139 (2008)CrossRefGoogle Scholar
  73. Prokopenko, M., Boschietti, F., Ryan, A.J.: An Information-Theoretic Primer on Complexity, Self-Organization, and Emergence. Complexity 15(1), 11–28 (2009)MathSciNetCrossRefGoogle Scholar
  74. Prokopenko, M., Gerasimov, V., Tanev, I.: Evolving Spatiotemporal Coordination in a Modular Robotic System. In: Nolfi, S., Baldassarre, G., Calabretta, R., Hallam, J.C.T., Marocco, D., Meyer, J.-A., Miglino, O., Parisi, D. (eds.) SAB 2006. LNCS (LNAI), vol. 4095, pp. 558–569. Springer, Heidelberg (2006)CrossRefGoogle Scholar
  75. Prokopenko, M., Lizier, J.T., Obst, O., Wang, X.R.: Relating Fisher information to order parameters. Physical Review E 84, 041116+ (2011)Google Scholar
  76. Prokopenko, M., Lizier, J.T., Price, D.C.: On thermodynamic interpretation of transfer entropy. Entropy 15(2), 524–543 (2013)MathSciNetCrossRefGoogle Scholar
  77. Sánchez-Montañés, M.A., Corbacho, F.J.: Towards a New Information Processing Measure for Neural Computation. In: Dorronsoro, J.R. (ed.) ICANN 2002. LNCS, vol. 2415, pp. 637–642. Springer, Heidelberg (2002)CrossRefGoogle Scholar
  78. Schreiber, T.: Measuring Information Transfer. Physical Review Letters 85(2), 461–464 (2000)CrossRefGoogle Scholar
  79. Shalizi, C.R.: Causal Architecture, Complexity and Self-Organization in Time Series and Cellular Automata. PhD thesis, University of Wisconsin-Madison (2001)Google Scholar
  80. Shalizi, C.R., Crutchfield, J.P.: Computational mechanics: Pattern and Prediction, Structure and Simplicity. Journal of Statistical Physics 104, 817–879 (2001)zbMATHMathSciNetCrossRefGoogle Scholar
  81. Shalizi, C.R., Haslinger, R., Rouquier, J.-B., Klinkner, K.L., Moore, C.: Automatic filters for the detection of coherent structure in spatiotemporal systems. Physical Review E 73(3), 036104 (2006)Google Scholar
  82. Shannon, C.E.: A mathematical theory of communication. Bell System Technical Journal 27 (1948)Google Scholar
  83. Takens, F.: Detecting strange attractors in turbulence. In: Rand, D., Young, L.-S. (eds.) Dynamical Systems and Turbulence, Warwick 1980. Lecture Notes in Mathematics, vol. 21, pp. 366–381. Springer, Heidelberg (1981)CrossRefGoogle Scholar
  84. Von Neumann, J.: Theory of self-reproducing automata. University of Illinois Press, Urbana (1966)Google Scholar
  85. Wang, X.R., Miller, J.M., Lizier, J.T., Prokopenko, M., Rossi, L.F.: Quantifying and Tracing Information Cascades in Swarms. PLoS ONE 7(7), e40084+ (2012)Google Scholar
  86. Wibral, M., Pampu, N., Priesemann, V., Siebenhühner, F., Seiwert, H., Lindner, M., Lizier, J.T., Vicente, R.: Measuring Information-Transfer delays. PLoS ONE 8(2), e55809+ (2013)Google Scholar
  87. Wibral, M., Rahm, B., Rieder, M., Lindner, M., Vicente, R., Kaiser, J.: Transfer entropy in magnetoencephalographic data: quantifying information flow in cortical and cerebellar networks. Progress in Biophysics and Molecular Biology 105(1-2), 80–97 (2011)CrossRefGoogle Scholar
  88. Williams, P.L., Beer, R.D.: Nonnegative Decomposition of Multivariate Information. arXiv:1004.2515 (2010)Google Scholar
  89. Wolfram, S.: Cellular automata as models of complexity. Nature 311(5985), 419–424 (1984a)CrossRefGoogle Scholar
  90. Wolfram, S.: Computation theory of cellular automata. Communications in Mathematical Physics 96(1), 15–57 (1984b)zbMATHMathSciNetCrossRefGoogle Scholar
  91. Wolfram, S.: Universality and complexity in cellular automata. Physica D 10(1-2), 1–35 (1984c)MathSciNetCrossRefGoogle Scholar
  92. Wolfram, S.: A New Kind of Science. Wolfram Media, Champaign (2002)zbMATHGoogle Scholar
  93. Wuensche, A.: Classifying cellular automata automatically: Finding gliders, filtering, and relating space-time patterns, attractor basins, and the Z parameter. Complexity 4(3), 47–66 (1999)MathSciNetCrossRefGoogle Scholar
  94. Yamada, T., Aihara, K.: Spatio-temporal complex dynamics and computation in chaotic neural networks. In: Proceedings of the IEEE Symposium on Emerging Technologies and Factory Automation (ETFA 1994), Tokyo, pp. 239–244. IEEE (1994)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2014

Authors and Affiliations

  • Joseph T. Lizier
    • 1
    • 4
  • Mikhail Prokopenko
    • 1
    • 2
    • 3
  • Albert Y. Zomaya
    • 4
  1. 1.CSIRO Computational InformaticsEppingAustralia
  2. 2.School of PhysicsThe University of SydneySydneyAustralia
  3. 3.Department of ComputingMacquarie UniversityNorth RydeAustralia
  4. 4.School of Information TechnologiesThe University of SydneySydneyAustralia

Personalised recommendations