Theory in Biosciences

, Volume 131, Issue 3, pp 205–213

Information processing in echo state networks at the edge of chaos

Authors

    • Department of Adaptive Machine SystemsOsaka University
    • JST ERATO Asada Synergistic Intelligence Project
  • Oliver Obst
    • CSIRO ICT Centre, Adaptive Systems Team
    • School of Information TechnologiesThe University of Sydney
  • Joseph T. Lizier
    • CSIRO ICT Centre, Adaptive Systems Team
    • School of Information TechnologiesThe University of Sydney
  • N. Michael Mayer
    • Department of Electrical EngineeringNational Chung Cheng University
  • Minoru Asada
    • Department of Adaptive Machine SystemsOsaka University
    • JST ERATO Asada Synergistic Intelligence Project
Original Paper

DOI: 10.1007/s12064-011-0146-8

Cite this article as:
Boedecker, J., Obst, O., Lizier, J.T. et al. Theory Biosci. (2012) 131: 205. doi:10.1007/s12064-011-0146-8

Abstract

We investigate information processing in randomly connected recurrent neural networks. It has been shown previously that the computational capabilities of these networks are maximized when the recurrent layer is close to the border between a stable and an unstable dynamics regime, the so called edge of chaos. The reasons, however, for this maximized performance are not completely understood. We adopt an information-theoretical framework and are for the first time able to quantify the computational capabilities between elements of these networks directly as they undergo the phase transition to chaos. Specifically, we present evidence that both information transfer and storage in the recurrent layer are maximized close to this phase transition, providing an explanation for why guiding the recurrent layer toward the edge of chaos is computationally useful. As a consequence, our study suggests self-organized ways of improving performance in recurrent neural networks, driven by input data. Moreover, the networks we study share important features with biological systems such as feedback connections and online computation on input streams. A key example is the cerebral cortex, which was shown to also operate close to the edge of chaos. Consequently, the behavior of model systems as studied here is likely to shed light on reasons why biological systems are tuned into this specific regime.

Keywords

Recurrent neural networksReservoir computingInformation transferActive information storagePhase transition

Supplementary material

12064_2011_146_MOESM1_ESM.pdf (211 kb)
PDF (211 KB)

Copyright information

© Springer-Verlag 2011