Original Paper

Theory in Biosciences

, Volume 131, Issue 3, pp 205-213

First online:

Information processing in echo state networks at the edge of chaos

  • Joschka BoedeckerAffiliated withDepartment of Adaptive Machine Systems, Osaka UniversityJST ERATO Asada Synergistic Intelligence Project Email author 
  • , Oliver ObstAffiliated withCSIRO ICT Centre, Adaptive Systems TeamSchool of Information Technologies, The University of Sydney
  • , Joseph T. LizierAffiliated withCSIRO ICT Centre, Adaptive Systems TeamSchool of Information Technologies, The University of Sydney
  • , N. Michael MayerAffiliated withDepartment of Electrical Engineering, National Chung Cheng University
  • , Minoru AsadaAffiliated withDepartment of Adaptive Machine Systems, Osaka UniversityJST ERATO Asada Synergistic Intelligence Project

Rent the article at a discount

Rent now

* Final gross prices may vary according to local VAT.

Get Access

Abstract

We investigate information processing in randomly connected recurrent neural networks. It has been shown previously that the computational capabilities of these networks are maximized when the recurrent layer is close to the border between a stable and an unstable dynamics regime, the so called edge of chaos. The reasons, however, for this maximized performance are not completely understood. We adopt an information-theoretical framework and are for the first time able to quantify the computational capabilities between elements of these networks directly as they undergo the phase transition to chaos. Specifically, we present evidence that both information transfer and storage in the recurrent layer are maximized close to this phase transition, providing an explanation for why guiding the recurrent layer toward the edge of chaos is computationally useful. As a consequence, our study suggests self-organized ways of improving performance in recurrent neural networks, driven by input data. Moreover, the networks we study share important features with biological systems such as feedback connections and online computation on input streams. A key example is the cerebral cortex, which was shown to also operate close to the edge of chaos. Consequently, the behavior of model systems as studied here is likely to shed light on reasons why biological systems are tuned into this specific regime.

Keywords

Recurrent neural networks Reservoir computing Information transfer Active information storage Phase transition