Echo State Property of Deep Reservoir Computing Networks
- 852 Downloads
In the last years, the Reservoir Computing (RC) framework has emerged as a state of-the-art approach for efficient learning in temporal domains. Recently, within the RC context, deep Echo State Network (ESN) models have been proposed. Being composed of a stack of multiple non-linear reservoir layers, deep ESNs potentially allow to exploit the advantages of a hierarchical temporal feature representation at different levels of abstraction, at the same time preserving the training efficiency typical of the RC methodology. In this paper, we generalize to the case of deep architectures the fundamental RC conditions related to the Echo State Property (ESP), based on the study of stability and contractivity of the resulting dynamical system. Besides providing a necessary condition and a sufficient condition for the ESP of layered RC networks, the results of our analysis provide also insights on the nature of the state dynamics in hierarchically organized recurrent models. In particular, we find out that by adding layers to a deep reservoir architecture, the regime of network’s dynamics can only be driven towards (equally or) less stable behaviors. Moreover, our investigation shows the intrinsic ability of temporal dynamics differentiation at the different levels in a deep recurrent architecture, with higher layers in the stack characterized by less contractive dynamics. Such theoretical insights are further supported by experimental results that show the effect of layering in terms of a progressively increased short-term memory capacity of the recurrent models.
KeywordsReservoir computing Deep learning Echo state property Stability analysis Contractivity
Compliance with Ethical Standards
Conflict of interests
The authors declare that they have no conflict of interest.
- 2.Angelov P, Sperduti A. 2016. Challenges in deep learning. In: Proceedings of the 24th European symposium on artificial neural networks (ESANN), p. 489–495. http://www.i6doc.com.
- 4.Bianchi F, Livi L, Alippi C. 2016. Investigating echo state networks dynamics by means of recurrence analysis. arXiv preprint arXiv:1601.07381, p. 1–25.
- 6.Cireşan D, Giusti A, Gambardella L, Schmidhuber J. 2013. Mitosis detection in breast cancer histology images with deep neural networks. In: International conference on medical image computing and computer-assisted intervention. Springer; p. 411–418.Google Scholar
- 8.Deng L, Yu D. Deep learning. Signal Process. 2014;7:3–4.Google Scholar
- 9.El Hihi S, Bengio Y. 1995. Hierarchical recurrent neural networks for long-term dependencies. In: NIPS, p. 493–499.Google Scholar
- 11.Gallicchio C, Micheli A. 2016. Deep reservoir computing: a critical analysis. In: Proceedings of the 24th European symposium on artificial neural networks (ESANN), p. 497–502. http://www.i6doc.com.
- 12.Gallicchio C, Micheli A, Pedrelli L. 2016. Deep reservoir computing: a critical experimental analysis. Neurocomputing. Accepted.Google Scholar
- 13.Gerstner W, Kistler W. 2002. Spiking neuron models: aingle neurons, populations, plasticity. Cambridge University Press.Google Scholar
- 14.Goodfellow I, Bengio Y, Courville A. 2016. Deep learning. Book in preparation for MIT Press. http://www.deeplearningbook.org.
- 15.Graves A, Mohamed AR, Hinton G. 2013. Speech recognition with deep recurrent neural networks. In: 2013 IEEE international conference on Acoustics, speech and signal processing (ICASSP). IEEE; p. 6645–6649.Google Scholar
- 17.Hermans M, Schrauwen B. 2013. Training and analysing deep recurrent neural networks. In: NIPS, p. 190–198.Google Scholar
- 18.Jaeger H. 2001. The “echo state” approach to analysing and training recurrent neural networks - with an erratum note. Tech. rep. GMD - German National Research Institute for Computer Science, Tech. Rep.Google Scholar
- 19.Jaeger H. 2001. Short term memory in echo state networks, Tech. rep., German National Research Center for Information Technology.Google Scholar
- 20.Jaeger H. 2007. Discovering multiscale dynamical features with hierarchical echo state networks. Tech. rep., Jacobs University Bremen.Google Scholar
- 24.Kolen JF, Kremer SC. 2001. A field guide to dynamical recurrent networks. IEEE Press.Google Scholar
- 25.Krizhevsky A, Sutskever I, Hinton G. Imagenet classification with deep convolutional neural networks. In: Pereira F, Burges CJC, Bottou L, and Weinberger KQ, editors. Advances in neural information processing systems; 2012. p. 1097–1105.Google Scholar
- 29.Malik ZK, Hussain A, Wu QJ. 2016. Multilayered echo state machine: a novel architecture and algorithm. IEEE Transactions on cybernetics. (In Press).Google Scholar
- 31.O’Searcoid M. 2006. Metric spaces. Springer Science & Business Media.Google Scholar
- 32.Pascanu R, Gulcehre C, Cho K, Bengio Y. 2014. How to construct deep recurrent neural networks arXiv preprint arXiv:1312.6026v5.
- 35.Rodan A, Tiňo P. 2011. Negatively correlated echo state networks. In: Proceedings of the 19th European symposium on artificial neural networks (ESANN), p. 53–58. http://www.i6doc.com.
- 39.Spratling M. A hierarchical predictive coding model of object recognition in natural images. Cogn Comput. 2016: 1–17.Google Scholar
- 40.Steil J. 2004. Backpropagation-decorrelation: online recurrent learning with o (n) complexity. In: Proceedings of the 2004 IEEE international joint conference on neural networks (IJCNN). IEEE; vol. 2, p. 843–848.Google Scholar
- 41.Tiṅo P, Hammer B, Bodén M. 2007. Markovian bias of neural-based architectures with feedback connections. In: Perspectives of neural-symbolic integration. Springer; , p. 95–133.Google Scholar
- 44.Triefenbach F, Jalalvand A, Schrauwen B, Martens JP. 2010. Phoneme recognition with large hierarchical reservoirs. In: Advances in neural information processing systems, p. 2307–2315.Google Scholar