Advertisement

Cognitive Computation

, Volume 9, Issue 3, pp 337–350 | Cite as

Echo State Property of Deep Reservoir Computing Networks

  • Claudio GallicchioEmail author
  • Alessio Micheli
Article

Abstract

In the last years, the Reservoir Computing (RC) framework has emerged as a state of-the-art approach for efficient learning in temporal domains. Recently, within the RC context, deep Echo State Network (ESN) models have been proposed. Being composed of a stack of multiple non-linear reservoir layers, deep ESNs potentially allow to exploit the advantages of a hierarchical temporal feature representation at different levels of abstraction, at the same time preserving the training efficiency typical of the RC methodology. In this paper, we generalize to the case of deep architectures the fundamental RC conditions related to the Echo State Property (ESP), based on the study of stability and contractivity of the resulting dynamical system. Besides providing a necessary condition and a sufficient condition for the ESP of layered RC networks, the results of our analysis provide also insights on the nature of the state dynamics in hierarchically organized recurrent models. In particular, we find out that by adding layers to a deep reservoir architecture, the regime of network’s dynamics can only be driven towards (equally or) less stable behaviors. Moreover, our investigation shows the intrinsic ability of temporal dynamics differentiation at the different levels in a deep recurrent architecture, with higher layers in the stack characterized by less contractive dynamics. Such theoretical insights are further supported by experimental results that show the effect of layering in terms of a progressively increased short-term memory capacity of the recurrent models.

Keywords

Reservoir computing Deep learning Echo state property Stability analysis Contractivity 

Notes

Compliance with Ethical Standards

Conflict of interests

The authors declare that they have no conflict of interest.

References

  1. 1.
    Aboudib A, Gripon V, Coppin G. A biologically inspired framework for visual information processing and an application on modeling bottom-up visual attention. Cogn Comput. 2016;8(6):1007–1026.CrossRefGoogle Scholar
  2. 2.
    Angelov P, Sperduti A. 2016. Challenges in deep learning. In: Proceedings of the 24th European symposium on artificial neural networks (ESANN), p. 489–495. http://www.i6doc.com.
  3. 3.
    Bengio Y. Learning deep architectures for ai Foundations and trends . Mach Learn. 2009;2(1):1–127.CrossRefGoogle Scholar
  4. 4.
    Bianchi F, Livi L, Alippi C. 2016. Investigating echo state networks dynamics by means of recurrence analysis. arXiv preprint arXiv:1601.07381, p. 1–25.
  5. 5.
    Buehner M, Young P. A tighter bound for the echo state property. IEEE Trans Neural Netw. 2006;17(3): 820–824.CrossRefPubMedGoogle Scholar
  6. 6.
    Cireşan D, Giusti A, Gambardella L, Schmidhuber J. 2013. Mitosis detection in breast cancer histology images with deep neural networks. In: International conference on medical image computing and computer-assisted intervention. Springer; p. 411–418.Google Scholar
  7. 7.
    Cireşan D, Meier U, Gambardella L, Schmidhuber J. Deep, big, simple neural nets for handwritten digit recognition. Neural Comput. 2010;22(12):3207–3220.CrossRefPubMedGoogle Scholar
  8. 8.
    Deng L, Yu D. Deep learning. Signal Process. 2014;7:3–4.Google Scholar
  9. 9.
    El Hihi S, Bengio Y. 1995. Hierarchical recurrent neural networks for long-term dependencies. In: NIPS, p. 493–499.Google Scholar
  10. 10.
    Gallicchio C, Micheli A. Architectural and markovian factors of echo state networks. Neural Netw. 2011;24 (5):440–456.CrossRefPubMedGoogle Scholar
  11. 11.
    Gallicchio C, Micheli A. 2016. Deep reservoir computing: a critical analysis. In: Proceedings of the 24th European symposium on artificial neural networks (ESANN), p. 497–502. http://www.i6doc.com.
  12. 12.
    Gallicchio C, Micheli A, Pedrelli L. 2016. Deep reservoir computing: a critical experimental analysis. Neurocomputing. Accepted.Google Scholar
  13. 13.
    Gerstner W, Kistler W. 2002. Spiking neuron models: aingle neurons, populations, plasticity. Cambridge University Press.Google Scholar
  14. 14.
    Goodfellow I, Bengio Y, Courville A. 2016. Deep learning. Book in preparation for MIT Press. http://www.deeplearningbook.org.
  15. 15.
    Graves A, Mohamed AR, Hinton G. 2013. Speech recognition with deep recurrent neural networks. In: 2013 IEEE international conference on Acoustics, speech and signal processing (ICASSP). IEEE; p. 6645–6649.Google Scholar
  16. 16.
    Hammer B, Tiňo P. Recurrent neural networks with small weights implement definite memory machines. Neural Comput. 2003;15(8):1897–1929.CrossRefGoogle Scholar
  17. 17.
    Hermans M, Schrauwen B. 2013. Training and analysing deep recurrent neural networks. In: NIPS, p. 190–198.Google Scholar
  18. 18.
    Jaeger H. 2001. The “echo state” approach to analysing and training recurrent neural networks - with an erratum note. Tech. rep. GMD - German National Research Institute for Computer Science, Tech. Rep.Google Scholar
  19. 19.
    Jaeger H. 2001. Short term memory in echo state networks, Tech. rep., German National Research Center for Information Technology.Google Scholar
  20. 20.
    Jaeger H. 2007. Discovering multiscale dynamical features with hierarchical echo state networks. Tech. rep., Jacobs University Bremen.Google Scholar
  21. 21.
    Jaeger H, Haas H. Harnessing nonlinearity: predicting chaotic systems and saving energy in wireless communication. Science 2004;304(5667):78–80.CrossRefPubMedGoogle Scholar
  22. 22.
    Jaeger H, Lukoṡeviċius M, Popovici D, Siewert U. Optimization and applications of echo state networks with leaky-integrator neurons. Neural Netw. 2007;20(3):335–352.CrossRefPubMedGoogle Scholar
  23. 23.
    Klopf A, Weaver S, Morgan J. A hierarchical network of control systems that learn: Modeling nervous system function during classical and instrumental conditioning. Adapt. Behav. 1993;1(3):263–319.CrossRefGoogle Scholar
  24. 24.
    Kolen JF, Kremer SC. 2001. A field guide to dynamical recurrent networks. IEEE Press.Google Scholar
  25. 25.
    Krizhevsky A, Sutskever I, Hinton G. Imagenet classification with deep convolutional neural networks. In: Pereira F, Burges CJC, Bottou L, and Weinberger KQ, editors. Advances in neural information processing systems; 2012. p. 1097–1105.Google Scholar
  26. 26.
    LeCun Y, Bengio Y, Hinton G. Deep learning. Nature 2015;521(7553):436–444.CrossRefPubMedGoogle Scholar
  27. 27.
    Lukoṡeviċius, M, Jaeger H. Reservoir computing approaches to recurrent neural network training. Comput Sci Rev. 2009;3(3):127–149.CrossRefGoogle Scholar
  28. 28.
    Maass W, Natschläger T, Markram H. Real-time computing without stable states: a new framework for neural computation based on perturbations. Neural Comput. 2002;14(11):2531–2560.CrossRefPubMedGoogle Scholar
  29. 29.
    Malik ZK, Hussain A, Wu QJ. 2016. Multilayered echo state machine: a novel architecture and algorithm. IEEE Transactions on cybernetics. (In Press).Google Scholar
  30. 30.
    Manjunath G, Jaeger H. Echo state property linked to an input: Exploring a fundamental characteristic of recurrent neural networks. Neural Comput. 2013;25(3):671–696.CrossRefPubMedGoogle Scholar
  31. 31.
    O’Searcoid M. 2006. Metric spaces. Springer Science & Business Media.Google Scholar
  32. 32.
    Pascanu R, Gulcehre C, Cho K, Bengio Y. 2014. How to construct deep recurrent neural networks arXiv preprint arXiv:1312.6026v5.
  33. 33.
    Rabinovich M, Huerta R, Varona P, Afraimovich V. Generation and reshaping of sequences in neural systems. Biol Cybern. 2006;95(6):519–536.CrossRefPubMedGoogle Scholar
  34. 34.
    Rabinovich M, Varona P, Selverston A, Abarbanel H. Dynamical principles in neuroscience. Rev Modern Phys. 2006;78(4):1213.CrossRefGoogle Scholar
  35. 35.
    Rodan A, Tiňo P. 2011. Negatively correlated echo state networks. In: Proceedings of the 19th European symposium on artificial neural networks (ESANN), p. 53–58. http://www.i6doc.com.
  36. 36.
    Sato Y, Nagatomi T, Horio K, Miyamoto H. The cognitive mechanisms of multi-scale perception for the recognition of extremely similar faces. Cogn Comput. 2015;7(5):501–508.CrossRefGoogle Scholar
  37. 37.
    Schmidhuber J. Deep learning in neural networks: an overview. Neural Netw. 2015;61:85–117.CrossRefPubMedGoogle Scholar
  38. 38.
    Schrauwen B, Wardermann M, Verstraeten D, Steil J, Stroobandt D. Improving reservoirs using intrinsic plasticity. Neurocomputing 2008;71(7):1159–1171.CrossRefGoogle Scholar
  39. 39.
    Spratling M. A hierarchical predictive coding model of object recognition in natural images. Cogn Comput. 2016: 1–17.Google Scholar
  40. 40.
    Steil J. 2004. Backpropagation-decorrelation: online recurrent learning with o (n) complexity. In: Proceedings of the 2004 IEEE international joint conference on neural networks (IJCNN). IEEE; vol. 2, p. 843–848.Google Scholar
  41. 41.
    Tiṅo P, Hammer B, Bodén M. 2007. Markovian bias of neural-based architectures with feedback connections. In: Perspectives of neural-symbolic integration. Springer; , p. 95–133.Google Scholar
  42. 42.
    Tiňo P, Dorffner G. Predicting the future of discrete sequences from fractal representations of the past. Mach Learn. 2001;45(2):187–217.CrossRefGoogle Scholar
  43. 43.
    Triefenbach F, Jalalvand A, Demuynck K, Martens JP. Acoustic modeling with hierarchical reservoirs. IEEE Trans Audio Speech Lang Process. 2013;21(11):2439–2450.CrossRefGoogle Scholar
  44. 44.
    Triefenbach F, Jalalvand A, Schrauwen B, Martens JP. 2010. Phoneme recognition with large hierarchical reservoirs. In: Advances in neural information processing systems, p. 2307–2315.Google Scholar
  45. 45.
    Tyrrell T. The use of hierarchies for action selection. Adapt Behav. 1993;1(4):387–420.CrossRefGoogle Scholar
  46. 46.
    Verstraeten D, Schrauwen B, D’haene M, Stroobandt D. An experimental unification of reservoir computing methods. Neural Netw. 2007;20(3):391–403.CrossRefPubMedGoogle Scholar
  47. 47.
    Wainrib G, Galtier M. A local echo state property through the largest lyapunov exponent. Neural Netw. 2016;76:39–45.CrossRefPubMedGoogle Scholar
  48. 48.
    Xue Y, Yang L, Haykin S. Decoupled echo state networks with lateral inhibition. Neural Netw. 2007;20 (3):365–376.CrossRefPubMedGoogle Scholar
  49. 49.
    Yildiz I, Jaeger H, Kiebel S. Re-visiting the echo state property. Neural Netw. 2012;35:1–9.CrossRefPubMedGoogle Scholar

Copyright information

© Springer Science+Business Media New York 2017

Authors and Affiliations

  1. 1.Department of Computer ScienceUniversity of PisaPisaItaly

Personalised recommendations