Skip to main content
SpringerLink
Log in

Deep Reservoir Computing

  • Chapter
  • First Online:
Reservoir Computing

Part of the book series: Natural Computing Series ((NCS))

Abstract

This chapter surveys the recent advancements on the extension of Reservoir Computing toward deep architectures, which is gaining increasing research attention in the neural networks community. Within this context, we focus on describing the major features of Deep Echo State Networks based on the hierarchical composition of multiple reservoirs. The intent is to provide a useful reference to guide applications and further developments of this efficient and effective class of approaches to deal with times-series and more complex data within a unified description and analysis.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 149.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Hardcover Book
USD 199.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    The latter set of conditions is ensured when the reservoir state transition functions in Eqs. 1 and 2 are squashing non-linearities, such is the case, e.g., of \(\tanh \).

References

  • M. Alizamir, S. Kim, O. Kisi, M. Zounemat-Kermani, Deep echo state network: a novel machine learning approach to model dew point temperature using meteorological variables. Hydrol. Sci. J. 65(7), 1173–1190 (2020)

    Article  Google Scholar 

  • P. Angelov, A. Sperduti, Challenges in deep learning, in Proceedings of the 24th European Symposium on Artificial Neural Networks (ESANN) (2016), pp. 489–495, i6doc.com

    Google Scholar 

  • F.M. Bianchi, S. Scardapane, S. Lokse, R. Jenssen, Bidirectional deep-readout echo state networks, in Proceedings of the 26th European Symposium on Artificial Neural Networks (ESANN) (2018), pp. 425–430

    Google Scholar 

  • P.S. Churchland, T.J. Sejnowski, The Computational Brain (The MIT Press, 1992)

    Google Scholar 

  • V. Colla, I. Matino, S. Dettori, S. Cateni, R. Matino, Reservoir computing approaches applied to energy management in industry, in International Conference on Engineering Applications of Neural Networks (Springer, 2019), pp. 66–79

    Google Scholar 

  • M. Dale, Neuroevolution of hierarchical reservoir computers, in Proceedings of the Genetic and Evolutionary Computation Conference. (ACM, 2018), pp. 410–417

    Google Scholar 

  • J. Del Ser, I. Lana, E.L. Manibardo, I. Oregi, E. Osaba, J.L. Lobo, M.N. Bilbao, E.I. Vlahogianni, Deep echo state networks for short-term traffic forecasting: performance comparison and statistical assessment (2020), arXiv:2004.08170

  • S. Dettori, I. Matino, V. Colla, R. Speets, Deep echo state networks in industrial applications, in IFIP International Conference on Artificial Intelligence Applications and Innovations (Springer, 2020), pp. 53–63

    Google Scholar 

  • S. El Hihi, Y. Bengio, Hierarchical recurrent neural networks for long-term dependencies, in Advances in Neural Information Processing Systems (NIPS) (1996), pp. 493–499

    Google Scholar 

  • M. Freiberger, S. Sackesyn, C. Ma, A. Katumba, P. Bienstman, J. Dambre, Improving time series recognition and prediction with networks and ensembles of passive photonic reservoirs. IEEE J. Sel. Top. Quantum Electron. 26(1), 1–11 (2019)

    Article  Google Scholar 

  • C. Gallicchio, A. Micheli, Graph echo state networks, in The 2010 International Joint Conference on Neural Networks (IJCNN) (IEEE, 2010), pp. 1–8

    Google Scholar 

  • C. Gallicchio, Short-term memory of deep RNN, in Proceedings of the 26th European Symposium on Artificial Neural Networks (ESANN) (2018), pp. 633–638

    Google Scholar 

  • C. Gallicchio, A. Micheli, Deep reservoir computing: a critical analysis, in Proceedings of the 24th European Symposium on Artificial Neural Networks (ESANN) (2016), pp. 497–502, i6doc.com

    Google Scholar 

  • C. Gallicchio, A. Micheli, Echo state property of deep reservoir computing networks. Cogn. Comput. 9(3), 337–350 (2017a)

    Article  Google Scholar 

  • C. Gallicchio, A. Micheli, Experimental analysis of deep echo state networks for ambient assisted living, in Proceedings of the 3rd Workshop on Artificial Intelligence for Ambient Assisted Living (AI*AAL 2017), Co-located with the 16th International Conference of the Italian Association for Artificial Intelligence (AI*IA 2017) (2017b)

    Google Scholar 

  • C. Gallicchio, A. Micheli, Deep Echo State Network (DeepESN): a brief survey (2018a), arXiv:1712.04323

  • C. Gallicchio, A. Micheli, Deep tree echo state networks, in Proceedings of the 2018 International Joint Conference on Neural Networks (IJCNN) (IEEE, 2018b), pp. 499–506

    Google Scholar 

  • C. Gallicchio, A. Micheli, Why layering in recurrent neural networks? a DeepESN survey, in Proceedings of the 2018 IEEE International Joint Conference on Neural Networks (IJCNN) (IEEE, 2018c), pp. 1800–1807

    Google Scholar 

  • C. Gallicchio, A. Micheli, Deep reservoir neural networks for trees. Inf. Sci. 174–193 (2019)

    Google Scholar 

  • C. Gallicchio, A. Micheli, Fast and deep graph neural networks, in Proceedings of the Thirty-Fourth AAAI Conference on Artificial Intelligence (AAAI-20) (2020), pp. 3898–3905

    Google Scholar 

  • C. Gallicchio, S. Scardapane, Deep randomized neural networks, in Recent Trends in Learning From Data. (Springer, 2020), pp. 43–68

    Google Scholar 

  • C. Gallicchio, J.D. Martin-Guerrero, A. Micheli, E. Soria-Olivas, Randomized machine learning approaches: Recent developments and challenges, in Proceedings of the 25th European Symposium on Artificial Neural Networks (ESANN) (2017a), pp. 77–86, i6doc.com

    Google Scholar 

  • C. Gallicchio, A. Micheli, L. Pedrelli, Deep reservoir computing: a critical experimental analysis. Neurocomputing 268, 87–99 (2017bb). https://doi.org/10.1016/j.neucom.2016.12.089

  • C. Gallicchio, A. Micheli, L. Silvestri, Local lyapunov exponents of deep RNN, in Proceedings of the 25th European Symposium on Artificial Neural Networks (ESANN) (2017c), pp. 559–564, i6doc.com

    Google Scholar 

  • C. Gallicchio, A. Micheli, L. Pedrelli, Deep echo state networks for diagnosis of Parkinson’s disease, in Proceedings of the 26th European Symposium on Artificial Neural Networks (ESANN) (2018a), pp. 397–402

    Google Scholar 

  • C. Gallicchio, A. Micheli, L. Pedrelli, Design of deep echo state networks. Neural Netw. 108, 33–47 (2018b)

    Article  Google Scholar 

  • C. Gallicchio, A. Micheli, L. Silvestri, Local lyapunov exponents of deep echo state networks. Neurocomputing 298, 34–45 (2018c)

    Article  Google Scholar 

  • C. Gallicchio, A. Micheli, P. Tiňo, Randomized recurrent neural networks, in Proceedings of the 26th European Symposium on Artificial Neural Networks (ESANN) (2018d), pp. 415–424, i6doc.com

    Google Scholar 

  • C. Gallicchio, A. Micheli, L. Pedrelli, Comparison between DeepESNs and gated RNNs on multivariate time-series prediction, in Proceedings of the 27th European Symposium on Artificial Neural Networks (ESANN) (2019a), pp. 619–624

    Google Scholar 

  • C. Gallicchio, A. Micheli, L. Pedrelli, Hierarchical temporal representation in linear reservoir computing, in Neural Advances in Processing Nonlinear Dynamic Signals, ed. by A. Esposito, M. Faundez-Zanuy, F.C. Morabito, E. Pasero (Springer International Publishing, Cham, 2019b), pp. 119–129. https://doi.org/10.1007/978-3-319-95098-3_11, arXiv:1705.05782

  • W. Gerstner, W.M. Kistler, Spiking Neuron Models: Single Neurons, Populations, Plasticity (Cambridge University Press, Cambridge, 2002)

    Book  Google Scholar 

  • I. Goodfellow, Y. Bengio, A. Courville, Deep Learning (MIT Press, 2016)

    Google Scholar 

  • A. Graves, A.R. Mohamed, G. Hinton, Speech recognition with deep recurrent neural networks, in IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (IEEE, 2013), pp. 6645–6649

    Google Scholar 

  • I. Harvey, The microbial genetic algorithm, in European Conference on Artificial Life (Springer, 2009), pp. 126–133

    Google Scholar 

  • M. Hermans, B. Schrauwen, Training and analysing deep recurrent neural networks, in NIPS (2013), pp. 190–198

    Google Scholar 

  • H. Hu, L. Wang, S.X. Lv, Forecasting energy consumption and wind power generation using deep echo state network. Renew. Energy 154, 598–613 (2020)

    Article  Google Scholar 

  • H. Jaeger, The “echo state” approach to analysing and training recurrent neural networks - with an erratum note. Technical report, GMD - German National Research Institute for Computer Science (2001)

    Google Scholar 

  • H. Jaeger, Short term memory in echo state networks. Technical report, German National Research Center for Information Technology, 2001

    Google Scholar 

  • H. Jaeger, Discovering multiscale dynamical features with hierarchical echo state networks. Technical report, Jacobs University Bremen (2007)

    Google Scholar 

  • H. Jaeger, H. Haas, Harnessing nonlinearity: predicting chaotic systems and saving energy in wireless communication. Science 304(5667), 78–80 (2004)

    Article  Google Scholar 

  • H. Jaeger, M. Lukoševičius, D. Popovici, U. Siewert, Optimization and applications of echo state networks with leaky-integrator neurons. Neural Netw. 20(3), 335–352 (2007)

    Article  Google Scholar 

  • T. Kim, B.R. King, Time series prediction using deep echo state networks. Neural Comput. Appl. 1–19 (2020)

    Google Scholar 

  • R. Legenstein, W. Maass, Edge of chaos and prediction of computational performance for neural circuit models. Neural Netw. 20(3), 323–334 (2007a)

    Article  Google Scholar 

  • R. Legenstein, W. Maass, What makes a dynamical system computationally powerful. New directions in statistical signal processing: from systems to brain 127–154 (2007b)

    Google Scholar 

  • Q. Li, Z. Wu, R. Ling, L. Feng, K. Liu, Multi-reservoir echo state computing for solar irradiance prediction: a fast yet efficient deep learning approach. Appl. Soft Comput. 95 (2020)

    Google Scholar 

  • A. Lugnan, A. Katumba, F. Laporte, M. Freiberger, S. Sackesyn, C. Ma, E. Gooskens, J. Dambre, P. Bienstman, Photonic neuromorphic information processing and reservoir computing. APL Photonics 5(2) (2020)

    Google Scholar 

  • M. Lukoševičius, H. Jaeger, Reservoir computing approaches to recurrent neural network training. Comput. Sci. Rev. 3(3), 127–149 (2009)

    Article  Google Scholar 

  • W. Maass, T. Natschläger, H. Markram, Real-time computing without stable states: a new framework for neural computation based on perturbations. Neural Comput. 14(11), 2531–2560 (2002)

    Article  Google Scholar 

  • Z.K. Malik, A. Hussain, Q.J. Wu, Multilayered echo state machine: a novel architecture and algorithm. IEEE Trans. Cybern. 47(4), 946–959 (2017)

    Article  Google Scholar 

  • S. Nichele, A. Molund, Deep learning with cellular automaton-based reservoir computing. Complex Syst. 26, 319–340 (2017)

    Article  MathSciNet  Google Scholar 

  • R. Pascanu, C. Gulcehre, K. Cho, Y. Bengio, How to construct deep recurrent neural networks (2014), arXiv:1312.6026v5

  • S. Scardapane, D. Wang, Randomness in neural networks: an overview. Wiley Interdiscip. Rev.: Data Mining Knowl. Discov. 7(2), e1200 (2017)

    Google Scholar 

  • J. Schmidhuber, Learning complex, extended sequences using the principle of history compression. Neural Comput. 4(2), 234–242 (1992)

    Article  Google Scholar 

  • J. Schmidhuber, Deep learning in neural networks: an overview. Neural Netw. 61, 85–117 (2015)

    Article  Google Scholar 

  • B. Schrauwen, M. Wardermann, D. Verstraeten, J. Steil, D. Stroobandt, Improving reservoirs using intrinsic plasticity. Neurocomputing 71(7), 1159–1171 (2008)

    Article  Google Scholar 

  • Z. Song, K. Wu, J. Shao, Destination prediction using deep echo state network. Neurocomputing 406, 343–353 (2020)

    Article  Google Scholar 

  • X. Sun, T. Li, Q. Li, Y. Huang, Y. Li, Deep belief echo-state network and its application to time series prediction. Knowl.-Based Syst. 130, 17–29 (2017)

    Google Scholar 

  • F. Triefenbach, A. Jalalvand, B. Schrauwen, J.P. Martens, Phoneme recognition with large hierarchical reservoirs, in Advances in Neural Information Processing Systems (2010), pp. 2307–2315

    Google Scholar 

  • F. Triefenbach, A. Jalalvand, K. Demuynck, J.P. Martens, Acoustic modeling with hierarchical reservoirs. IEEE Trans. Audio Speech Lang. Process. 21(11), 2439–2450 (2013)

    Article  Google Scholar 

  • D. Verstraeten, B. Schrauwen, M. d’Haene, D. Stroobandt, An experimental unification of reservoir computing methods. Neural Netw. 20(3), 391–403 (2007)

    Article  Google Scholar 

  • J. Von Neumann, A.W. Burks, Theory of Self-Reproducing Automata (University of Illinois Press, Urbana, 1996)

    Google Scholar 

  • S. Wolfram, Universality and complexity in cellular automata. Phys. D 10(1–2), 1–35 (1984)

    Article  MathSciNet  Google Scholar 

  • I.B. Yildiz, H. Jaeger, S.J. Kiebel, Re-visiting the echo state property. Neural Netw. 35, 1–9 (2012)

    Article  Google Scholar 

  • O. Yilmaz, Reservoir computing using cellular automata (2014), arXiv:1410.0162

  • B. Zajzon, R. Duartel, A. Morrison, Transferring state representations in hierarchical spiking neural networks, in Proceedings of the 2018 International Joint Conference on Neural Networks (IJCNN) (IEEE, 2018), pp. 1785–1793

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Computer Science, University of Pisa, Largo Bruno Pontecorvo 3, 56127, Pisa, Italy

    Claudio Gallicchio & Alessio Micheli

Authors
  1. Claudio Gallicchio
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Alessio Micheli
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Claudio Gallicchio .

Editor information

Editors and Affiliations

  1. University of Tokyo, Tokyo, Japan

    Kohei Nakajima

  2. Consejo Superior de Investigaciones Científicas, IFISC (UIB-CSIC), Palma, Spain

    Ingo Fischer

Rights and permissions

Reprints and permissions

Copyright information

© 2021 Springer Nature Singapore Pte Ltd.

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Gallicchio, C., Micheli, A. (2021). Deep Reservoir Computing. In: Nakajima, K., Fischer, I. (eds) Reservoir Computing. Natural Computing Series. Springer, Singapore. https://doi.org/10.1007/978-981-13-1687-6_4

Download citation

  • DOI: https://doi.org/10.1007/978-981-13-1687-6_4

  • Published:

  • Publisher Name: Springer, Singapore

  • Print ISBN: 978-981-13-1686-9

  • Online ISBN: 978-981-13-1687-6

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation