Advertisement

An Introduction to Delay-Coupled Reservoir Computing

  • Johannes SchumacherEmail author
  • Hazem Toutounji
  • Gordon Pipa
Part of the Springer Series in Bio-/Neuroinformatics book series (SSBN, volume 4)

Abstract

Reservoir computing has been successfully applied in difficult time series prediction tasks by injecting an input signal into a spatially extended reservoir of nonlinear subunits to perform history-dependent nonlinear computation. Recently, the network was replaced by a single nonlinear node, delay-coupled to itself. Instead of a spatial topology, subunits are arrayed in time along one delay span of the system. As a result, the reservoir exists only implicitly in a single delay differential equation, the numerical solving of which is costly.We give here a brief introduction to the general topic of delay-coupled reservoir computing and derive approximate analytical equations for the reservoir by solving the underlying system explicitly. The analytical approximation represents the system accurately and yields comparable performance in reservoir benchmark tasks, while reducing computational costs practically by several orders of magnitude. This has important implications with respect to electronic realizations of the reservoir and opens up new possibilities for optimization and theoretical investigation.

Keywords

Delay Differential Equation Predictive Distribution Virtual Node Gaussian Process Regression Bayesian Model Selection 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Albert, A.: The Penrose-Moore Pseudo Inverse with Diverse Statistical Applications. Part I. The General Theory and Computational Methods. Defense Technical Information Center (1971)Google Scholar
  2. 2.
    Appeltant, L., Soriano, M.C., Van der Sande, G., Danckaert, J., Massar, S., Dambre, J., Schrauwen, B., Mirasso, C.R., Fischer, I.: Information processing using a single dynamical node as complex system. Nature Communications 2, 468 (2011)CrossRefGoogle Scholar
  3. 3.
    Berger, J.O.: Statistical Decision Theory and Bayesian Analysis. Springer Series in Statistics. Springer (1985)Google Scholar
  4. 4.
    Bishop, C.M.: Pattern Recognition and Machine Learning (Information Science and Statistics). Springer-Verlag New York, Inc., Secaucus (2006)Google Scholar
  5. 5.
    Efron, B., Tibshirani, R.J.: An Introduction to the Bootstrap. Chapman & Hall, New York (1993)CrossRefzbMATHGoogle Scholar
  6. 6.
    Ganguli, S., Huh, D., Sompolinsky, H.: Memory traces in dynamical systems. Proceedings of the National Academy of Sciences 105(48), 18970–18975 (2008)CrossRefGoogle Scholar
  7. 7.
    Guo, S., Wu, J.: Bifurcation Theory of Functional Differential Equations. Applied Mathematical Sciences. Springer London, Limited (2013)Google Scholar
  8. 8.
    Hastie, T.J., Tibshirani, R.J.: Generalized Additive Models. Chapman & Hall/CRC Monographs on Statistics & Applied Probability. Taylor & Francis (1990)Google Scholar
  9. 9.
    Heuser, H.: Lehrbuch der Analysis. Number pt. 1 in Mathematische Leitfäden. Teubner Verlag (2009)Google Scholar
  10. 10.
    Hida, T., Hitsuda, M.: Gaussian Processes. Translations of Mathematical Monographs. American Mathematical Society (2007)Google Scholar
  11. 11.
    Hoerl, A.E., Kennard, R.W.: Ridge regression: Biased estimation for nonorthogonal problems. Technometrics 12(1), 55–67 (1970)CrossRefzbMATHMathSciNetGoogle Scholar
  12. 12.
    Huebner, U., Abraham, N.B., Weiss, C.O.: Dimensions and entropies of chaotic intensity pulsations in a single-mode far-infrared nh3 laser. Phys. Rev. A 40(11), 6354–6365 (1989)CrossRefGoogle Scholar
  13. 13.
    Jäger, H.: The echo state approach to analysing and training recurrent neural networks. Technical report (2001)Google Scholar
  14. 14.
    Jaynes, E.T., Bretthorst, G.L.: Probability Theory: The Logic of Science. Cambridge University Press (2003)Google Scholar
  15. 15.
    Konishi, S., Kitagawa, G.: Information Criteria and Statistical Modeling. Springer Series in Statistics. Springer (2008)Google Scholar
  16. 16.
    Larger, L., Soriano, M.C., Brunner, D., Appeltant, L., Gutierrez, J.M., Pesquera, L., Mirasso, C.R., Fischer, I.: Photonic information processing beyond turing: an optoelectronic implementation of reservoir computing. Opt. Express 20(3), 3241–3249 (2012)CrossRefGoogle Scholar
  17. 17.
    Lazar, A., Pipa, G., Triesch, J.: SORN: a self-organizing recurrent neural network. Frontiers in Computational Neuroscience 3 (2009)Google Scholar
  18. 18.
    Lindley, D.V.: The 1988 wald memorial lectures: The present position in bayesian statistics. Statistical Science 5(1), 44–65 (1990)zbMATHMathSciNetGoogle Scholar
  19. 19.
    Maass, W., Natschläger, T., Markram, H.: Real-time computing without stable states: a new framework for neural computation based on perturbations. Neural Computation 14(11), 2531–2560 (2002)CrossRefzbMATHGoogle Scholar
  20. 20.
    Quarteroni, A., Sacco, R., Saleri, F.: Numerical Mathematics, 2nd edn. Texts in Applied Mathematics, vol. 37. Springer, Berlin (2006)Google Scholar
  21. 21.
    Rasmussen, C.E., Williams, C.K.I.: Gaussian Processes for Machine Learning. Adaptative computation and machine learning series. University Press Group Limited (2006)Google Scholar
  22. 22.
    Rugh, W.J.: Nonlinear system theory: the Volterra/Wiener approach. Johns Hopkins series in information sciences and systems. Johns Hopkins University Press (1981)Google Scholar
  23. 23.
    Schrauwen, B., Buesing, L., Legenstein, R.A.: On computational power and the order-chaos phase transition in reservoir computing. In: Koller, D., Schuurmans, D., Bengio, Y., Bottou, L. (eds.) NIPS, pp. 1425–1432. Curran Associates, Inc. (2008)Google Scholar
  24. 24.
    Shampine, L.F., Thompson, S.: Solving ddes in matlab. Applied Numerical Mathematics 37, 441–458 (2001)CrossRefzbMATHMathSciNetGoogle Scholar
  25. 25.
    Smith, H.: An Introduction to Delay Differential Equations with Applications to the Life Sciences. Texts in Applied Mathematics. Springer (2010)Google Scholar
  26. 26.
    Soriano, M.C., Ortín, S., Brunner, D., Larger, L., Mirasso, C.R., Fischer, I., Pesquera, L.: Optoelectronic reservoir computing: tackling noise-induced performance degradation. Optics Express 21(1), 12–20 (2013)CrossRefGoogle Scholar
  27. 27.
    Sundararajan, S., Sathiya Keerthi, S.: Predictive approaches for choosing hyperparameters in gaussian processes. Neural Computation 13(5), 1103–1118 (2001)CrossRefzbMATHGoogle Scholar
  28. 28.
    Toutounji, H., Schumacher, J., Pipa, G.: Optimized Temporal Multiplexing for Reservoir Computing with a Single Delay-Coupled Node. In: The 2012 International Symposium on Nonlinear Theory and its Applications (NOLTA 2012) (2012)Google Scholar
  29. 29.
    Weigend, A., Gershenfeld, N. (eds.): Time series prediction: forecasting the future and understanding the past. SFI studies in the sciences of complexity. Addison-Wesley (1993)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2015

Authors and Affiliations

  • Johannes Schumacher
    • 1
    Email author
  • Hazem Toutounji
    • 1
  • Gordon Pipa
    • 1
  1. 1.Institute of Cognitive ScienceUniversity of OsnabrückOsnabrückGermany

Personalised recommendations