Reinforcement Learning of a Memory Task Using an Echo State Network with Multi-layer Readout

  • Toshitaka MatsukiEmail author
  • Katsunari Shibata
Conference paper
Part of the Advances in Intelligent Systems and Computing book series (AISC, volume 751)


Training a neural network (NN) through reinforcement learning (RL) has been focused on recently, and a recurrent NN (RNN) is used in learning tasks that require memory. Meanwhile, to cover the shortcomings in learning an RNN, the reservoir network (RN) has been often employed mainly in supervised learning. The RN is a special RNN and has attracted much attention owing to its rich dynamic representations. An approach involving the use of a multi-layer readout (MLR), which comprises a multi-layer NN, was studied for acquiring complex representations using the RN. This study demonstrates that an RN with MLR can learn a “memory task” through RL with back propagation. In addition, non-linear representations required to clear the task are not observed in the RN but are constructed by learning in the MLR. The results suggest that the MLR can make up for the limited computational ability in an RN.


Echo state network Reservoir network Reservoir computing Reinforcement learning 



This work was supported by JSPS KAKENHI Grant Number 15K00360.


  1. 1.
    LeCun, Y., Bengio, Y., Hinton, G.: Deep learning. Nature 521, 436–444 (2015)Google Scholar
  2. 2.
    Shibata, K.: Functions that emerge through end-to-end reinforcement learning (2017). arXiv:1703.02239
  3. 3.
    Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., Riedmiller, M.: Playing Atari with deep reinforcement learning (2013). arXiv:1312.5602
  4. 4.
    Shibata, K., Utsunomiya, H.: Discovery of pattern meaning from delayed rewards by reinforcement learning with a recurrent neural network. In: Proceedings of IJCNN, pp. 1445–1452 (2011)Google Scholar
  5. 5.
    Shibata, K., Goto, K.: Emergence of flexible prediction-based discrete decision making and continuous motion generation through Actor-Q-Learning. In: Proceedings of ICDL-Epirob, ID 15 (2013)Google Scholar
  6. 6.
    Jaeger, H.: The “echo state approach to analysing and training recurrent neural networks-with an erratum note. Bonn, Germany: German National Research Center for Information Technology GMD Technical Report 148.34, p. 13 (2001)Google Scholar
  7. 7.
    Maass, W., Natschlger, T., Markram, H.: Real-time computing without stable states: a new framework for neural computation based on perturbations. Neural Comput. 14(11), 2531–2560 (2002)Google Scholar
  8. 8.
    Salmen, M., Ploger, P.G.: Echo state networks used for motor control. In: Proceedings of the 2005 IEEE International Conference on Robotics and Automation, ICRA 2005. IEEE (2005)Google Scholar
  9. 9.
    Sussillo, D., Abbott, L.F.: Generating coherent patterns of activity from chaotic neural networks. Neuron Article 63(4), 544–557(2009)Google Scholar
  10. 10.
    Hoerzer, G.M., Legenstein, R., Maass, W.: Emergence of complex computational structures from chaotic neural networks through reward-modulated Hebbian learning. Cerebral Cortex 24(3), 677–690 (2014)Google Scholar
  11. 11.
    Lukusevic̆ius, M., Jaeger, H.: Reservoir computing approaches to recurrent neural network training. Comput. Sci. Rev. 3(3), 127–149 (2009)Google Scholar
  12. 12.
    Bush, K., Anderson, C.: Modeling reward functions for incomplete state representations via echo state networks. In: Proceedings. 2005 IEEE International Joint Conference on Neural Networks, 2005. IJCNN ’05, vol. 5. IEEE (2005)Google Scholar
  13. 13.
    Babinec, S̆., Pospchal, J.: Merging echo state and feedforward neural networks for time series forecasting. In: Artificial Neural Networks ICANN 2006, pp. 367–375 (2006)Google Scholar
  14. 14.
    Goto, Y., Shibata, K.: Emergence of higher exploration in reinforcement learning using a chaotic neural network. In: Proceedings of International Conference on Neural Information Processing (ICONIP) 2016, LNCS 9947, pp. 40–48 (2016)Google Scholar
  15. 15.
    Matsuki, T., Shibata, K.: Reward-based learning of a memory-required task based on the internal dynamics of a chaotic neural network. In: Proceedings of International Conference on Neural Information Processing (ICONIP) 2016, LNCS 9947, pp. 376–383 (2016)Google Scholar

Copyright information

© Springer International Publishing AG, part of Springer Nature 2019

Authors and Affiliations

  1. 1.Oita UniversityOitaJapan

Personalised recommendations