ICONIP 2016: Neural Information Processing pp 376-383 | Cite as
Reward-Based Learning of a Memory-Required Task Based on the Internal Dynamics of a Chaotic Neural Network
Abstract
We have expected that dynamic higher functions such as “thinking” emerge through the growth from exploration in the framework of reinforcement learning (RL) using a chaotic Neural Network (NN). In this frame, the chaotic internal dynamics is used for exploration and that eliminates the necessity of giving external exploration noises. A special RL method for this framework has been proposed in which “traces” were introduced. On the other hand, reservoir computing has shown its excellent ability in learning dynamic patterns. Hoerzer et al. showed that the learning can be done by giving rewards and exploration noises instead of explicit teacher signals. In this paper, aiming to introduce the learning ability into our new RL framework, it was shown that the memory-required task in the work of Hoerzer et al. could be learned without giving exploration noises by utilizing the chaotic internal dynamics while the exploration level was adjusted flexibly and autonomously. The task could be learned also using “traces”, but still with problems.
Keywords
Chaotic neural network Reservoir computing Reward-Modulated Hebbian Learning Traces Dynamic higher functionsNotes
Acknowledgement
The authors wish to thank Prof. Hiromichi Suetani for introducing FORCE Learning and the work of Hoerzer et al. to us. This work was supported by JSPS KAKENHI Grant Number 15K00360.
References
- 1.Shibata, K., Okabe, Y.: Reinforcement learning when visual signals are directly given as inputs. In: Proceedings of ICNN 1997, vol. 3, pp. 1716–1720 (1997)Google Scholar
- 2.Shibata, K.: Emergence of intelligence through reinforcement learning with a neural network. In: Mellouk, A. (ed.) Advances in Reinforcement Learning, pp. 99–120. InTech, Rijeka (2011)Google Scholar
- 3.Shibata, K., Utsunomiya, H.: Discovery of pattern meaning from delayed rewards by reinforcement learning with a recurrent neural network. In: Proceedings of IJCNN, pp. 1445–1452 (2011)Google Scholar
- 4.Shibata, K., Goto, K.: Emergence of flexible prediction-based discrete decision making and continuous motion generation through actor-q-learning. In: Proceedings of ICDL-Epirob, ID 15 (2013)Google Scholar
- 5.Sawatsubashi, Y., et al.: Emergence of discrete and abstract state representation in continuous input task. In: Robot Intelligence Technology and Applications, pp. 13–22 (2012)Google Scholar
- 6.Shibata, K., Sakashita, Y.: Reinforcement learning with internal-dynamics-based exploration using a chaotic neural network. In: Proceedings of International Joint Conference on Neural Networks (IJCNN) (2015). 2015.7Google Scholar
- 7.Goto, Y., Shibata, K.: Emergence of higher exploration in reinforcement learning using a chaotic neural network. In: Akira, H., Seiichi, O., Kenji, D., Kazushi, I., Minho, L., Derong, L. (eds.) ICONIP 2016. LNCS, pp. 40–48. Springer, Heidelberg (2016)Google Scholar
- 8.Jaeger, H.: The “echo state” approach to analysing and training recurrent neural networks. GMD report 148, p. 43 (2001)Google Scholar
- 9.Maass, W., Natschlger, T., Markram, H.: Real-time computing without stable states: a new framework for neural computation based on perturbations. Neural Comput. 14(11), 2531–2560 (2002)CrossRefMATHGoogle Scholar
- 10.Sussillo, D., Abbott, L.F.: Generating coherent patterns of activity from chaotic neural networks. Neuron 63(4), 544–557 (2009)CrossRefGoogle Scholar
- 11.Hoerzer, G.M., Legenstein, R., Maass, W.: Emergence of complex computational structures from chaotic neural networks through reward-modulated Hebbian learning. Cereb. Cortex 24(3), 677–690 (2014)CrossRefGoogle Scholar