A Spiking Neural Network Based Autonomous Reinforcement Learning Model and Its Application in Decision Making
Inspired by biological spike information processing and the multiple brain region coordination mechanism, we propose an autonomous spiking neural network model for decision making. The proposed model is an expansion of the basal ganglia circuitry with automatic environment perception. It automatically constructs environmental states from image inputs. Contributions of this investigation can be summarized as the following: (1) In our model, the simplified Hodgkin-Huxley computing model is developed to achieve calculation efficiency closed to the LIF model and is used to obtain and test the ionic level properties in cognition. (2) A spike based motion perception mechanism is proposed to extract key elements for learning process from raw pixels without large amount of training. We apply our model in the “flappy bird” game and after dozens of training times, it can automatically generate rules to play well in the game. Besides, our model simulates cognitive defects when blocking some of sodium or potassium ion channels in the Hodgkin-Huxley model and this can be considered as a computational exploration on the mechanisms of cognition deep into ionic level.
KeywordsSpiking neural network Hodgkin-Huxley model Autonomous reinforcement learning Decision making Basal Ganglia
This study was funded by the Strategic Priority Research Program of the Chinese Academy of Sciences (XDB02060007), and Beijing Municipal Commission of Science and Technology (Z151100000915070, Z161100000216124).
- 1.Stewart, T.C., Bekolay, T., Eliasmith, C.: Learning to select actions with spiking neurons in the Basal Ganglia. Front. Neurosci. 6(2), 1–14 (2012)Google Scholar
- 2.Bekolay, T., Eliasmith, C.: A general error-modulated STDP learning rule applied to reinforcement learning in the Basal Ganglia. In: Computational and Systems Neuroscience Conference, Salt Lake City, Utah, pp. 24–27 (2011)Google Scholar
- 8.Stewart, T.C., Choo, X., Eliasmith, C.: Dynamic behavior of a spiking model of action selection in the Basal Ganglia. In: Proceedings of the 10th International Conference on Cognitive Modeling, pp. 5–8 (2010)Google Scholar
- 12.MacNeil, D., Eliasmith, C.: Fine-tuning and the stability of recurrent neural networks. Public Lib. Sci. (PLoS One) 6(9), 1–16 (2011)Google Scholar
- 18.Dayan, P., Abbott, L.F.: Computational and Mathematical Modeling of Neural Systems: Model Neurons I: Neuroelectronic. MIT Press, Cambridge (2003)Google Scholar
- 19.Izhikevich, E.M.: Dynamical Systems in Neuroscience: The Geometry of Excitability and Bursting. MIT Press, Cambridge (2004)Google Scholar
- 21.Nelson, M.E.: Electrophysiological models. In: Databasing the Brain: From Data to Knowledge. Wiley, New York (2004)Google Scholar
- 23.Wells, R.B.: Introduction to biological signal processing and computational neuroscience. Moscow (2010)Google Scholar
- 24.Long, L.N., Fang, G.L.: A review of biologically plausible neuron models for spiking neural networks. In: AIAA InfoTech Aerospace Conference, Atlanta, 20–22 April 2010Google Scholar
- 25.Weber, C., Elshaw, M., Wermter, S., Triesch, J., Willmot, C.: Reinforcement Learning: Theory and Applications: Reinforcement Learning Embedded in Brains and Robots. Austria (2008)Google Scholar