A Spiking Neural Network Based Autonomous Reinforcement Learning Model and Its Application in Decision Making

  • Guixiang Wang
  • Yi Zeng
  • Bo Xu
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10023)


Inspired by biological spike information processing and the multiple brain region coordination mechanism, we propose an autonomous spiking neural network model for decision making. The proposed model is an expansion of the basal ganglia circuitry with automatic environment perception. It automatically constructs environmental states from image inputs. Contributions of this investigation can be summarized as the following: (1) In our model, the simplified Hodgkin-Huxley computing model is developed to achieve calculation efficiency closed to the LIF model and is used to obtain and test the ionic level properties in cognition. (2) A spike based motion perception mechanism is proposed to extract key elements for learning process from raw pixels without large amount of training. We apply our model in the “flappy bird” game and after dozens of training times, it can automatically generate rules to play well in the game. Besides, our model simulates cognitive defects when blocking some of sodium or potassium ion channels in the Hodgkin-Huxley model and this can be considered as a computational exploration on the mechanisms of cognition deep into ionic level.


Spiking neural network Hodgkin-Huxley model Autonomous reinforcement learning Decision making Basal Ganglia 



This study was funded by the Strategic Priority Research Program of the Chinese Academy of Sciences (XDB02060007), and Beijing Municipal Commission of Science and Technology (Z151100000915070, Z161100000216124).


  1. 1.
    Stewart, T.C., Bekolay, T., Eliasmith, C.: Learning to select actions with spiking neurons in the Basal Ganglia. Front. Neurosci. 6(2), 1–14 (2012)Google Scholar
  2. 2.
    Bekolay, T., Eliasmith, C.: A general error-modulated STDP learning rule applied to reinforcement learning in the Basal Ganglia. In: Computational and Systems Neuroscience Conference, Salt Lake City, Utah, pp. 24–27 (2011)Google Scholar
  3. 3.
    Eliasmith, C.: How to Build a Brain, pp. 121–171. Oxford, New York (2013). Reprint editionCrossRefGoogle Scholar
  4. 4.
    Chakravarthy, V.S., Joseph, D., Bapi, R.S.: What do the Basal Ganglia do? Model. Perspect. Biol. Cybern. 103(3), 237–253 (2010)MathSciNetCrossRefzbMATHGoogle Scholar
  5. 5.
    Frank, M.J.: Dynamic dopamine modulation in the Basal Ganglia: a neuro computational account of cognitive deficits in medicated and nonmedicated Parkinsonism. J. Cogn. Neurosci. 17(1), 51–72 (2005)CrossRefGoogle Scholar
  6. 6.
    Utter, A.A., Basso, M.A.: The Basal Ganglia: an overview of circuits and function. Neurosci. Biobehav. Rev. 32(3), 333–342 (2008)CrossRefGoogle Scholar
  7. 7.
    Redgrave, P., Rodriguez, M., Smith, Y., Rodriguez-Oroz, M.C., et al.: Goal-directed and habitual control in the Basal Ganglia: implications for Parkinson’s disease. Nat. Rev. Neurosci. 11, 760–772 (2011)CrossRefGoogle Scholar
  8. 8.
    Stewart, T.C., Choo, X., Eliasmith, C.: Dynamic behavior of a spiking model of action selection in the Basal Ganglia. In: Proceedings of the 10th International Conference on Cognitive Modeling, pp. 5–8 (2010)Google Scholar
  9. 9.
    Frank, M.J.: Hold your horses: a dynamic computational role for the subthalamic nucleus in decision making. Neural Netw. 19(8), 1120–1136 (2006)CrossRefzbMATHGoogle Scholar
  10. 10.
    Gurney, K., Prescott, T.J., Redgrave, P.: A computational model of action selection in the Basal Ganglia. Biol. Cybern. 84(6), 401–410 (2001)CrossRefzbMATHGoogle Scholar
  11. 11.
    Stewart, T.C., Eliasmith, C.: Large-scale synthesis of functional spiking neural circuits. Proc. IEEE 102(5), 881–898 (2014)CrossRefGoogle Scholar
  12. 12.
    MacNeil, D., Eliasmith, C.: Fine-tuning and the stability of recurrent neural networks. Public Lib. Sci. (PLoS One) 6(9), 1–16 (2011)Google Scholar
  13. 13.
    Gurney, K., Prescott, T.J., Wickens, J.R., Redgrave, P.: Computational models of the Basal Ganglia: from robots to membranes. Trends Neurosci. 27(8), 453–459 (2004)CrossRefGoogle Scholar
  14. 14.
    Albin, R.L., Young, A.B., Penney, J.B.: The functional anatomy of Basal Ganglia disorders. Trends Neurosci. 12(10), 366–375 (1989)CrossRefGoogle Scholar
  15. 15.
    Bar-Gad, I., Bergman, H.: Stepping out of the box: information processing in the neural networks of the Basal Ganglia. Curr. Opin. Neurobiol. 11(6), 689–695 (2011)CrossRefGoogle Scholar
  16. 16.
    Iqarashi, J., Shouno, O., Fukai, T., Tsujino, H.: Real-time simulation of a spiking neural network model of the Basal Ganglia circuitry using general purpose computing on graphics processing units. Neural Netw. 24(9), 950–960 (2011)CrossRefGoogle Scholar
  17. 17.
    Cessac, B., Paugam-Moisy, H., Viéville, T.: Overview of facts and issues about neural coding by spikes. J. Physiol. Paris 104(1), 5–18 (2010)CrossRefGoogle Scholar
  18. 18.
    Dayan, P., Abbott, L.F.: Computational and Mathematical Modeling of Neural Systems: Model Neurons I: Neuroelectronic. MIT Press, Cambridge (2003)Google Scholar
  19. 19.
    Izhikevich, E.M.: Dynamical Systems in Neuroscience: The Geometry of Excitability and Bursting. MIT Press, Cambridge (2004)Google Scholar
  20. 20.
    Hodgkin, A.L., Huxley, A.F.: A quantitative description of membrane current and its application to conduction and excitation in nerve. J. Physiol. 117(4), 500–544 (1952)CrossRefGoogle Scholar
  21. 21.
    Nelson, M.E.: Electrophysiological models. In: Databasing the Brain: From Data to Knowledge. Wiley, New York (2004)Google Scholar
  22. 22.
    Gerstner, W., Kistler, W.M.: Spiking Neuron Models. Single Neurons, Populations, Plasticity. Cambridge University Press, Cambridge (2002)CrossRefzbMATHGoogle Scholar
  23. 23.
    Wells, R.B.: Introduction to biological signal processing and computational neuroscience. Moscow (2010)Google Scholar
  24. 24.
    Long, L.N., Fang, G.L.: A review of biologically plausible neuron models for spiking neural networks. In: AIAA InfoTech Aerospace Conference, Atlanta, 20–22 April 2010Google Scholar
  25. 25.
    Weber, C., Elshaw, M., Wermter, S., Triesch, J., Willmot, C.: Reinforcement Learning: Theory and Applications: Reinforcement Learning Embedded in Brains and Robots. Austria (2008)Google Scholar
  26. 26.
    Bohte, S.M., Poutre, H.L., Kok, J.N.: Unsupervised clustering with spiking neurons by sparse temporal coding and multilayer RBF networks. IEEE Trans. Neural Netw. 13(2), 426–435 (2002)CrossRefGoogle Scholar

Copyright information

© Springer International Publishing AG 2016

Authors and Affiliations

  1. 1.Institute of Automation, Chinese Academy of ScienceBeijingChina
  2. 2.Center for Excellence in Brain Science and Intelligence TechnologyChinese Academy of SciencesShanghaiChina
  3. 3.University of Chinese Academy of SciencesBeijingChina

Personalised recommendations