Skip to main content

A Spiking Neural Network Based Autonomous Reinforcement Learning Model and Its Application in Decision Making

Part of the Lecture Notes in Computer Science book series (LNAI,volume 10023)

Abstract

Inspired by biological spike information processing and the multiple brain region coordination mechanism, we propose an autonomous spiking neural network model for decision making. The proposed model is an expansion of the basal ganglia circuitry with automatic environment perception. It automatically constructs environmental states from image inputs. Contributions of this investigation can be summarized as the following: (1) In our model, the simplified Hodgkin-Huxley computing model is developed to achieve calculation efficiency closed to the LIF model and is used to obtain and test the ionic level properties in cognition. (2) A spike based motion perception mechanism is proposed to extract key elements for learning process from raw pixels without large amount of training. We apply our model in the “flappy bird” game and after dozens of training times, it can automatically generate rules to play well in the game. Besides, our model simulates cognitive defects when blocking some of sodium or potassium ion channels in the Hodgkin-Huxley model and this can be considered as a computational exploration on the mechanisms of cognition deep into ionic level.

Keywords

  • Spiking neural network
  • Hodgkin-Huxley model
  • Autonomous reinforcement learning
  • Decision making
  • Basal Ganglia

This is a preview of subscription content, access via your institution.

Buying options

Chapter
USD   29.95
Price excludes VAT (USA)
  • DOI: 10.1007/978-3-319-49685-6_12
  • Chapter length: 13 pages
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
eBook
USD   59.99
Price excludes VAT (USA)
  • ISBN: 978-3-319-49685-6
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
Softcover Book
USD   79.99
Price excludes VAT (USA)
Fig. 1.
Fig. 2.
Fig. 3.
Fig. 4.
Fig. 5.
Fig. 6.
Fig. 7.
Fig. 8.

Notes

  1. 1.

    Chris Eliasmith gave an invited keynote talk at the 2014 Computational Neuroscience Meeting (CNS 2014) on the effect of sodium channel for learning in SPAUN. This talk inspired our investigation introduced in this subsection.

References

  1. Stewart, T.C., Bekolay, T., Eliasmith, C.: Learning to select actions with spiking neurons in the Basal Ganglia. Front. Neurosci. 6(2), 1–14 (2012)

    Google Scholar 

  2. Bekolay, T., Eliasmith, C.: A general error-modulated STDP learning rule applied to reinforcement learning in the Basal Ganglia. In: Computational and Systems Neuroscience Conference, Salt Lake City, Utah, pp. 24–27 (2011)

    Google Scholar 

  3. Eliasmith, C.: How to Build a Brain, pp. 121–171. Oxford, New York (2013). Reprint edition

    CrossRef  Google Scholar 

  4. Chakravarthy, V.S., Joseph, D., Bapi, R.S.: What do the Basal Ganglia do? Model. Perspect. Biol. Cybern. 103(3), 237–253 (2010)

    MathSciNet  CrossRef  MATH  Google Scholar 

  5. Frank, M.J.: Dynamic dopamine modulation in the Basal Ganglia: a neuro computational account of cognitive deficits in medicated and nonmedicated Parkinsonism. J. Cogn. Neurosci. 17(1), 51–72 (2005)

    CrossRef  Google Scholar 

  6. Utter, A.A., Basso, M.A.: The Basal Ganglia: an overview of circuits and function. Neurosci. Biobehav. Rev. 32(3), 333–342 (2008)

    CrossRef  Google Scholar 

  7. Redgrave, P., Rodriguez, M., Smith, Y., Rodriguez-Oroz, M.C., et al.: Goal-directed and habitual control in the Basal Ganglia: implications for Parkinson’s disease. Nat. Rev. Neurosci. 11, 760–772 (2011)

    CrossRef  Google Scholar 

  8. Stewart, T.C., Choo, X., Eliasmith, C.: Dynamic behavior of a spiking model of action selection in the Basal Ganglia. In: Proceedings of the 10th International Conference on Cognitive Modeling, pp. 5–8 (2010)

    Google Scholar 

  9. Frank, M.J.: Hold your horses: a dynamic computational role for the subthalamic nucleus in decision making. Neural Netw. 19(8), 1120–1136 (2006)

    CrossRef  MATH  Google Scholar 

  10. Gurney, K., Prescott, T.J., Redgrave, P.: A computational model of action selection in the Basal Ganglia. Biol. Cybern. 84(6), 401–410 (2001)

    CrossRef  MATH  Google Scholar 

  11. Stewart, T.C., Eliasmith, C.: Large-scale synthesis of functional spiking neural circuits. Proc. IEEE 102(5), 881–898 (2014)

    CrossRef  Google Scholar 

  12. MacNeil, D., Eliasmith, C.: Fine-tuning and the stability of recurrent neural networks. Public Lib. Sci. (PLoS One) 6(9), 1–16 (2011)

    Google Scholar 

  13. Gurney, K., Prescott, T.J., Wickens, J.R., Redgrave, P.: Computational models of the Basal Ganglia: from robots to membranes. Trends Neurosci. 27(8), 453–459 (2004)

    CrossRef  Google Scholar 

  14. Albin, R.L., Young, A.B., Penney, J.B.: The functional anatomy of Basal Ganglia disorders. Trends Neurosci. 12(10), 366–375 (1989)

    CrossRef  Google Scholar 

  15. Bar-Gad, I., Bergman, H.: Stepping out of the box: information processing in the neural networks of the Basal Ganglia. Curr. Opin. Neurobiol. 11(6), 689–695 (2011)

    CrossRef  Google Scholar 

  16. Iqarashi, J., Shouno, O., Fukai, T., Tsujino, H.: Real-time simulation of a spiking neural network model of the Basal Ganglia circuitry using general purpose computing on graphics processing units. Neural Netw. 24(9), 950–960 (2011)

    CrossRef  Google Scholar 

  17. Cessac, B., Paugam-Moisy, H., Viéville, T.: Overview of facts and issues about neural coding by spikes. J. Physiol. Paris 104(1), 5–18 (2010)

    CrossRef  Google Scholar 

  18. Dayan, P., Abbott, L.F.: Computational and Mathematical Modeling of Neural Systems: Model Neurons I: Neuroelectronic. MIT Press, Cambridge (2003)

    Google Scholar 

  19. Izhikevich, E.M.: Dynamical Systems in Neuroscience: The Geometry of Excitability and Bursting. MIT Press, Cambridge (2004)

    Google Scholar 

  20. Hodgkin, A.L., Huxley, A.F.: A quantitative description of membrane current and its application to conduction and excitation in nerve. J. Physiol. 117(4), 500–544 (1952)

    CrossRef  Google Scholar 

  21. Nelson, M.E.: Electrophysiological models. In: Databasing the Brain: From Data to Knowledge. Wiley, New York (2004)

    Google Scholar 

  22. Gerstner, W., Kistler, W.M.: Spiking Neuron Models. Single Neurons, Populations, Plasticity. Cambridge University Press, Cambridge (2002)

    CrossRef  MATH  Google Scholar 

  23. Wells, R.B.: Introduction to biological signal processing and computational neuroscience. Moscow (2010)

    Google Scholar 

  24. Long, L.N., Fang, G.L.: A review of biologically plausible neuron models for spiking neural networks. In: AIAA InfoTech Aerospace Conference, Atlanta, 20–22 April 2010

    Google Scholar 

  25. Weber, C., Elshaw, M., Wermter, S., Triesch, J., Willmot, C.: Reinforcement Learning: Theory and Applications: Reinforcement Learning Embedded in Brains and Robots. Austria (2008)

    Google Scholar 

  26. Bohte, S.M., Poutre, H.L., Kok, J.N.: Unsupervised clustering with spiking neurons by sparse temporal coding and multilayer RBF networks. IEEE Trans. Neural Netw. 13(2), 426–435 (2002)

    CrossRef  Google Scholar 

Download references

Acknowledgements

This study was funded by the Strategic Priority Research Program of the Chinese Academy of Sciences (XDB02060007), and Beijing Municipal Commission of Science and Technology (Z151100000915070, Z161100000216124).

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Yi Zeng .

Editor information

Editors and Affiliations

Rights and permissions

Reprints and Permissions

Copyright information

© 2016 Springer International Publishing AG

About this paper

Cite this paper

Wang, G., Zeng, Y., Xu, B. (2016). A Spiking Neural Network Based Autonomous Reinforcement Learning Model and Its Application in Decision Making. In: Liu, CL., Hussain, A., Luo, B., Tan, K., Zeng, Y., Zhang, Z. (eds) Advances in Brain Inspired Cognitive Systems. BICS 2016. Lecture Notes in Computer Science(), vol 10023. Springer, Cham. https://doi.org/10.1007/978-3-319-49685-6_12

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-49685-6_12

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-49684-9

  • Online ISBN: 978-3-319-49685-6

  • eBook Packages: Computer ScienceComputer Science (R0)