The Role of the Basal Ganglia in Exploratory Behavior in a Model Based on Reinforcement Learning

  • Sridharan Devarajan
  • P. S. Prashanth
  • V. S. Chakravarthy
Part of the Lecture Notes in Computer Science book series (LNCS, volume 3316)

Abstract

We present a model of basal ganglia as a key player in exploratory behavior. The model describes exploration of a virtual rat in a simulated “water pool” experiment. The virtual rat is trained using a reward-based or reinforcement learning paradigm which requires units with stochastic behavior for exploration of the system’s state space. We model the STN-GPe system as a pair of neuronal layers with oscillatory dynamics, exhibiting a variety of dynamic regimes like chaos, traveling waves and clustering. Invoking the property of chaotic systems to explore a state space, we suggest that the complex “exploratory” dynamics of STN-GPe system in conjunction with dopamine-based reward signaling present the two key ingredients of a reinforcement learning system.

References

  1. 1.
    Redgrave, P., Prescott, T.J., Gurney, K.: The basal ganglia: a vertebrate solution to the selection problem? Neuroscience 89, 1009–1023 (1999)CrossRefGoogle Scholar
  2. 2.
    Berns, G.S., Sejnowski, T.J.: A computational model of how the Basal Ganglia produce sequences. Journal of Cognitive Neuroscience 10(1), 108–121 (1998)CrossRefGoogle Scholar
  3. 3.
    Houk, J.C., Davis, J.L., Beiser, D.G.: Models of Information Processing in the Basal Ganglia. MIT Press, Cambridge (1995)Google Scholar
  4. 4.
    Bevan, M.D., Magill, P.J., Terman, D., Bolam, J.P., Wilson, C.J.: Move To The Rhythm: Oscillations In The Subthalamic Nucleus-External Globus Pallidus Network. In: Trends in Neuroscience (2003) (In press)Google Scholar
  5. 5.
    Chirikov, B.: A universal instability of many-dimensional oscillator systems. Phys. Rev. 52, 263–379 (1979)MathSciNetGoogle Scholar
  6. 6.
    Chakravarthy, V.S., Thomas, S.T., Nair, N.: A model for scheduling motor unit recruitment in skeletal muscle. In: International Conference on Theoretical Neurobiology, National Brain Research Center, Gurgoan, February, 24-26 (2003)Google Scholar
  7. 7.
    Gillies, A., Willshaw, D., Li, Z.: Subthalamic-pallidal interactions are critical in determining normal and abnormal functioning of the basal ganglia. In: Proc R Soc Lond B Biol Sci., March 22, vol. 269(1491), pp. 545–551 (2002)Google Scholar
  8. 8.
    Terman, D., Rubin, J.E., Yew, A.C., Wilson, C.J.: Activity Patterns in a Model for the Subthalamopallidal Network of the Basal Ganglia. In: J Neurosci., April 1, vol. 22(7), pp. 2963–2976 (2002)Google Scholar
  9. 9.
    Harner, A.M.: An Introduction to Basal Ganglia Function. Boston University, Boston (1997)Google Scholar
  10. 10.
    Obeso, J.A., Rodriguez-Oroz, M.C., Rodriguez, M., Arbizu, J., Gimenez-Amaya, J.M.: The Basal Ganglia and Disorders of Movement: Pathophysiological Mechanisms. News Physiol Sci. 17, 51–55 (2002)Google Scholar
  11. 11.
    Montague, Dayan, Sejnowski: A Framework for Mesencephalic Dopamine Systems Based on Predictive Hebbian Learning. The Journal of Neuroscience 16(5), 1936–1947 (1996)Google Scholar
  12. 12.
    Sridharan, D.: Human Factors in Aviation: Willed action and its disorders, MTech Thesis, Department of Aerospace Engineering, IIT, Madras, India (2004)Google Scholar
  13. 13.
    Barto, A.G.: Reinforcement Learning. In: Arbib, M.A. (ed.) The Handbook of Brain Theory and Neural Networks (1st Edition), MIT Press, Cambridge (1999)Google Scholar
  14. 14.
    Morris, R.G.M., Garrud, P., Rawlins, J.N.P.: Place navigation impaired in rats with hippocampal lesions. Nature 297, 681–683 (1982)CrossRefGoogle Scholar
  15. 15.
    Skarda, C.A., Freeman, W.J.: How brain makes chaos in order to make sense of the world. Behavioral and Brain Sciences 10, 161–195 (1987)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2004

Authors and Affiliations

  • Sridharan Devarajan
    • 1
  • P. S. Prashanth
    • 1
  • V. S. Chakravarthy
    • 1
  1. 1.Department of Aerospace Engineering and Department of Electrical EngineeringIndian Institute of TechnologyMadrasIndia

Personalised recommendations