Classical Computational Approaches to Modeling the Basal Ganglia

Chapter
Part of the Cognitive Science and Technology book series (CSAT)

Abstract

There have been several modeling approaches to simulate BG structure and function. In this chapter, we discuss major modeling frameworks that have been proposed to simulate many functions of the BG. Many of such modeling studies are classical approaches in the field of BG modeling, which have been repeatedly to simulate many BG functions. In short, here we discuss the following model approaches: dimensionality reduction models, action section selection models, Go/NoGo models, reinforcement learning (RL) models of the basal ganglia, and Actor–Critic models. Importantly, this chapter mainly provides an overview of main architectures used to simulate the BG structure and function. In addition, we discuss many other models, such as those of gait, reaching, and other in the following chapters.

References

  1. Albin, R. L., Young, A. B., & Penney, J. B. (1989). The functional anatomy of basal ganglia disorders. Trends in Neurosciences, 12(10), 366–375.CrossRefGoogle Scholar
  2. Amos, A. (2000). A computational model of information processing in the frontal cortex and basal ganglia. Journal of Cognitive Neuroscience, 12(3), 505–519.CrossRefGoogle Scholar
  3. Aron, A. R., & Poldrack, R. A. (2006). Cortical and subcortical contributions to Stop signal response inhibition: Role of the subthalamic nucleus. Journal of Neuroscience, 26(9), 2424–2433.CrossRefGoogle Scholar
  4. Baldassarre, G. (2002). A modular neural-network model of the basal ganglia’s role in learning and selecting motor behaviours. Journal of Cognitive Systems Research, 3, 5–13.CrossRefGoogle Scholar
  5. Baldassarre, G., & Parisi, D. (2000). Classical and instrumental conditioning: From laboratory phenomena to integrated mechanisms for adaptation.Google Scholar
  6. Balleine, B. W., Delgado, M. R., & Hikosaka, O. (2007). The role of the dorsal striatum in reward and decision-making. Journal of Neuroscience, 27(31), 8161–8165.  https://doi.org/10.1523/JNEUROSCI.1554-07.2007.CrossRefGoogle Scholar
  7. Bar-Gad, I., Havazelet-Heimer, G., Goldberg, J. A., Ruppin, E., & Bergman, H. (2000). Reinforcement-driven dimensionality reduction-a model for information processing in the basal ganglia. Journal of Basic and Clinical Physiology and Pharmacology, 11(4), 305–320.CrossRefGoogle Scholar
  8. Bar-Gad, I., Morris, G., & Bergman, H. (2003). Information processing, dimensionality reduction and reinforcement learning in the basal ganglia. Progress in Neurobiology, 71(6), 439–473.  https://doi.org/10.1016/j.pneurobio.2003.12.001.CrossRefGoogle Scholar
  9. Barto, A. G. (1995). Adaptive critics and the basal ganglia. In J. C. Houk, J. L. Davis & D. G. Beiser (Eds.), Models of information processing in the basal ganglia (pp. xii, 382p). Cambridge, MA: MIT Press.Google Scholar
  10. Barto, A. G. (2003). Reinforcement learning. In M. A. Arbib (Ed.), The handbook of brain theory and neural networks (pp. 963–968). Cambridge, MA: MIT Press.Google Scholar
  11. Baston, C., & Ursino, M. (2015). A biologically inspired computational model of basal ganglia in action selection. Computational Intelligence and Neuroscience, 2015, 187417.  https://doi.org/10.1155/2015/187417.CrossRefGoogle Scholar
  12. Baunez, C., & Robbins, T. W. (1997). Bilateral lesions of the subthalamic nucleus induce multiple deficits in an attentional task in rats. European Journal of Neuroscience, 9(10), 2086–2099.CrossRefGoogle Scholar
  13. Beiser, D. G., & Houk, J. C. (1998). Model of cortical-basal ganglionic processing: Encoding the serial order of sensory events. Journal of Neurophysiology, 79(6), 3168–3188.  https://doi.org/10.1152/jn.1998.79.6.3168.CrossRefGoogle Scholar
  14. Benazzouz, A., & Hallett, M. (2000). Mechanism of action of deep brain stimulation. Neurology, 55(12 Suppl 6), S13–S16.Google Scholar
  15. Berns, G. S., & Sejnowski, T. J. (1996). How the basal ganglia make decisions. In A. Damasio, H. Damasio, & Y. Christen (Eds.), The neurobiology of decision making. Berlin: Springer.Google Scholar
  16. Blenkinsop, A., Anderson, S., & Gurney, K. (2017). Frequency and function in the basal ganglia: The origins of beta and gamma band activity. Journal of Physiology, 595(13), 4525–4548.  https://doi.org/10.1113/JP273760.CrossRefGoogle Scholar
  17. Bogacz, R., & Gurney, K. (2007). The basal ganglia and cortex implement optimal decision making between alternative actions. Neural Computation, 19(2), 442–477.MathSciNetCrossRefMATHGoogle Scholar
  18. Braver, T. S., Barch, D. M., Keys, B. A., Carter, C. S., Cohen, J. D., Kaye, J. A., … Reed, B. R. (2001). Context processing in older adults: Evidence for a theory relating cognitive control to neurobiology in healthy aging. Journal of Experimental Psychology: General, 130(4), 746–763.Google Scholar
  19. Buxton, D., Bracci, E., Overton, P. G., & Gurney, K. (2017). Striatal neuropeptides enhance selection and rejection of sequential actions. Frontiers in Computational Neuroscience, 11, 62.  https://doi.org/10.3389/fncom.2017.00062.CrossRefGoogle Scholar
  20. Chakravarthy, V., Joseph, D., & Bapi, R. S. (2010). What do the basal ganglia do? A modeling perspective. Biological Cybernetics, 103(3), 237–253.MathSciNetCrossRefMATHGoogle Scholar
  21. Cohen, J. D., Barch, D. M., Carter, C., & Servan-Schreiber, D. (1999). Context-processing deficits in schizophrenia: Converging evidence from three theoretically motivated cognitive tasks. Journal of Abnormal Psychology, 108(1), 120–133.CrossRefGoogle Scholar
  22. Cohen, J. D., & Servan-Schreiber, D. (1992). Context, cortex, and dopamine: A connectionist approach to behavior and biology in schizophrenia. Psychological Review, 99(1), 45–77.CrossRefGoogle Scholar
  23. Czernecki, V., Pillon, B., Houeto, J. L., Pochon, J. B., Levy, R., & Dubois, B. (2002). Motivation, reward, and Parkinson’s disease: Influence of dopatherapy. Neuropsychologia, 40(13), 2257–2267.CrossRefGoogle Scholar
  24. Delgado, M. R. (2007). Reward-related responses in the human striatum. Annals of the New York Academy of Sciences, 1104(1), 70–88.CrossRefGoogle Scholar
  25. Delgado, M. R., Miller, M. M., Inati, S., & Phelps, E. A. (2005). An fMRI study of reward-related probability learning. Neuroimage, 24(3), 862–873.CrossRefGoogle Scholar
  26. Frank, M. J. (2005). Dynamic dopamine modulation in the basal ganglia: A neurocomputational account of cognitive deficits in medicated and nonmedicated Parkinsonism. Journal of Cognitive Neuroscience, 17(1), 51–72.CrossRefGoogle Scholar
  27. Frank, M. J., Loughry, B., & O’Reilly, R. C. (2001). Interactions between frontal cortex and basal ganglia in working memory: A computational model. Cognitive, Affective, & Behavioral Neuroscience, 1(2), 137–160.CrossRefGoogle Scholar
  28. Frank, M. J., & O’Reilly, R. C. (2006). A mechanistic account of striatal dopamine function in human cognition: Psychopharmacological studies with cabergoline and haloperidol. Behavioral Neuroscience, 120(3), 497–517.CrossRefGoogle Scholar
  29. Frank, M. J., Seeberger, L. C., & O’Reilly, R. C. (2004). By carrot or by stick: Cognitive reinforcement learning in parkinsonism. Science, 306(5703), 1940–1943.CrossRefGoogle Scholar
  30. Funkiewiez, A., Ardouin, C., Krack, P., Fraix, V., Van Blercom, N., Xie, J., … Pollak, P. (2003). Acute psychotropic effects of bilateral subthalamic nucleus stimulation and levodopa in Parkinson’s disease. Movement Disorders, 18(5), 524–530.Google Scholar
  31. Gershman, S. J., Moustafa, A. A., & Ludvig, E. A. (2014). Time representation in reinforcement learning models of the basal ganglia. Frontiers in Computational Neuroscience, 7, 194.  https://doi.org/10.3389/fncom.2013.00194.CrossRefGoogle Scholar
  32. Gurney, K., Prescott, T. J., & Redgrave, P. (2001a). A computational model of action selection in the basal ganglia. I. A new functional anatomy. Biological Cybernetics, 84(6), 401–410.CrossRefMATHGoogle Scholar
  33. Gurney, K., Prescott, T. J., & Redgrave, P. (2001b). A computational model of action selection in the basal ganglia. II. Analysis and simulation of behaviour. Biological Cybernetics, 84(6), 411–423.CrossRefMATHGoogle Scholar
  34. Hershey, T., Revilla, F. J., Wernle, A., Gibson, P. S., Dowling, J. L., & Perlmutter, J. S. (2004). Stimulation of STN impairs aspects of cognitive control in PD. Neurology, 62(7), 1110–1114.CrossRefGoogle Scholar
  35. Houk, J. C. (1995a). Information processing in modular circuits linking basal ganglia and cerebral Cortex. In J. C. Houk, J. L. Davis & D. G. Beiser (Eds.), Models of information processing in the basal ganglia (pp. xii, 382p). Cambridge, MA: MIT Press.Google Scholar
  36. Houk, J. C. (1995b). A model of how the basal ganglia generate and use neural signals that predict reinforcement. In J. C. Houk, J. L. Davis & D. G. Beiser (Eds.), Models of information processing in the basal ganglia (pp. xii, 382p). Cambridge, MA: MIT Press.Google Scholar
  37. Houk, J. C. (2005). Agents of the mind. Biological Cybernetics, 92(6), 427–437.CrossRefMATHGoogle Scholar
  38. Houk, J. C., Adams, J. L., & Barto, A. G. (1995). A model of how the basal ganglia generate and use neural signals that predict reinforcement. Models of Information Processing in the Basal Ganglia, 249–270.Google Scholar
  39. Humphries, M. D., Khamassi, M., & Gurney, K. (2012). Dopaminergic control of the exploration-exploitation trade-off via the basal ganglia. Frontiers in Neuroscience, 6.Google Scholar
  40. Humphries, M. D., Stewart, R. D., & Gurney, K. N. (2006). A physiologically plausible model of action selection and oscillatory activity in the basal ganglia. The Journal of Neuroscience, 26(50), 12921–12942.CrossRefGoogle Scholar
  41. Joel, D., Niv, Y., & Ruppin, E. (2002). Actor–critic models of the basal ganglia: New anatomical and computational perspectives. Neural Networks, 15(4), 535–547.CrossRefGoogle Scholar
  42. Karachi, C., Yelnik, J., Tande, D., Tremblay, L., Hirsch, E. C., & Francois, C. (2005). The pallidosubthalamic projection: An anatomical substrate for nonmotor functions of the subthalamic nucleus in primates. Movement Disorders, 20(2), 172–180.CrossRefGoogle Scholar
  43. Khamassi, M., Girard, B., Berthoz, A., & Guillot, A. (2004). Comparing three Critic models of reinforcement learning in the basal ganglia connected to a detailed actor part in a S-R task. Paper presented at the Proceedings of the Eighth International Conference on Intelligent Autonomous Systems IAS-8, Amsterdam, The Netherlands.Google Scholar
  44. Kim, T., Hamade, K. C., Todorov, D., Barnett, W. H., Capps, R. A., Latash, E. M., … Molkov, Y. I. (2017). Reward based motor adaptation mediated by basal ganglia. Frontiers in Computational Neuroscience, 11, 19.  https://doi.org/10.3389/fncom.2017.00019.
  45. Krack, P., Kumar, R., Ardouin, C., Dowsey, P. L., McVicker, J. M., Benabid, A. L., & Pollak, P. (2001). Mirthful laughter induced by subthalamic nucleus stimulation. Movement Disorders, 16(5), 867–875.Google Scholar
  46. Limousin, P., Greene, J., Pollak, P., Rothwell, J., Benabid, A. L., & Frackowiak, R. (1997). Changes in cerebral activity pattern due to subthalamic nucleus or internal pallidum stimulation in Parkinson’s disease. Annals of Neurology, 42(3), 283–291.CrossRefGoogle Scholar
  47. Meissner, W., Leblois, A., Hansel, D., Bioulac, B., Gross, C. E., Benazzouz, A., & Boraud, T. (2005). Subthalamic high frequency stimulation resets subthalamic firing and reduces abnormal oscillations. Brain, 128(10), 2372–2382.Google Scholar
  48. Middleton, F. A., & Strick, P. L. (2000). Basal ganglia output and cognition: Evidence from anatomical, behavioral, and clinical studies. Brain and Cognition, 42(2), 183–200.CrossRefGoogle Scholar
  49. Middleton, F. A., & Strick, P. L. (2002). Basal-ganglia ‘projections’ to the prefrontal cortex of the primate. Cerebral Cortex, 12(9), 926–935.CrossRefGoogle Scholar
  50. Mink, J. W. (1996). The basal ganglia: Focused selection and inhibition of competing motor programs. Progress in Neurobiology, 50(4), 381.CrossRefGoogle Scholar
  51. Moustafa, A. A., & Gluck, M. A. (2011a). A neurocomputational model of dopamine and prefrontal-striatal interactions during multicue category learning by Parkinson patients. Journal of Cognitive Neuroscience, 23(1), 151–167.  https://doi.org/10.1162/jocn.2010.21420.CrossRefGoogle Scholar
  52. Moustafa, A. A., & Gluck, M. A. (2011b). Computational cognitive models of prefrontal-striatal-hippocampal interactions in Parkinson’s disease and schizophrenia. Neural Netw, 24(6), 575–591.  https://doi.org/10.1016/j.neunet.2011.02.006.CrossRefGoogle Scholar
  53. Moustafa, A. A., & Maida, A. S. (2007). Using TD learning to simulate working memory performance in a model of the prefrontal cortex and basal ganglia. Cognitive Systems Research, 8, 262–281.CrossRefGoogle Scholar
  54. Moustafa, A. A., Cohen, M. X., Sherman, S. J., & Frank, M. J. (2008). A role for dopamine in temporal decision making and reward maximization in parkinsonism. Journal of Neuroscience, 28(47), 12294–12304.  https://doi.org/10.1523/JNEUROSCI.3116-08.2008.CrossRefGoogle Scholar
  55. Moustafa, A. A., Herzallah, M. M., & Gluck, M. A. (2014). A model of reversal learning and working memory in medicated and unmedicated patients with Parkinson’s disease. Journal of Mathematical Psychology, 59, 120–131.MathSciNetCrossRefMATHGoogle Scholar
  56. Niv, Y. (2009). Reinforcement learning in the brain. Journal of Mathematical Psychology, 53(3), 139–154.MathSciNetCrossRefMATHGoogle Scholar
  57. O’Doherty, J., Dayan, P., Schultz, J., Deichmann, R., Friston, K., & Dolan, R. J. (2004). Dissociable roles of ventral and dorsal striatum in instrumental conditioning. Science, 304(5669), 452–454.CrossRefGoogle Scholar
  58. O’Reilly, R. C., & Frank, M. J. (2006). Making working memory work: A computational model of learning in the prefrontal cortex and basal ganglia. Neural Computation, 18(2), 283–328.MathSciNetCrossRefMATHGoogle Scholar
  59. O’Reilly, R. C. (2003). Making working memory work: A computational model of learning in the prefrontal cortex and basal ganglia. ICS Technical Report, (pp. 1–23).Google Scholar
  60. Prescott, T. J. (2002). Basal ganglia. In M. A. Arbib (Ed.), The handbook of brain theory and neural networks (pp. xvii, 1290p). Cambridge, MA: MIT Press.Google Scholar
  61. Redgrave, P., Prescott, T. J., & Gurney, K. (1999). The basal ganglia: A vertebrate solution to the selection problem? Neuroscience, 89(4), 1009–1023.CrossRefGoogle Scholar
  62. Reynolds, J. N. J., & Wickens, J. R. (2002). Dopamine-dependent plasticity of corticostriatal synapses. Neural Networks, 15(4), 507–521.CrossRefGoogle Scholar
  63. Saint-Cyr, J. A., Trepanier, L. L., Kumar, R., Lozano, A. M., & Lang, A. E. (2000). Neuropsychological consequences of chronic bilateral stimulation of the subthalamic nucleus in Parkinson’s disease. Brain, 123(10), 2091–2108.CrossRefGoogle Scholar
  64. Samejima, K., Ueda, Y., Doya, K., & Kimura, M. (2005). Representation of action-specific reward values in the striatum. Science, 310(5752), 1337–1340.CrossRefGoogle Scholar
  65. Schultz, W. (1998). Predictive reward signal of dopamine neurons. Journal of Neurophysiology, 80(1), 1–27.MathSciNetCrossRefGoogle Scholar
  66. Schultz, W., Apicella, P., & Ljungberg, T. (1993). Responses of monkey dopamine neurons to reward and conditioned stimuli during successive steps of learning a delayed response task. Journal of Neuroscience, 13(3), 900–913.Google Scholar
  67. Schultz, W., Dayan, P., & Montague, P. R. (1997). A neural substrate of prediction and reward. Science, 275(5306), 1593–1599.CrossRefGoogle Scholar
  68. Seo, M., Lee, E., & Averbeck, B. B. (2012). Action selection and action value in frontal-striatal circuits. Neuron, 74(5), 947–960.  https://doi.org/10.1016/j.neuron.2012.03.037.CrossRefGoogle Scholar
  69. Servan-Schreiber, D., Cohen, J. D., & Steingard, S. (1996). Schizophrenic deficits in the processing of context. A test of a theoretical model. Archives of General Psychiatry, 53(12), 1105–1112.CrossRefGoogle Scholar
  70. Shivkumar, S., Muralidharan, V., & Chakravarthy, V. S. (2017). A biologically plausible architecture of the striatum to solve context-dependent reinforcement learning tasks. Frontiers in Neural Circuits, 11(45).  https://doi.org/10.3389/fncir.2017.00045.
  71. Shohamy, D., Myers, C. E., Geghman, K. D., Sage, J., & Gluck, M. A. (2006). L-dopa impairs learning, but spares generalization, Parkinson’s disease. Neuropsychologia, 44(5), 774–784.CrossRefGoogle Scholar
  72. Stocco, A. (2017). A biologically plausible action selection system for cognitive architectures: Implications of basal ganglia anatomy for learning and decision-making models. Cognitive Science  https://doi.org/10.1111/cogs.12506.
  73. Suri, R. E., Bargas, J., & Arbib, M. A. (2001). Modeling functions of striatal dopamine modulation in learning and planning. Neuroscience, 103(1), 65–85.CrossRefGoogle Scholar
  74. Suri, R. E., & Schultz, W. (1998). Learning of sequential movements by neural network model with dopamine-like reinforcement signal. Experimental Brain Research, 121(3), 350–354.CrossRefGoogle Scholar
  75. Suri, R. E., & Schultz, W. (1999). A neural network model with dopamine-like reinforcement signal that learns a spatial delayed response task. Neuroscience, 91(3), 871–890.CrossRefGoogle Scholar
  76. Surmeier, D. J., Ding, J., Day, M., Wang, Z., & Shen, W. (2007). D1 and D2 dopamine-receptor modulation of striatal glutamatergic signaling in striatal medium spiny neurons. Trends in Neurosciences, 30(5), 228–235.CrossRefGoogle Scholar
  77. Sutton, R. S., & Barto, A. G. (1998). Reinforcement learning: An introduction (Vol. 1). Cambridge: Cambridge University Press.Google Scholar
  78. Tanaka, S. C., Schweighofer, N., Asahi, S., Shishida, K., Okamoto, Y., Yamawaki, S., & Doya, K. (2007). Serotonin differentially regulates short- and long-term prediction of rewards in the ventral and dorsal striatum. PLoS One, 2(12), e1333.  https://doi.org/10.1371/journal.pone.0001333.
  79. Tian, L., Arnold, M., Sejnowski, T., & Jabri, M. (2003). A biologically inspired computational model of the block copying task. Paper presented at the Proceedings of the third international workshop on Epigenetic robotics, Lund University Cognitive Studies.Google Scholar
  80. Wickens, J., & Kötter, R. (1995). Cellular models of reinforcement.Google Scholar
  81. Wickens, J. R. (1997). Basal Ganglia: Structure and computations [Invited Review]. Network: Computation in Neural Systems, 8, R77–R109.CrossRefMATHGoogle Scholar
  82. Wickens, J. R., Horvitz, J. C., Costa, R. M., & Killcross, S. (2007). Dopaminergic mechanisms in actions and habits. Journal of Neuroscience, 27(31), 8181–8183.CrossRefGoogle Scholar
  83. Wilson, C. J. (2004). Basal ganglia. In G. M. Shepherd (Ed.), The synaptic organization of the 136 brain (pp. 361–413). New York: Oxford University Press.Google Scholar
  84. Wise, R. A. (2004). Dopamine, learning and motivation. Nature Reviews Neuroscience, 5(6), 483–494.CrossRefGoogle Scholar
  85. Wise, R. A., & Rompre, P.-P. (1989). Brain dopamine and reward. Annual Review of Psychology, 40(1), 191–225.CrossRefGoogle Scholar

Copyright information

© Springer Nature Singapore Pte Ltd. 2018

Authors and Affiliations

  1. 1.School of Social Sciences and Psychology & Marcs Institute for Brain and BehaviourWestern Sydney UniversitySydneyAustralia
  2. 2.Department of Biotechnology, Bhupat and Jyoti Mehta School of BiosciencesIndian Institute of TechnologyMadras, ChennaiIndia

Personalised recommendations