Advertisement

Psychological and Neuroscientific Connections with Reinforcement Learning

  • Ashvin Shah
Part of the Adaptation, Learning, and Optimization book series (ALO, volume 12)

Abstract

The field of Reinforcement Learning (RL) was inspired in large part by research in animal behavior and psychology. Early research showed that animals can, through trial and error, learn to execute behavior that would eventually lead to some (presumably satisfactory) outcome, and decades of subsequent research was (and is still) aimed at discovering the mechanisms of this learning process. This chapter describes behavioral and theoretical research in animal learning that is directly related to fundamental concepts used in RL. It then describes neuroscientific research that suggests that animals and many RL algorithms use very similar learning mechanisms. Along the way, I highlight ways that research in computer science contributes to and can be inspired by research in psychology and neuroscience.

Keywords

Basal Ganglion Classical Conditioning Dopamine Neuron Ventral Striatum Associative Strength 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Aldridge, J.W., Berridge, K.C.: Coding of serial order by neostriatal neurons: a “natural action” approach to movement sequence. The Journal of Neuroscience 18, 2777–2787 (1998)Google Scholar
  2. Alexander, G.E., DeLong, M.R., Strick, P.L.: Parallel organization of functionally segregated circuits linking basal ganglia and cortex. Annual Review of Neuroscience 9, 357–381 (1986)Google Scholar
  3. Ashby, F.G., Ennis, J., Spiering, B.: A neurobiological theory of automaticity in perceptual categorization. Psychological Review 114, 632–656 (2007)Google Scholar
  4. Ashby, F.G., Turner, B.O., Horvitz, J.C.: Cortical and basal ganglia contributions to habit learning and automaticity. Trends in Cognitive Sciences 14, 208–215 (2010)Google Scholar
  5. Atallah, H.E., Lopez-Paniagua, D., Rudy, J.W., O’Reilly, R.C.: Separate neural substrates for skill learning and performance in ventral and dorsal striatum. Nature Neuroscience 10, 126–131 (2007)Google Scholar
  6. Balleine, B.W., O’Dohrety, J.P.: Human and rodent homologies in action control: Corticostriatal determinants of goal-directed and habitual action. Neuropsychopharmacology 35, 48–69 (2010)Google Scholar
  7. Balleine, B.W., Delgado, M.R., Hikosaka, O.: The role of the dorsal striatum in reward and decision-making. The Journal of Neuroscience 27, 8161–8165 (2007)Google Scholar
  8. Balleine, B.W., Liljeholm, M., Ostlund, S.B.: The integrative function of the basal ganglia in instrumental conditioning. Behavioural Brain Research 199, 43–52 (2009)Google Scholar
  9. Bar-Gad, I., Morris, G., Bergman, H.: Information processing, dimensionality reduction, and reinforcement learning in the basal ganglia. Progress in Neurobiology 71, 439–473 (2003)Google Scholar
  10. Barnes, T.D., Kubota, Y., Hu, D., Jin, D.Z., Graybiel, A.M.: Activity of striatal neurons reflects dynamic encoding and recoding of procedural memories. Nature 437, 1158–1161 (2005)Google Scholar
  11. Barto, A.G.: Learning by statistical cooperation of self-interested neuron-like computing elements. Human Neurobiology 4, 229–256 (1985)Google Scholar
  12. Barto, A.G.: Adaptive critics and the basal ganglia. In: Houk, J.C., Davis, J.L., Beiser, D.G. (eds.) Models of Information Processing in the Basal Ganglia, ch. 11, pp. 215–232. MIT Press, Cambridge (1995)Google Scholar
  13. Barto, A.G., Mahadevan, S.: Recent advances in hierarchical reinforcement learning. Discrete Event Dynamic Systems 13, 341–379 (2003)MathSciNetGoogle Scholar
  14. Barto, A.G., Sutton, R.S.: Simulation of anticipatory responses in classical conditioning by a neuron-like adaptive element. Behavioral Brain Research 4, 221–235 (1982)Google Scholar
  15. Barto, A.G., Sutton, R.S., Anderson, C.W.: Neuronlike elements that can solve difficult learning control problems. IEEE Transactions on Systems, Man, and Cybernectics 13, 835–846 (1983)Google Scholar
  16. Bayer, H.M., Glimcher, P.W.: Midbrain dopamine neurons encode a quantitative reward prediction error signal. Neuron 47, 129–141 (2005)Google Scholar
  17. Belin, D., Jonkman, S., Dickinson, A., Robbins, T.W., Everitt, B.J.: Parallel and interactive learning processes within the basal ganglia: relevance for the understanding of addiction. Behavioural Brain Research 199, 89–102 (2009)Google Scholar
  18. Berridge, K.C.: The debate over dopamine’s role in reward: The case for incentive salience. Psychopharmacology 191, 391–431 (2007)Google Scholar
  19. Berridge, K.C., Robinson, T.E.: What is the role of dopamine in reward: Hedonic impact, reward learning, or incentive salience? Brain Research Reviews 28, 309–369 (1998)Google Scholar
  20. Berridge, K.C., Robinson, T.E., Aldridge, J.W.: Dissecting components of reward: ’Liking,’ ’wanting,’ and learning. Current Opinion in Pharamacology 9, 65–73 (2009)Google Scholar
  21. Björklund, A., Dunnett, S.B.: Dopamine neuron systems in the brain: an update. Trends in Neurosciences 30, 194–202 (2007)Google Scholar
  22. Bogacz, R., Gurney, K.: The basal ganglia and cortex implement optimal decision making between alternative actions. Neural Computation 19, 442–477 (2007)MathSciNetGoogle Scholar
  23. Botvinick, M.M., Niv, Y., Barto, A.G.: Hierarchically organized behavior and its neural foundations: A reinforcement-learning perspective. Cognition 113, 262–280 (2009)Google Scholar
  24. Brandon, S.E., Vogel, E.G., Wagner, A.R.: Computational theories of classical conditioning. In: Moore, J.W. (ed.) A Neuroscientist’s Guide to Classical Conditioning, ch. 7, pp. 232–310. Springer, New York (2002)Google Scholar
  25. Bromberg-Martin, E.S., Matsumoto, M., Hikosaka, O.: Dopamine in motivational control: Rewarding, aversive, and alerting. Neuron 68, 815–834 (2010)Google Scholar
  26. Brown, P.L., Jenkins, H.M.: Auto-shaping of the pigeon’s key-peck. Journal of the Experimental Analysis of Behavior 11, 1–8 (1968)Google Scholar
  27. Calabresi, P., Picconi, B., Tozzi, A., DiFilippo, M.: Dopamine-mediated regulation of corticostriatal synaptic plasticity. Trends in Neuroscience 30, 211–219 (2007)Google Scholar
  28. Cannon, C.M., Palmiter, R.D.: Reward without dopamine. Journal of Neuroscience 23, 10,827–10,831 (2003)Google Scholar
  29. Cardinal, R.N., Parkinson, J.A., Hall, J., Everitt, B.J.: Emotion and motivation: The role of the amygdala, ventral striatum, and prefrontal cortex. Neuroscience and Biobehavioural Reviews 26, 321–352 (2002)Google Scholar
  30. Cohen, M.X.: Neurocomputational mechanisms of reinforcement-guided learning in humans: a review. Cognitive, Affective, and Behavioral Neuroscience 8, 113–125 (2008)Google Scholar
  31. Cohen, M.X., Frank, M.J.: Neurocomputational models of the basal ganglia in learning, memory, and choice. Behavioural Brain Research 199, 141–156 (2009)Google Scholar
  32. Corrado, G., Doya, K.: Understanding neural coding through the model-based analysis of decision-making. The Journal of Neuroscience 27, 8178–8180 (2007)Google Scholar
  33. Daw, N.D., Doya, K.: The computational neurobiology of learning and reward. Current Opinion in Neurobiology 16, 199–204 (2006)Google Scholar
  34. Daw, N.D., Touretzky, D.S.: Long-term reward prediction in TD models of the dopamine system. Neural Computation 14, 2567–2583 (2002)Google Scholar
  35. Daw, N.D., Kakade, S., Dayan, P.: Opponent interactions between serotonin and dopamine. Neural Networks 15, 603–616 (2002)Google Scholar
  36. Daw, N.D., Niv, Y., Dayan, P.: Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control. Nature Neuroscience 8, 1704–1711 (2005)Google Scholar
  37. Daw, N.D., Courville, A.C., Tourtezky, D.S.: Representation and timing in theories of the dopamine system. Neural Computation 18, 1637–1677 (2006a)MathSciNetGoogle Scholar
  38. Daw, N.D., O’Doherty, J.P., Dayan, P., Seymour, B., Dolan, R.J.: Cortical substrates for exploratory decisions in humans. Nature 441, 876–879 (2006b)Google Scholar
  39. Dayan, P., Daw, N.D.: Connections between computational and neurobiological perspectives on decision making. Cognitive, Affective, and Behavioral Neuroscience 8, 429–453 (2008)Google Scholar
  40. Dayan, P., Niv, Y.: Reinforcement learning: the good, the bad, and the ugly. Current Opinion in Neurobiology 18, 185–196 (2008)Google Scholar
  41. Dayan, P., Niv, Y., Seymour, B., Daw, N.D.: The misbehavior of value and the discipline of the will. Neural Networks 19, 1153–1160 (2006)Google Scholar
  42. Dickinson, A.: Actions and habits: the development of behavioural autonomy. Philosophical Transactions of the Royal Society of London B: Biological Sciences 308, 67–78 (1985)Google Scholar
  43. Dickinson, A., Balleine, B.W.: Motivational control of goal-directed action. Animal Learning and Behavior 22, 1–18 (1994)Google Scholar
  44. Doll, B.B., Frank, M.J.: The basal ganglia in reward and decision making: computational models and empirical studies. In: Dreher, J., Tremblay, L. (eds.) Handbook of Reward and Decision Making, ch. 19, pp. 399–425. Academic Press, Oxford (2009)Google Scholar
  45. Dommett, E., Coizet, V., Blaha, C.D., Martindale, J., Lefebvre, V., Mayhew, N.W.J.E., Overton, P.G., Redgrave, P.: How visual stimuli activate dopaminergic neurons at short latency. Science 307, 1476–1479 (2005)Google Scholar
  46. Doya, K.: What are the computations of the cerebellum, the basal ganglia, and the cerebral cortex? Neural Networks 12, 961–974 (1999)Google Scholar
  47. Doya, K.: Reinforcement learning: Computational theory and biological mechanisms. HFSP Journal 1, 30–40 (2007)Google Scholar
  48. Doya, K.: Modulators of decision making. Nature Neuroscience 11, 410–416 (2008)Google Scholar
  49. Doyon, J., Bellec, P., Amsel, R., Penhune, V., Monchi, O., Carrier, J., Lehéricy, S., Benali, H.: Contributions of the basal ganglia and functionally related brain structures to motor learning. Behavioural Brain Research 199, 61–75 (2009)Google Scholar
  50. Eckerman, D.A., Hienz, R.D., Stern, S., Kowlowitz, V.: Shaping the location of a pigeon’s peck: Effect of rate and size of shaping steps. Journal of the Experimental Analysis of Behavior 33, 299–310 (1980)Google Scholar
  51. Ferster, C.B., Skinner, B.F.: Schedules of Reinforcement. Appleton-Century-Crofts, New York (1957)Google Scholar
  52. Fiorillo, C.D., Tobler, P.N., Schultz, W.: Discrete coding of reward probability and uncertainty by dopamine neurons. Science 299, 1898–1902 (2003)Google Scholar
  53. Frank, M.J.: Dynamic dopamine modulation in the basal ganglia: a neurocomputational account of cognitive deficits in medicated and nonmedicated Parkinsonism. Journal of Cognitive Neuroscience 17, 51–72 (2005)Google Scholar
  54. Frank, M.J., Claus, E.D.: Anatomy of a decision: Striato-orbitofrontal interactions in reinforcement learning, decision making, and reversal. Psychological Review 113, 300–326 (2006)Google Scholar
  55. Frank, M.J., Seeberger, L.C., O’Reilly, R.C.: By carrot or by stick: Cognitive reinforcement learning in parkinsonism. Science 306, 1940–1943 (2004)Google Scholar
  56. Gardner, R.: Multiple-choice decision behavior. American Journal of Psychology 71, 710–717 (1958)Google Scholar
  57. Gläscher, J.P., O’Doherty, J.P.: Model-based approaches to neuroimaging combining reinforcement learning theory with fMRI data. Wiley Interdisciplinary Reviews: Cognitive Science 1, 501–510 (2010)Google Scholar
  58. Gläscher, J.P., Daw, N.D., Dayan, P., O’Doherty, J.P.: States versus rewards: Dissociable neural prediction error signals underlying model-based and model-free reinforcement learning. Neuron 66, 585–595 (2010)Google Scholar
  59. Glimcher, P.W.: Decisions, Uncertainty, and the Brain: The Science of Neuroeconomics. MIT Press, Cambridge (2003)Google Scholar
  60. Glimcher, P.W., Rustichini, A.: Neuroeconomics: The consilience of brain and decision. Science 306, 447–452 (2004)Google Scholar
  61. Gluck, M.A.: Behavioral and neural correlates of error correction in classical conditioning and human category learning. In: Gluck, M.A., Anderson, J.R., Kosslyn, S.M. (eds.) Memory and Mind: A Festschrift for Gordon H. Bower, ch. 18, pp. 281–305. Lawrence Earlbaum Associates, New York (2008)Google Scholar
  62. Gold, J.I., Shadlen, M.N.: The neural basis of decision making. Annual Review of Neuroscience 30, 535–574 (2007)Google Scholar
  63. Goldman-Rakic, P.S.: Cellular basis of working memory. Neuron 14, 447–485 (1995)Google Scholar
  64. Goodnow, J.T.: Determinants of choice-distribution in two-choice situations. The American Journal of Psychology 68, 106–116 (1955)Google Scholar
  65. Gormezano, I., Schneiderman, N., Deaux, E.G., Fuentes, I.: Nictitating membrane: Classical conditioning and extinction in the albino rabbit. Science 138, 33–34 (1962)Google Scholar
  66. Grafton, S.T., Hamilton, A.F.: Evidence for a distributed hierarchy of action representation in the brain. Human Movement Science 26, 590–616 (2007)Google Scholar
  67. Graybiel, A.M.: The basal ganglia: learning new tricks and loving it. Current Opinion in Neurobiology 15, 638–644 (2005)Google Scholar
  68. Graybiel, A.M.: Habits, rituals, and the evaluative brain. Annual Review of Neuroscience 31, 359–387 (2008)Google Scholar
  69. Graybiel, A.M., Aosaki, T., Flahrety, A.W., Kimura, M.: The basal ganglia and adaptive motor control. Science 265, 1826–1831 (1994)Google Scholar
  70. Green, L., Myerson, J.: A discounting framework for choice with delayed and probabilistic rewards. Psychological Bulletin 130, 769–792 (2004)Google Scholar
  71. Grupen, R., Huber, M.: A framework for the development of robot behavior. In: 2005 AAAI Spring Symposium Series: Developmental Robotics. American Association for the Advancement of Artificial Intelligence, Palo Alta (2005)Google Scholar
  72. Gurney, K.: Reverse engineering the vertebrate brain: Methodological principles for a biologically grounded programme of cognitive modelling. Cognitive Computation 1, 29–41 (2009)Google Scholar
  73. Gurney, K., Prescott, T.J., Redgrave, P.: A computational model of action selection in the basal ganglia. I. A new functional anatomy. Biological Cybernetics 84, 401–410 (2001)Google Scholar
  74. Gurney, K., Prescott, T.J., Wickens, J.R., Redgrave, P.: Computational models of the basal ganglia: From robots to membranes. Trends in Neuroscience 27, 453–459 (2004)Google Scholar
  75. Haber, S.N.: The primate basal ganglia: Parallel and integrative networks. Journal of Chemical Neuroanatomy 26, 317–330 (2003)Google Scholar
  76. Haber, S.N., Kim, K.S., Mailly, P., Calzavara, R.: Reward-related cortical inputs define a large striatal region in primates that interface with associative cortical inputs, providing a substrate for incentive-based learning. The Journal of Neuroscience 26, 8368–8376 (2006)Google Scholar
  77. Haruno, M., Kawato, M.: Heterarchical reinforcement-learning model for integration of multiple cortico-striatal loops: fMRI examination in stimulus-action-reward association learning. Neural Networks 19, 1242–1254 (2006)Google Scholar
  78. Hazy, T.E., Frank, M.J., O’Reilly, R.C.: Neural mechanisms of acquired phasic dopamine repsonses in learning. Neuroscience and Biobehavioral Reviews 34, 701–720 (2010)Google Scholar
  79. Herrnstein, R.J.: Relative and absolute strength of response as a function of frequency of reinforcement. Journal of the Experimental Analysis of Behavior 4, 267–272 (1961)Google Scholar
  80. Hikosaka, O.: Basal ganglia mechanisms of reward-oriented eye movement. Annals of the New York Academy of Science 1104, 229–249 (2007)Google Scholar
  81. Hollerman, J.R., Schultz, W.: Dopamine neurons report an error in the temporal prediction of reward during learning. Nature Neuroscience 1, 304–309 (1998)Google Scholar
  82. Horvitz, J.C.: Mesolimbocortical and nigrostriatal dopamine responses to salient non-reward events. Neuroscience 96, 651–656 (2000)Google Scholar
  83. Houk, J.C., Wise, S.P.: Distributed modular architectures linking basal ganglia, cerebellum, and cerebral cortex: Their role in planning and controlling action. Cerebral Cortex 5, 95–110 (1995)Google Scholar
  84. Houk, J.C., Adams, J.L., Barto, A.G.: A model of how the basal ganglia generate and use neural signals that predict reinforcement. In: Houk, J.C., Davis, J.L., Beiser, D.G. (eds.) Models of Information Processing in the Basal Ganglia, ch. 13, pp. 249–270. MIT Press, Cambridge (1995)Google Scholar
  85. Houk, J.C., Bastianen, C., Fansler, D., Fishbach, A., Fraser, D., Reber, P.J., Roy, S.A., Simo, L.S.: Action selection and refinement in subcortical loops through basal ganglia and cerebellum. Philosophical Transactions of the Royal Society of London B: Biological Sciences 362, 1573–1583 (2007)Google Scholar
  86. Hull, C.L.: Principles of Behavior. Appleton-Century-Crofts, New York (1943)Google Scholar
  87. Humphries, M.D., Prescott, T.J.: The ventral basal ganglia, a selection mechanism at the crossroads of space, strategy, and reward. Progress in Neurobiology 90, 385–417 (2010)Google Scholar
  88. Ito, M., Doya, K.: Validation of decision-making models and analysis of decision variables in the rat basal ganglia. The Journal of Neuroscience 29, 9861–9874 (2009)Google Scholar
  89. Joel, D., Weiner, I.: The organization of the basal ganglia-thalamocortical circuits: Open interconnected rather than closed segregated. Neuroscience 63, 363–379 (1994)Google Scholar
  90. Joel, D., Niv, Y., Ruppin, E.: Actor-critic models of the basal ganglia: New anatomical and computational perspectives. Neural Networks 15, 535–547 (2002)Google Scholar
  91. Joshua, M., Adler, A., Bergman, H.: The dynamics of dopamine in control of motor behavior. Current Opinion in Neurobiology 19, 615–620 (2009)Google Scholar
  92. Kamin, L.J.: Predictability, surprise, attention, and conditioning. In: Campbell, B.A., Church, R.M. (eds.) Punishment and Aversive Behavior, pp. 279–296. Appleton-Century-Crofts, New York (1969)Google Scholar
  93. Kehoe, E.J., Schreurs, B.G., Graham, P.: Temporal primacy overrides prior training in serial compound conditioning of the rabbit’s nictitating membrane response. Animal Learning and Behavior 15, 455–464 (1987)Google Scholar
  94. Kim, H., Sul, J.H., Huh, N., Lee, D., Jung, M.W.: Role of striatum in updating values of chosen actions. The Journal of Neuroscience 29, 14,701–14,712 (2009)Google Scholar
  95. Kishida, K.T., King-Casas, B., Montague, P.R.: Neuroeconomic approaches to mental disorders. Neuron 67, 543–554 (2010)Google Scholar
  96. Klopf, A.H.: The Hedonistic Neuron: A Theory of Memory, Learning and Intelligence. Hemisphere Publishing Corporation, Washington DC (1982)Google Scholar
  97. Kobayashi, S., Schultz, W.: Influence of reward delays on responses of dopamine neurons. The Journal of Neuroscience 28, 7837–7846 (2008)Google Scholar
  98. Konidaris, G.D., Barto, A.G.: Skill discovery in continuous reinforcement learning domains using skill chaining. In: Bengio, Y., Schuurmans, D., Lafferty, J., Williams, C.K.I., Culotta, A. (eds.) Advances in Neural Information Processing Systems (NIPS), vol. 22, pp. 1015–1023. MIT Press, Cambridge (2009)Google Scholar
  99. Lau, B., Glimcher, P.W.: Value representations in the primate striatum during matching behavior. Neuron 58, 451–463 (2008)Google Scholar
  100. Ljungberg, T., Apicella, P., Schultz, W.: Responses of monkey dopamine neurons during learning of behavioral reactions. Journal of Neurophysiology 67, 145–163 (1992)Google Scholar
  101. Ludvig, E.A., Sutton, R.S., Kehoe, E.J.: Stimulus representation and the timing of reward-prediction errors in models of the dopamine system. Neural Computation 20, 3034–3054 (2008)Google Scholar
  102. Maia, T.V.: Reinforcement learning, conditioning, and the brain: Successes and challenges. Cognitive, Affective, and Behavioral Neuroscience 9, 343–364 (2009)Google Scholar
  103. Maia, T.V., Frank, M.J.: From reinforcement learning models to psychiatric and neurobiological disorders. Nature Neuroscience 14, 154–162 (2011)Google Scholar
  104. Matsumoto, K., Suzuki, W., Tanaka, K.: Neuronal correlates of goal-based motor selection in the prefrontal cortex. Science 301, 229–232 (2003)Google Scholar
  105. Matsuzaka, Y., Picard, N., Strick, P.: Skill representation in the primary motor cortex after long-term practice. Journal of Neurophysiology 97, 1819–1832 (2007)Google Scholar
  106. McHaffie, J.G., Stanford, T.R., Stein, B.E., Coizet, V., Redgrave, P.: Subcortical loops through the basal ganglia. Trends in Neurosciences 28, 401–407 (2005)Google Scholar
  107. Middleton, F.A., Strick, P.L.: Basal-ganglia“projections” to the prefrontal cortex of the primate. Cerebral Cortex 12, 926–935 (2002)Google Scholar
  108. Miller, E.K., Cohen, J.D.: An integrative theory of prefrontal cortex function. Annual Review of Neuroscience 24, 167–202 (2001)Google Scholar
  109. Miller, J.D., Sanghera, M.K., German, D.C.: Mesencephalic dopaminergic unit activity in the behaviorally conditioned rat. Life Sciences 29, 1255–1263 (1981)Google Scholar
  110. Mink, J.W.: The basal ganglia: Focused selection and inhibition of competing motor programs. Progress in Neurobiology 50, 381–425 (1996)Google Scholar
  111. Mirolli, M., Mannella, F., Baldassarre, G.: The roles of the amygdala in the affective regulation of body, brain, and behaviour. Connection Science 22, 215–245 (2010)Google Scholar
  112. Montague, P.R., Dayan, P., Sejnowski, T.J.: A framework for mesencephalic dopamine systems based on predictive Hebbian learning. Journal of Neuroscience 16, 1936–1947 (1996)Google Scholar
  113. Montague, P.R., Hyman, S.E., Cohen, J.D.: Computational roles for dopamine in behavioural control. Nature 431, 760–767 (2004)Google Scholar
  114. Montague, P.R., King-Casas, B., Cohen, J.D.: Imaging valuation models in human choice. Annual Review of Neuroscience 29, 417–448 (2006)Google Scholar
  115. Moore, J.W., Choi, J.S.: Conditioned response timing and integration in the cerebellum. Learning and Memory 4, 116–129 (1997)Google Scholar
  116. Morris, G., Nevet, A., Arkadir, D., Vaadia, E., Bergman, H.: Midbrain dopamine neurons encode decisions for future action. Nature Neuroscience 9, 1057–1063 (2006)Google Scholar
  117. Mushiake, H., Saito, N., Sakamoto, K., Itoyama, Y., Tanji, J.: Activity in the lateral prefrontal cortex reflects multiple steps of future events in action plans. Neuron 50, 631–641 (2006)Google Scholar
  118. Nakahara, H., Itoh, H., Kawagoe, R., Takikawa, Y., Hikosaka, O.: Dopamine neurons can represent context-dependent prediction error. Neuron 41, 269–280 (2004)Google Scholar
  119. Ng, A., Harada, D., Russell, S.: Policy invariance under reward transformations: theory and applications to reward shaping. In: Proceedings of the Sixteenth International Conference on Machine Learning, pp. 278–287 (1999)Google Scholar
  120. Nicola, S.M.: The nucleus accumbens as part of a basal ganglia action selection circuit. Psychopharmacology 191, 521–550 (2007)Google Scholar
  121. Niv, Y.: Reinforcement learning in the brain. Journal of Mathematical Psychology 53, 139–154 (2009)MathSciNetGoogle Scholar
  122. Niv, Y., Duff, M.O., Dayan, P.: Dopamine, uncertainty, and TD learning. Behavioral and Brain Functions 1, 6 (2005)Google Scholar
  123. Niv, Y., Daw, N.D., Dayan, P.: Choice values. Nature Neuroscience 9, 987–988 (2006a)Google Scholar
  124. Niv, Y., Joel, D., Dayan, P.: A normative perspective on motivation. Trends in Cognitive Sciences 10, 375–381 (2006b)Google Scholar
  125. Nomoto, K., Schultz, W., Watanabe, T., Sakagami, M.: Temporally extended dopamine responses to perceptually demanding reward-predictive stimuli. The Journal of Neuroscience 30, 10,692–10,702 (2010)Google Scholar
  126. O’Doherty, J.P., Dayan, P., Schultz, J., Deichmann, R., Friston, K., Dolan, R.J.: Dissociable roles of ventral and dorsal striatum in instrumental conditioning. Science 304, 452–454 (2004)Google Scholar
  127. Olds, J., Milner, P.: Positive reinforcement produced by electrical stimulation of septal area and other regions of rat brain. Journal of Comparative and Physiological Psychology 47, 419–427 (1954)Google Scholar
  128. O’Reilly, R.C., Frank, M.J.: Making working memory work: a computational model of learning in the prefrontal cortex and basal ganglia. Neural Computation 18, 283–328 (2006)MathSciNetGoogle Scholar
  129. Packard, M.G., Knowlton, B.J.: Learning and memory functions of the basal ganglia. Annual Review of Neuroscience 25, 563–593 (2002)Google Scholar
  130. Pasupathy, A., Miller, E.K.: Different time courses of learning-related activity in the prefrontal cortex and striatum. Nature 433, 873–876 (2005)Google Scholar
  131. Pavlov, I.P.: Conditioned Reflexes: An Investigation of the Physiological Activity of the Cerebral Cortex. Oxford University Press, Toronto (1927)Google Scholar
  132. Pennartz, C.M., Berke, J.D., Graybiel, A.M., Ito, R., Lansink, C.S., van der Meer, M., Redish, A.D., Smith, K.S., Voorn, P.: Corticostriatal interactions during learning, memory processing, and decision making. The Journal of Neuroscience 29, 12,831–12,838 (2009)Google Scholar
  133. Pessiglione, M., Seymour, B., Flandin, G., Dolan, R.J., Frith, C.D.: Dopamine-dependent prediction errors underpin reward-seeking behaviour in humans. Nature 442, 1042–1045 (2006)Google Scholar
  134. Phelps, E.A., LeDoux, J.E.: Contributions of the amygdala to emotion processing: From animal models to human behavior. Neuron 48, 175–187 (2005)Google Scholar
  135. Poldrack, R.A., Sabb, F.W., Foerde, K., Tom, S.M., Asarnow, R.F., Bookheimer, S.Y., Knowlton, B.J.: The neural correlates of motor skill automaticity. The Journal of Neuroscience 25, 5356–5364 (2005)Google Scholar
  136. Pompilio, L., Kacelnik, A.: State-dependent learning and suboptimal choice: when starlings prefer long over short delays to food. Animal Behaviour 70, 571–578 (2005)Google Scholar
  137. Redgrave, P., Gurney, K.: The short-latency dopamine signal: a role in discovering novel actions? Nature Reviews Neuroscience 7, 967–975 (2006)Google Scholar
  138. Redgrave, P., Gurney, K., Reynolds, J.: What is reinforced by phasic dopamine signals? Brain Research Reviews 58, 322–339 (2008)Google Scholar
  139. Redgrave, P., Rodriguez, M., Smith, Y., Rodriguez-Oroz, M.C., Lehericy, S., Bergman, H., Agid, Y., DeLong, M.R., Obeso, J.A.: Goal-directed and habitual control in the basal ganglia: implications for Parkinson’s disease. Nature Reviews Neuroscience 11, 760–772 (2010)Google Scholar
  140. Redish, A.D., Jensen, S., Johnson, A.: A unified framework for addiction: Vulnerabilities in the decision process. Behavioral and Brain Sciences 31, 415–487 (2008)Google Scholar
  141. Rescorla, R.A., Wagner, A.R.: A theory of pavlovian conditioning: Variations in the effectiveness of reinforcement and nonreinforcement. In: Black, A.H., Prokasy, W.F. (eds.) Classical Conditioning II: Current Research and Theory, pp. 64–99. Appleton-Century-Crofts, New York (1972)Google Scholar
  142. Richardson, W.K., Warzak, W.J.: Stimulus stringing by pigeons. Journal of the Experimental Analysis of Behavior 36, 267–276 (1981)Google Scholar
  143. Roesch, M.R., Calu, D.J., Schoenbaum, G.: Dopamine neurons encode the better option in rats deciding between differently delayed or sized rewards. Nature Neuroscience 10, 1615–1624 (2007)Google Scholar
  144. Roesch, M.R., Singh, T., Brown, P.L., Mullins, S.E., Schoenbaum, G.: Ventral striatal neurons encode the value of the chosen action in rats deciding between differently delayed or sized rewards. The Journal of Neuroscience 29, 13,365–13,376 (2009)Google Scholar
  145. Samejima, K., Doya, K.: Multiple representations of belief states and action values in corticobasal ganglia loops. Annals of the New York Academy of Sciences 1104, 213–228 (2007)Google Scholar
  146. Samejima, K., Ueda, Y., Doya, K., Kimura, M.: Representation of action-specific reward values in the striatum. Science 310, 1337–1340 (2005)Google Scholar
  147. Satoh, T., Nakai, S., Sato, T., Kimura, M.: Correlated coding of motivation and outcome of decision by dopamine neurons. The Journal of Neuroscience 23, 9913–9923 (2003)Google Scholar
  148. Schultz, W.: Responses of midbrain dopamine neurons to behavioral trigger stimuli in the monkey. Journal of Neurophysiology 56, 1439–1461 (1986)Google Scholar
  149. Schultz, W.: Predictive reward signal of dopamine neurons. Journal of Neurophysiology 80, 1–27 (1998)Google Scholar
  150. Schultz, W.: Behavioral theories and the neurophysiology of reward. Annual Review of Psychology 57, 8–115 (2006)Google Scholar
  151. Schultz, W.: Multiple dopamine functions at different time courses. Annual Review of Neuroscience 30, 259–288 (2007)Google Scholar
  152. Schultz, W.: Dopamine signals for reward value and risk: basic and recent data. Behavioral and Brain Functions 6, 24 (2010)Google Scholar
  153. Schultz, W., Apicella, P., Ljungberg, T.: Responses of monkey dopamine neurons to reward and conditioned stimuli during successive steps of learning a delayed response task. The Journal of Neuroscience 13, 900–913 (1993)Google Scholar
  154. Schultz, W., Dayan, P., Montague, P.R.: A neural substrate of prediction and reward. Science 275, 1593–1599 (1997)Google Scholar
  155. Schultz, W., Tremblay, L., Hollerman, J.R.: Changes in behavior-related neuronal activity in the striatum during learning. Trends in Neuroscience 26, 321–328 (2003)Google Scholar
  156. Seger, C.A., Miller, E.K.: Category learning in the brain. Annual Review of Neuroscience 33, 203–219 (2010)Google Scholar
  157. Selfridge, O.J., Sutton, R.S., Barto, A.G.: Training and tracking in robotics. In: Joshi, A. (ed.) Proceedings of the Ninth International Joint Conference on Artificial Intelligence, pp. 670–672. Morgan Kaufmann, San Mateo (1985)Google Scholar
  158. Shah, A.: Biologically-based functional mechanisms of motor skill acquisition. PhD thesis, University of Massachusetts Amherst (2008)Google Scholar
  159. Shah, A., Barto, A.G.: Effect on movement selection of an evolving sensory representation: A multiple controller model of skill acquisition. Brain Research 1299, 55–73 (2009)Google Scholar
  160. Shanks, D.R., Tunney, R.J., McCarthy, J.D.: A re-examination of probability matching and rational choice. Journal of Behavioral Decision Making 15, 233–250 (2002)Google Scholar
  161. Siegel, S., Goldstein, D.A.: Decision making behaviour in a two-choice uncertain outcome situation. Journal of Experimental Psychology 57, 37–42 (1959)Google Scholar
  162. Skinner, B.F.: The Behavior of Organisms. Appleton-Century-Crofts, New York (1938)Google Scholar
  163. Staddon, J.E.R., Cerutti, D.T.: Operant behavior. Annual Review of Psychology 54, 115–144 (2003)Google Scholar
  164. Sutton, R.S.: Learning to predict by methods of temporal differences. Machine Learning 3, 9–44 (1988)Google Scholar
  165. Sutton, R.S., Barto, A.G.: Toward a modern theory of adaptive networks: Expectation and prediction. Psychological Review 88, 135–170 (1981)Google Scholar
  166. Sutton, R.S., Barto, A.G.: A temporal-difference model of classical conditioning. In: Proceedings of the Ninth Annual Conference of the Cognitive Science Society, pp. 355–378 (1987)Google Scholar
  167. Sutton, R.S., Barto, A.G.: Reinforcement Learning: An Introduction. MIT Press, Cambridge (1998)Google Scholar
  168. Tanji, J., Hoshi, E.: Role of the lateral prefrontal cortex in executive behavioral control. Physiological Reviews 88, 37–57 (2008)Google Scholar
  169. Thorndike, E.L.: Animal Intelligence: Experimental Studies. Macmillan, New York (1911)Google Scholar
  170. Tindell, A.J., Berridge, K.C., Zhang, J., Pecina, S., Aldridge, J.W.: Ventral pallidal neurons code incentive motivation: Amplification by mesolimbic sensitization and amphetamine. European Journal of Neuroscience 22, 2617–2634 (2005)Google Scholar
  171. Tobler, P.N., Dickinson, A., Schultz, W.: Coding of predicted reward omission by dopamine neurons in a conditioned inhibition paradigm. The Journal of Neuroscience 23, 10,402–10,410 (2003)Google Scholar
  172. Tobler, P.N., Fiorillo, C.D., Schultz, W.: Adaptive coding of reward value by dopamine neurons. Science 307, 1642–1645 (2005)Google Scholar
  173. Tolman, E.C.: Cognitive maps in rats and men. The Psychological Review 55, 189–208 (1948)Google Scholar
  174. Tolman, E.C.: There is more than one kind of learning. Psychological Review 56, 44–55 (1949)Google Scholar
  175. Waelti, P., Dickinson, A., Schultz, W.: Dopamine responses comply with basic assumptions of formal learning theory. Nature 412, 43–48 (2001)Google Scholar
  176. Wallis, J.D.: Orbitofrontal cortex and its contribution to decision-making. Annual Review of Neuroscience 30, 31–56 (2007)Google Scholar
  177. Watson, J.B.: Behavior: An Introduction to Comparative Psychology. Holt, New York (1914)Google Scholar
  178. Wickens, J.R.: Synaptic plasticity in the basal ganglia. Behavioural Brain Research 199, 119–128 (2009)Google Scholar
  179. Wickens, J.R., Budd, C.S., Hyland, B.I., Arbuthnott, G.W.: Striatal contributions to reward and decision making. Making sense of regional variations in a reiterated processing matrix. Annals of the New York Academy of Sciences 1104, 192–212 (2007)Google Scholar
  180. Widrow, B., Hoff, M.E.: Adaptive switching circuits. In: 1960 WESCON Convention Record Part IV, pp. 96–104. Institute of Radio Engineers, New York (1960)Google Scholar
  181. Wilson, C.J.: Basal ganglia. In: Shepherd, G.M. (ed.) The Synaptic Organization of the Brain, ch. 9, 5th edn., pp. 361–414. Oxford University Press, Oxford (2004)Google Scholar
  182. Wise, R.A.: Dopamine, learning and motivation. Nature Reviews Neuroscience 5, 483–494 (2004)Google Scholar
  183. Wolpert, D.: Probabilistic models in human sensorimotor control. Human Movement Science 27, 511–524 (2007)Google Scholar
  184. Wörgötter, F., Porr, B.: Temporal sequence learning, prediction, and control: A review of different models and their relation to biological mechanisms. Neural Computation 17, 245–319 (2005)Google Scholar
  185. Wrase, J., Kahnt, T., Schlagenhauf, F., Beck, A., Cohen, M.X., Knutson, B., Heinz, A.: Different neural systems adjust motor behavior in response to reward and punishment. NeuroImage 36, 1253–1262 (2007)Google Scholar
  186. Wyvell, C.L., Berridge, K.C.: Intra-accumbens amphetamine increases the conditioned incentive salience of sucrose reward: Enhancement of reward “wanting” without enhanced “liking” or response reinforcement. Journal of Neuroscience 20, 8122–8130 (2000)Google Scholar
  187. Yin, H.H., Ostlund, S.B., Balleine, B.W.: Reward-guided learning beyond dopamine in the nucleus accumbens: the integrative functions of cortico-basal ganglia networks. European Journal of Neuroscience 28, 1437–1448 (2008)Google Scholar
  188. Yu, A., Dayan, P.: Uncertainty, neuromodulation and attention. Neuron 46, 681–692 (2005)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2012

Authors and Affiliations

  1. 1.Department of PsychologyUniversity of SheffieldSheffieldUK

Personalised recommendations