Skip to main content
SpringerLink
Log in

Computational models of reinforcement learning: the role of dopamine as a reward signal

  • Review
  • Published:
Cognitive Neurodynamics Aims and scope Submit manuscript

Abstract

Reinforcement learning is ubiquitous. Unlike other forms of learning, it involves the processing of fast yet content-poor feedback information to correct assumptions about the nature of a task or of a set of stimuli. This feedback information is often delivered as generic rewards or punishments, and has little to do with the stimulus features to be learned. How can such low-content feedback lead to such an efficient learning paradigm? Through a review of existing neuro-computational models of reinforcement learning, we suggest that the efficiency of this type of learning resides in the dynamic and synergistic cooperation of brain systems that use different levels of computations. The implementation of reward signals at the synaptic, cellular, network and system levels give the organism the necessary robustness, adaptability and processing speed required for evolutionary and behavioral success.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

References

  • Abercrombie ED, Keefe KA et al (1989) Differential effect of stress on in vivo dopamine release in striatum, nucleus accumbens, and medial frontal cortex. J Neurochem 52:1655–1658

    Article  CAS  PubMed  Google Scholar 

  • Arbuthnott GW, Wickens J (2007) Space, time and dopamine. Trends Neurosci 30:62–69

    Article  CAS  PubMed  Google Scholar 

  • Baras D, Meir R (2007) Reinforcement learning, spike-time-dependent plasticity, and the bcm rule. Neural Comput 19:2245–2279

    Article  PubMed  Google Scholar 

  • Barto AG (1995) Adaptive critics and the basal ganglia. Models of information processing in the basal ganglia. 215–232

  • Barto AG, Sutton RS, Anderson C (1983) Neuron-like adaptive elements that can solve difficult learning control problems, IEEE transactions on systems, man, and cybernetics. SMC 13:834–846

    Google Scholar 

  • Bauer EP, LeDoux JE (2004) Heterosynaptic long-term potentiation of inhibitory interneurons in the lateral amygdala. J Neurosci 24:9507–9512

    Article  CAS  PubMed  Google Scholar 

  • Bauer EP, Schafe GE, LeDoux JE (2002) NMDA receptors and L-type voltage-gated calcium channels contribute to long-term potentiation and different components of fear memory formation in the lateral amygdala. J Neurosci 22:5239–5249

    CAS  PubMed  Google Scholar 

  • Bayer HM, Glimcher PW (2005) Midbrain dopamine neurons encode a quantitative reward prediction error signal. Neuron 47:129–141

    Article  CAS  PubMed  Google Scholar 

  • Bayer HM, Lau B, Glimcher PW (2007) Statistics of midbrain dopamine neuron spike trains in the awake primate. J Neurophysiol 98:1428–1439

    Article  PubMed  Google Scholar 

  • Bergstrom BP, Garris PA (2003) Passive stabilization of striatal extracellular dopamine across the lesion spectrum encompassing the presymptomatic phase of Parkinson’s disease: a voltammetric study in the 6-OHDA-lesioned rat. J Neurochem 87:1224–1236

    Article  CAS  PubMed  Google Scholar 

  • Berke JD, Hyman SE (2000) Addiction, dopamine, and the molecular mechanisms of memory. Neuron 25:515–532

    Article  CAS  PubMed  Google Scholar 

  • Berridge KC (2007) The debate over dopamine’s role in reward: the case for incentive salience. Psychopharmacology (Berl) 191:391–431

    Google Scholar 

  • Berridge KC, Robinson TE (1998) What is the role of dopamine in reward: hedonic impact, reward learning, or incentive salience? Brain Res Brain Res Rev 28:309–369

    Article  CAS  PubMed  Google Scholar 

  • Bertin M, Schweighofer N, Doya K (2007) Multiple model-based reinforcement learning explains dopamine neuronal activity. Neural Netw 20:668–675

    Article  PubMed  Google Scholar 

  • Bi G, Poo M (1998) Synaptic modifications in cultured hippocampal neurons: dependence on spike timing, synaptic strength, and postsynaptic cell type. J Neurosci 18:10464–10472

    CAS  PubMed  Google Scholar 

  • Bissière S, Humeau Y, Luthi A (2003) Dopamine gates ltp induction in lateral amygdala by suppressing feedforward inhibition. Nature Neurosci 6:587–592

    Article  PubMed  CAS  Google Scholar 

  • Bouton ME (2004) Context and behavioral processes in extinction. Learn Mem 11:485–494

    Article  PubMed  Google Scholar 

  • Calabresi P, Picconi B, Tozzi A, Di Filippo M (2007) Dopamine-mediated regulation of corticostriatal synaptic plasticity. Trends Neurosci 30:211–219

    Article  CAS  PubMed  Google Scholar 

  • Camerer CF (2008) Neuroeconomics: opening the gray box. Neuron 60:416–419

    Article  CAS  PubMed  Google Scholar 

  • Canavan AG, Passingham RE et al (1989) The performance on learning tasks of patients in the early stages of Parkinson’s disease. Neuropsychologia 27:141–156

    Article  CAS  PubMed  Google Scholar 

  • Cardinal RN, Parkinson JA et al (2002) Emotion and motivation: the role of the amygdala, ventral striatum, and prefrontal cortex. Neurosci Biobehav Rev 26:321–352

    Article  PubMed  Google Scholar 

  • Centonze D, Grande C et al (2003) Distinct roles of D1 and D5 dopamine receptors in motor activity and striatal synaptic plasticity. J Neurosci 23:8506–8512

    CAS  PubMed  Google Scholar 

  • Cepeda C, Levine MS (1998) Dopamine and N-methyl-d-aspartate receptor interactions in the neostriatum. Dev Neurosci 20:1–18

    Article  CAS  PubMed  Google Scholar 

  • Chuhma N, Choi WY et al (2009) Dopamine neuron glutamate cotransmission: frequency-dependent modulation in the mesoventromedial projection. Neuroscience 164:1068–1083

    Article  CAS  PubMed  Google Scholar 

  • Cohen MX, Frank MJ (2009) Neurocomputational models of basal ganglia function in learning, memory and choice. Behav Brain Res 199:141–156

    Article  PubMed  Google Scholar 

  • Dalley JW, Laane K et al (2005) Time-limited modulation of appetitive Pavlovian memory by D1 and NMDA receptors in the nucleus accumbens. Proc Natl Acad Sci USA 102:6189–6194

    Article  CAS  PubMed  Google Scholar 

  • Daniel JA, Galbraith S et al (2009) Functional heterogeneity at dopamine release sites. J Neurosci 29:14670–14680

    Article  CAS  PubMed  Google Scholar 

  • Daw ND, Niv Y et al (2005) Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control. Nat Neurosci 8:1704–1711

    Article  CAS  PubMed  Google Scholar 

  • Daw ND, Courville AC, Touretzky DS (2006) Representation and timing in theories of the dopamine system. Neural Comput 18:1637–1677

    Article  PubMed  Google Scholar 

  • Dayan P, Abbott LF (2001) Theoretical neuroscience. Computational and mathematical modeling of neural systems. The MIT Press, Cambridge

  • Dayan P, Balleine BW (2002) Reward, motivation, and reinforcement learning. Neuron 36:285–298

    Article  CAS  PubMed  Google Scholar 

  • Dayan P, Daw ND (2008) Decision theory, reinforcement learning, and the brain. Cogn Affect Behav Neurosci 8:429–453

    Article  PubMed  Google Scholar 

  • Di Chiara G (2002) Nucleus accumbens shell and core dopamine: differential role in behavior and addiction. Behav Brain Res 137:75–114

    Article  CAS  PubMed  Google Scholar 

  • Doya K, Samejima K et al (2002) Multiple model-based reinforcement learning. Neural Comput 14:1347–1369

    Article  PubMed  Google Scholar 

  • Ehrlich I, Humeau Y et al (2009) Amygdala inhibitory circuits and the control of fear memory. Neuron 62:757–771

    Article  CAS  PubMed  Google Scholar 

  • Everitt BJ, Parkinson JA et al (1999) Associative processes in addiction and reward. The role of amygdala-ventral striatal subsystems. Ann N Y Acad Sci 877:412–438

    Google Scholar 

  • Everitt BJ, Robbins TW (2005) Neural systems of reinforcement for drug addiction: from actions to habits to compulsion. Nat Neurosci 8:1481–1489

    Article  CAS  PubMed  Google Scholar 

  • Fellous J-M, Suri RE (2003) The roles of dopamine. The handbook of brain theory and neural networks. MIT Press, Cambridge, pp 361–365

    Google Scholar 

  • Fenu S, Di Chiara G (2003) Facilitation of conditioned taste aversion learning by systemic amphetamine: role of nucleus accumbens shell dopamine D1 receptors. Eur J Neurosci 18:2025–2030

    Article  PubMed  Google Scholar 

  • Fibiger HC, Phillips AG (1986) Reward, motivation, cognition, psychobiology of mesotelencephalic dopamine systems. Handbook of physiology—The nervous system IV. F E Bloom Baltimore, Williams and Wilkins, 647–675

  • Fields HL, Hjelmstad GO et al (2007) Ventral tegmental area neurons in learned appetitive behavior and positive reinforcement. Annu Rev Neurosci 30:289–316

    Article  CAS  PubMed  Google Scholar 

  • Floresco SB, West AR et al (2003) Afferent modulation of dopamine neuron firing differentially regulates tonic and phasic dopamine transmission. Nature Neurosci 6:968–973

    Article  CAS  PubMed  Google Scholar 

  • Florian RV (2007) Reinforcement learning through modulation of spike-timing-dependent synaptic plasticity. Neural Comput 19:1468–1502

    Article  PubMed  Google Scholar 

  • Fourcaudot E, Gambino F et al (2009) L-type voltage-dependent Ca(2+) channels mediate expression of presynaptic LTP in amygdala. Nat Neurosci 12:1093–1095

    Article  CAS  PubMed  Google Scholar 

  • Frank MJ (2005) Dynamic dopamine modulation in the basal ganglia: a neurocomputational account of cognitive deficits in medicated and nonmedicated Parkinsonism. J Cogn Neurosci 17:51–72

    Article  PubMed  Google Scholar 

  • Frank MJ, Seeberger LC, O’reilly RC (2004) By carrot or by stick: cognitive reinforcement learning in Parkinsonism. Science 306:1940–1943

    Article  CAS  PubMed  Google Scholar 

  • Frank MJ, Moustafa AA et al (2007) Genetic triple dissociation reveals multiple roles for dopamine in reinforcement learning. Proc Natl Acad Sci USA 104:16311–16316

    Article  CAS  PubMed  Google Scholar 

  • Frank MJ, Doll BB et al (2009) Prefrontal and striatal dopaminergic genes predict individual differences in exploration and exploitation. Nature Neurosci 12:1062–1068

    Article  CAS  PubMed  Google Scholar 

  • Freeman WJ (2007) Definitions of state variables and state space for brain-computer interface: part 1. Multiple hierarchical levels of brain function. Cogn Neurodyn 1:3–14

    Article  PubMed  Google Scholar 

  • Frey U, Hartmann S et al (1989) Domperidone, an inhibitor of the D2-receptor, blocks a late phase of an electrically induced long-term potentiation in the CA1-region in rats. Biomed Biochim Acta 48:473–476

    CAS  PubMed  Google Scholar 

  • Frey U, Schroeder H et al (1990) Dopaminergic antagonists prevent long-term maintenance of posttetanic LTP in the CA1 region of rat hippocampal slices. Brain Res 522:69–75

    Article  CAS  PubMed  Google Scholar 

  • Frey U, Matthies H et al (1991) The effect of dopaminergic D1 receptor blockade during tetanization on the expression of long-term potentiation in the rat CA1 region in vitro. Neurosci Lett 129:111–114

    Article  CAS  PubMed  Google Scholar 

  • Furuyashiki T, Gallagher M (2007) Neural encoding in the orbitofrontal cortex related to goal-directed behavior. Ann N Y Acad Sci 1121:193–215

    Article  PubMed  Google Scholar 

  • Furuyashiki T, Holland PC et al (2008) Rat orbitofrontal cortex separately encodes response and outcome information during performance of goal-directed behavior. J Neurosci 28:5127–5138

    Article  CAS  PubMed  Google Scholar 

  • Gall CM, Hendry SH et al (1987) Evidence for coexistence of GABA and dopamine in neurons of the rat olfactory bulb. J Comp Neurol 266:307–318

    Article  CAS  PubMed  Google Scholar 

  • Gallagher M, McMahan RW et al (1999) Orbitofrontal cortex and representation of incentive value in associative learning. J Neurosci 19:6610–6614

    CAS  PubMed  Google Scholar 

  • Glimcher PW, Rustichini A (2004) Neuroeconomics: the consilience of brain and decision. Science 306:447–452

    Article  CAS  PubMed  Google Scholar 

  • Goto Y, Grace AA (2005) Dopaminergic modulation of limbic and cortical drive of nucleus accumbens in goal-directed behavior. Nat Neurosci 8:805–812

    Article  CAS  PubMed  Google Scholar 

  • Grace AA (1991) Phasic versus tonic dopamine release and the modulation of dopamine system responsivity: a hypothesis for the etiology of schizophrenia. Neuroscience 41:1–24

    Article  CAS  PubMed  Google Scholar 

  • Gurden H, Tassin JP et al (1999) Integrity of the mesocortical dopaminergic system is necessary for complete expression of in vivo hippocampal–prefrontal cortex long-term potentiation. Neuroscience 94:1019–1027

    Article  CAS  PubMed  Google Scholar 

  • Hatfield T, Han JS et al (1996) Neurotoxic lesions of basolateral, but not central, amygdala interfere with pavlovian second-order conditioning and reinforcer devaluation effects. J Neurosci 16:5256–5265

    CAS  PubMed  Google Scholar 

  • Hernandez PJ, Andrzejewski ME et al (2005) AMPA/kainate, NMDA, and dopamine D1 receptor function in the nucleus accumbens core: a context-limited role in the encoding and consolidation of instrumental memory. Learn Mem 12:285–295

    Article  PubMed  Google Scholar 

  • Hikosaka O, Bromberg-Martin E et al (2008) New insights on the subcortical representation of reward. Curr Opin Neurobiol 18:203–208

    Article  CAS  PubMed  Google Scholar 

  • Hiraoka K, Yoshida M, Mishima T (2009) Parallel reinforcement learning for weighted multi-criteria model with adaptive margin. Cogn Neurodyn 3:17–24

    Article  PubMed  Google Scholar 

  • Holland PC, Gallagher M (1999) Amygdala circuitry in attentional and representational processes. Trends Cogn Sci 3:65–73

    Article  PubMed  Google Scholar 

  • Huang Y-Y, Kandel ER (1998) Postsynaptic induction and PKA-dependent expression of LTP in the lateral amygdala. Neuron 21:169–178

    Article  CAS  PubMed  Google Scholar 

  • Huang Y-Y, Kandel ER (2007) Low-frequency stimulation induces a pathway-specific late phase of LTP in the amygdala that is mediated by PKA and dependent on protein synthesis. Learn Mem 14:497–503

    Article  PubMed  Google Scholar 

  • Huang CC, Lin HJ et al (2007) Repeated cocaine administration promotes long-term potentiation induction in rat medial prefrontal cortex. Cereb Cortex 17:1877–1888

    Article  PubMed  Google Scholar 

  • Humeau Y, Shaban H et al (2003) Presynaptic induction of heterosynaptic associative plasticity in the mammalian brain. Nature 426:841–845

    Article  CAS  PubMed  Google Scholar 

  • Ikemoto S, Panksepp J (1999) The role of nucleus accumbens dopamine in motivated behavior: a unifying interpretation with special reference to reward-seeking. Brain Res Brain Res Rev 31:6–41

    Article  CAS  PubMed  Google Scholar 

  • Izhikevich EM (2007) Solving the distal reward problem through linkage of stdp and dopamine signaling. Cereb Cortex 17:2443–2452

    Article  PubMed  Google Scholar 

  • Joel D, Niv Y et al (2002) Actor-critic models of the basal ganglia: new anatomical and computational perspectives. Neural Netw 15:535–547

    Article  PubMed  Google Scholar 

  • Kaelbling LP, Littman ML et al (1996) Reinforcement learning: a survey. JAIR 4:237–285

    Google Scholar 

  • Kelley AE, Berridge KC (2002) The neuroscience of natural rewards: relevance to addictive drugs. J Neurosci 22:3306–3311

    CAS  PubMed  Google Scholar 

  • Kerr JN, Wickens JR (2001) Dopamine D-1/D-5 receptor activation is required for long-term potentiation in the rat neostriatum in vitro. J Neurophysiol 85:117–124

    CAS  PubMed  Google Scholar 

  • Kilpatrick MR, Rooney MB et al (2000) Extracellular dopamine dynamics in rat caudate-putamen during experimenter-delivered and intracranial self-stimulation. Neuroscience 96:697–706

    Article  CAS  PubMed  Google Scholar 

  • Knowlton BJ, Mangels JA et al (1996) A neostriatal habit learning system in humans. Science 273:1399–1402

    Article  CAS  PubMed  Google Scholar 

  • Kobayashi S, Schultz W (2008) Influence of reward delays on responses of dopamine neurons. J Neurosci 28:7837–7846

    Article  CAS  PubMed  Google Scholar 

  • Kolomiets B, Marzo A et al (2009) Background dopamine concentration dependently facilitates long-term potentiation in rat prefrontal cortex through postsynaptic activation of extracellular signal-regulated kinases. Cereb Cortex 19:2708–2718

    Article  CAS  PubMed  Google Scholar 

  • Kombian SB, Malenka RC (1994) Simultaneous LTP of non-NMDA- and LTD of NMDA-receptor-mediated responses in the nucleus accumbens. Nature 368:242–246

    Article  CAS  PubMed  Google Scholar 

  • Konda VR, Borkar VS (1999) Actor-critic—type learning algorithms for markov decision processes. SIAM J Control Optim 38:94–123

    Article  Google Scholar 

  • Koob GF (1992) Drugs of abuse: anatomy, pharmacology and function of reward pathways. Trends Pharmacol Sci 13:177–184

    Google Scholar 

  • Kötter R, Wickens J (1995) Interactions of glutamate and dopamine in a computational model of the striatum. J Comput Neurosci 2:195–214

    Article  PubMed  Google Scholar 

  • Kroener S, Chandler LJ et al (2009) Dopamine modulates persistent synaptic activity and enhances the signal-to-noise ratio in the prefrontal cortex. PLoS One 4:e6507

    Article  PubMed  CAS  Google Scholar 

  • Lapish CC, Kroener S et al (2007) The ability of the mesocortical dopamine system to operate distinct temporal modes. Psychopharmacology (Berl) 191:609–626

    Article  CAS  Google Scholar 

  • Legenstein R, Pecevski D et al (2008) A learning theory for reward-modulated spike-timing-dependent plasticity with application to biofeedback. PLoS Comput Biol 4:e1000180

    Article  PubMed  CAS  Google Scholar 

  • Li Y, Kauer JA (2004) Repeated exposure to amphetamine disrupts dopaminergic modulation of excitatory synaptic plasticity and neurotransmission in nucleus accumbens. Synapse 51:1–10

    Article  CAS  PubMed  Google Scholar 

  • Lindvall O, Bjorklund A (1978) Anatomy of the dopaminergic neuron systems in the rat brain. Adv Biochem Psychopharmacol 19:1–23

    CAS  PubMed  Google Scholar 

  • Ljungberg T, Apicella P et al (1992) Responses of monkey dopamine neurons during learning of behavioral reactions. J Neurophys 67:145–163

    CAS  Google Scholar 

  • Louilot A, Le Moal M et al (1986) Differential reactivity of dopaminergic neurons in the nucleus accumbens in response to different behavioral situations. An in vivo voltammetric study in free moving rats. Brain Res 397(2):395–400

    Article  CAS  PubMed  Google Scholar 

  • Mackintosh NJ (1983) Conditioning and associative learning. Oxford University Press, New York

    Google Scholar 

  • Maher BJ, Westbrook GL (2008) Co-transmission of dopamine and GABA in periglomerular cells. J Neurophysiol 99:1559–1564

    Article  CAS  PubMed  Google Scholar 

  • Markram H, Lübke J et al (1997) Regulation of synaptic effocacy by coincidence of postsynaptic aps and epsps. Science 275:213–215

    Article  CAS  PubMed  Google Scholar 

  • Matsumoto M, Hikosaka O (2009) Two types of dopamine neuron distinctly convey positive and negative motivational signals. Nature 459:837–841

    Article  CAS  PubMed  Google Scholar 

  • McGaugh JL (2002) Memory consolidation and the amygdala: a systems perspective. Trends Neurosci 25:456

    Article  CAS  PubMed  Google Scholar 

  • Mirenowicz J, Schultz W (1994) Importance of unpredictability for reward responses in primate dopamine neurons. J Neurophys 72:1024–1027

    CAS  Google Scholar 

  • Mirenowicz J, Schultz W (1996) Preferential activation of midbrain dopamine neurons by appetitive rather than aversive stimuli. Nature 379:449–451

    Article  CAS  PubMed  Google Scholar 

  • Montague PR, Dayan P et al (1996) A framework for mesencephalic dopamine systems based on predictive hebbian learning. J Neurosci 16:1936–1947

    CAS  PubMed  Google Scholar 

  • Montague PR, Hyman SE et al (2004a) Computational roles for dopamine in behavioural control. Nature 431:760–767

    Article  CAS  PubMed  Google Scholar 

  • Montague PR, McClure SM et al (2004b) Dynamic gain control of dopamine delivery in freely moving animals. J Neurosci 24:1754–1759

    Article  CAS  PubMed  Google Scholar 

  • Morris G, Arkadir D et al (2004) Coincident but distinct messages of midbrain dopamine and striatal tonically active neurons. Neuron 43:133–143

    Article  CAS  PubMed  Google Scholar 

  • Morris G, Nevet A et al (2006) Midbrain dopamine neurons encode decisions for future action. Nature Neurosci 9:1057–1063

    Article  CAS  PubMed  Google Scholar 

  • Moustafa AA, Cohen MX et al (2008) A role for dopamine in temporal decision making and reward maximization in parkinsonism. J Neurosci 28:12294–12304

    Article  CAS  PubMed  Google Scholar 

  • Nakahara H, Doya K et al (2001) Parallel cortico-basal ganglia mechanisms for acquisition and execution of visuomotor sequences—a computational approach. J Cogn Neurosci 13:626–647

    Article  CAS  PubMed  Google Scholar 

  • Nakahara H, Itoh H et al (2004) Dopamine neurons can represent context-dependent prediction error. Neuron 41:269–280

    Article  CAS  PubMed  Google Scholar 

  • Nicola SM, Surmeier J et al (2000) Dopaminergic modulation of neuronal excitability in the striatum and nucleus accumbens. Annu Rev Neurosci 23:185–215

    Article  CAS  PubMed  Google Scholar 

  • O’Doherty JP, Deichmann R et al (2002) Neural responses during anticipation of a primary taste reward. Neuron 33:815–826

    Article  PubMed  Google Scholar 

  • O’Doherty J, Dayan P et al (2003) Temporal difference models and reward-related learning in the human brain. Neuron 38:329–337

    Article  PubMed  Google Scholar 

  • O’Doherty J, Dayan P et al (2004) Dissociable roles of ventral and dorsal striatum in instrumental conditioning. Science 304:452–454

    Article  PubMed  CAS  Google Scholar 

  • Otani S, Daniel H et al (2003) Dopaminergic modulation of long-term synaptic plasticity in rat prefrontal neurons. Cereb Cortex 13:1251–1256

    Article  PubMed  Google Scholar 

  • Otmakhova NA, Lisman JE (1996) D1/d5 dopamine receptor activation increases the magnitude of early long-term potentiation at ca1 hippocampal synapses. J Neurosci 16:7478–7486

    CAS  PubMed  Google Scholar 

  • Otmakhova NA, Lisman JE (1998) D1/d5 dopamine receptors inhibit depotentiation at ca1 synapses via camp-dependent mechanism. J Neurosci 18:1270–1279

    CAS  PubMed  Google Scholar 

  • Pan WX, Schmidt R et al (2005) Dopamine cells respond to predicted events during classical conditioning: Evidence for eligibility traces in the reward-learning network. J Neurosci 25:6235–6242

    Article  CAS  PubMed  Google Scholar 

  • Pawlak V, Kerr JN (2008) Dopamine receptor activation is required for corticostriatal spike-timing-dependent plasticity. J Neurosci 28:2435–2446

    Article  CAS  PubMed  Google Scholar 

  • Pennartz CM, Ameerun RF et al (1993) Synaptic plasticity in an in vitro slice preparation of the rat nucleus accumbens. Eur J Neurosci 5:107–117

    Article  CAS  PubMed  Google Scholar 

  • Phillips PE, Stuber GD et al (2003) Subsecond dopamine release promotes cocaine seeking. Nature 422:614–618

    Article  CAS  PubMed  Google Scholar 

  • Potjans W, Morrison A, Diesmann M (2009) A spiking neural network model of an actor-critic learning agent. Neural Comput 21:301–339

    Article  PubMed  Google Scholar 

  • Quirk GJ, Mueller D (2007) Neural mechanisms of extinction learning and retrieval. Neuropsychopharmacology 33:56–72

    Article  PubMed  Google Scholar 

  • Redish AD, Jensen S et al (2007) Reconciling reinforcement learning models with behavioral extinction and renewal: Implications for addiction, relapse, and problem gambling. Psychol Rev 114:784–805

    Article  PubMed  Google Scholar 

  • Reynolds JN, Wickens JR (2002) Dopamine-dependent plasticity of corticostriatal synapses. Neural Netw 15:507–521

    Article  PubMed  Google Scholar 

  • Reynolds JN, Hyland BI et al (2001) A cellular mechanism of reward-related learning. Nature 413:67–70

    Article  CAS  PubMed  Google Scholar 

  • Robbe D, Alonso G et al (2002) Role of p/q-Ca2+ channels in metabotropic glutamate receptor 2/3-dependent presynaptic long-term depression at nucleus accumbens synapses. J Neurosci 22:4346–4356

    CAS  PubMed  Google Scholar 

  • Robbins TW, Everitt BJ (1996) Neurobehavioural mechanisms of reward and motivation. Curr Opin Neurobiol 6:228–236

    Article  CAS  PubMed  Google Scholar 

  • Roberts PD, Santiago RA et al (2008) An implementation of reinforcement learning based on spike timing dependent plasticity. Biol Cybern 99:517–523

    Article  PubMed  Google Scholar 

  • Robertson EM, Cohen DA (2006) Understanding consolidation through the architecture of memories. Neuroscientist 12:261–271

    Article  PubMed  Google Scholar 

  • Robinson TE, Berridge KC (2008) Review. The incentive sensitization theory of addiction: some current issues. Philos Trans R Soc Lond B Biol Sci 363:3137–3146

    Article  PubMed  Google Scholar 

  • Robinson DL, Heien ML et al (2002) Frequency of dopamine concentration transients increases in dorsal and ventral striatum of male rats during introduction of conspecifics. J Neurosci 22:10477–10486

    CAS  PubMed  Google Scholar 

  • Roesch MR, Calu DJ et al (2007) Dopamine neurons encode the better option in rats deciding between differently delayed or sized rewards. Nature Neurosci 10:1615–1624

    Article  CAS  PubMed  Google Scholar 

  • Roitman MF, Stuber GD et al (2004) Dopamine operates as a subsecond modulator of food seeking. J Neurosci 24:1265–1271

    Article  CAS  PubMed  Google Scholar 

  • Rolls ET (2000) Precis of the brain and emotion. Behav Brain Sci 23:177–191 discussion 192–233

    Article  CAS  PubMed  Google Scholar 

  • Romo R, Schultz W (1990) Dopamine neurons of the monkey midbrain: contingencies of responses to active touch during self-initiated arm movements. J Neurophysiol 63:592–606

    CAS  PubMed  Google Scholar 

  • Samejima K, Doya K et al (2003) Inter-module credit assignment in modular reinforcement learning. Neural Netw 16:985–994

    Article  PubMed  Google Scholar 

  • Samson RD, Pare D (2005) Activity-dependent synaptic plasticity in the central nucleus of the amygdala. J Neurosci 25:1847–1855

    Article  CAS  PubMed  Google Scholar 

  • Satoh T, Nakai S et al (2003) Correlated coding of motivation and outcome of decision by dopamine neurons. J Neurosci 23:9913–9923

    CAS  PubMed  Google Scholar 

  • Schimchowitsch S, Vuillez P et al (1991) Systematic presence of GABA-immunoreactivity in the tubero-infundibular and tubero-hypophyseal dopaminergic axonal systems: an ultrastructural immunogold study on several mammals. Exp Brain Res 83:575–586

    Article  CAS  PubMed  Google Scholar 

  • Schoenbaum G, Chiba AA et al (1999) Neural encoding in orbitofrontal cortex and basolateral amygdala during olfactory discrimination learning. J Neurosci 19:1876–1884

    CAS  PubMed  Google Scholar 

  • Schultz W (1992) Activity of dopamine neurons in the behaving primate. Semi Neurosci 4(2):129–138

    Article  Google Scholar 

  • Schultz W (1998) Predictive reward signal of dopamine neurons. J Neurophysiol 80:1–27

    CAS  PubMed  Google Scholar 

  • Schultz W (2002) Getting formal with dopamine and reward. Neuron 36:241–263

    Article  CAS  PubMed  Google Scholar 

  • Schultz W, Dayan P et al (1997) A neural substrate of prediction and reward. Science 275:1593–1599

    Article  CAS  PubMed  Google Scholar 

  • Servan-Schreiber D, Printz H et al (1990) A network model of catecholamine effects: gain, signal-to-noise ratio, and behavior. Science 249:892–895

    Article  CAS  PubMed  Google Scholar 

  • Seymour B, O’Doherty JP et al (2004) Temporal difference models describe higher-order learning in humans. Nature 429:664–667

    Article  CAS  PubMed  Google Scholar 

  • Shen W, Flajolet M et al (2008) Dichotomous dopaminergic control of striatal synaptic plasticity. Science 321:848

    Article  CAS  PubMed  Google Scholar 

  • Suri RE, Bargas J et al (2001) Modeling functions of striatal dopamine modulation in learning and planning. Neuroscience 103:65–85

    Article  CAS  PubMed  Google Scholar 

  • Surmeier DJ, Ding J et al (2007) D1 and D2 dopamine-receptor modulation of striatal glutamatergic signaling in striatal medium spiny neurons. Trends Neurosci 30:228–235

    Article  CAS  PubMed  Google Scholar 

  • Sutton RS (1984) Temporal credit assignment in reinforcement learning Ph.D. dissertation, Department of Computer Science, University of Massachusetts, Amherst, MA. Published as COINS Technical Report 84-2

  • Sutton RS (1988) Learning to predict by the methods of temporal differences. Mach Learn 3:9–44

    Google Scholar 

  • Sutton RS, Barto AG (1990) Time-derivative models of Pavlovian reinforcement. MIT Press, Cambridge

    Google Scholar 

  • Sutton RS, Barto AG (1998) Reinforcement learning, an introduction. MIT Press, Cambridge

    Google Scholar 

  • Takahashi Y, Schoenbaum G et al (2008) Silencing the critics: understanding the effects of cocaine sensitization on dorsolateral and ventral striatum in the context of an actor/critic model. Front Neurosci 2:86–99

    Article  PubMed  Google Scholar 

  • Taverna S, Pennartz CM (2003) Postsynaptic modulation of AMPA- and NMDA-receptor currents by group III metabotropic glutamate receptors in rat nucleus accumbens. Brain Res 976:60–68

    Article  CAS  PubMed  Google Scholar 

  • Thivierge JP, Rivest F et al (2007) Spiking neurons, dopamine, and plasticity: timing is everything, but concentration also matters. Synapse 61:375–390

    Article  CAS  PubMed  Google Scholar 

  • Thomas MJ, Beurrier C et al (2001) Long-term depression in the nucleus accumbens: a neural correlate of behavioral sensitization to cocaine. Nat Neurosci 4:1217–1223

    Article  CAS  PubMed  Google Scholar 

  • Tremblay L, Schultz W (2000a) Modifications of reward expectation-related neuronal activity during learning in primate orbitofrontal cortex. J Neurophysiol 83:1877–1885

    CAS  PubMed  Google Scholar 

  • Tremblay L, Schultz W (2000b) Reward-related neuronal activity during go-nogo task performance in primate orbitofrontal cortex. J Neurophysiol 83:1864–1876

    CAS  PubMed  Google Scholar 

  • Vijayraghavan S, Wang M et al (2007) Inverted-U dopamine D1 receptor actions on prefrontal neurons engaged in working memory. Nat Neurosci 10:376–384

    Article  CAS  PubMed  Google Scholar 

  • Voon V, Reynolds B, Brezing C et al (2010) Impulsive choice and response in dopamine agonist-related impulse control behaviors. Psychopharmacology (Berl) 207:645–659

    Article  CAS  Google Scholar 

  • Waelti P, Dickinson A et al (2001) Dopamine responses comply with basic assumptions of formal learning theory. Nature 412:43–48

    Article  CAS  PubMed  Google Scholar 

  • Watkins C, Dayan P (1992) Q-learning. Mach Learning 8:279–292

    Google Scholar 

  • White DJ (1993) Markov decision processes. Willey, New York

    Google Scholar 

  • Wickens JR (2009) Synaptic plasticity in the basal ganglia. Behav Brain Res 199:119–128

    Article  PubMed  Google Scholar 

  • Wickens JR, Horvitz JC et al (2007) Dopaminergic mechanisms in actions and habits. J Neuroscience 27:8181

    Article  CAS  Google Scholar 

  • Wiecki TV, Riedinger K et al (2009) A neurocomputational account of catalepsy sensitization induced by D2 receptor blockade in rats: Context dependency, extinction, and renewal. Psychopharmacology 204:265–277

    Article  CAS  PubMed  Google Scholar 

  • Wise RA (1996a) Addictive drugs and brain stimulation reward. Annu Rev Neurosci 19:319–340

    Article  CAS  PubMed  Google Scholar 

  • Wise RA (1996b) Neurobiology of addiction. Curr Opin Neurobiol 6:243–251

    Article  CAS  PubMed  Google Scholar 

  • Wise RA (2004) Dopamine, learning and motivation. Nat Rev Neurosci 5:483–494

    Article  CAS  PubMed  Google Scholar 

  • Wise RA (2005) Forebrain substrates of reward and motivation. J Comp Neurol 493:115–121

    Article  CAS  PubMed  Google Scholar 

  • Wise RA, Hoffman DC (1992) Localization of drug reward mechanisms by intracranial injections. Synapse 10:247–263

    Article  CAS  PubMed  Google Scholar 

  • Wise RA, Rompre PP (1989) Brain dopamine and reward. Annu Rev Psychol 40:191–225

    Article  CAS  PubMed  Google Scholar 

  • Worgotter F, Porr B (2005) Temporal sequence learning, prediction, and control: a review of different models and their relation to biological mechanisms. Neural Comput 17:245–319

    Article  PubMed  Google Scholar 

  • Xie X, Seung HS (2004) Learning in neural networks by reinforcement of irregular spiking. Phys Rev E Stat Nonlin Soft Matter Phys 69:041909

    PubMed  Google Scholar 

  • Yao WD, Spealman RD et al (2008) Dopaminergic signaling in dendritic spines. Biochem Pharmacol 75:2055–2069

    Article  CAS  PubMed  Google Scholar 

  • Yavich L, MacDonald E (2000) Dopamine release from pharmacologically distinct storage pools in rat striatum following stimulation at frequency of neuronal bursting. Brain Res 870:73–79

    Article  CAS  PubMed  Google Scholar 

  • Yin HH, Ostlund SB et al (2008) Reward-guided learning beyond dopamine in the nucleus accumbens: The integrative functions of cortico-basal ganglia networks. Eur J Neurosci 28:1437–1448

    Article  PubMed  Google Scholar 

  • Young AM, Joseph MH et al (1992) Increased dopamine release in vivo in nucleus accumbens and caudate nucleus of the rat during drinking: a microdialysis study. Neuroscience 48:871–876

    Article  CAS  PubMed  Google Scholar 

Download references

Acknowledgments

The authors wish to thank Dr. Ian Fasel, Nathan Insel and Minryung Song for useful comments on the manuscript. R.D.S. was supported by the Canadian Institute of Health Research SIB 171357.

Author information

Authors and Affiliations

  1. Evelyn F. McKnight Brain Institute and Neural Systems, Memory and Aging, University of Arizona, Tucson, AZ, 85724, USA

    R. D. Samson

  2. Department of Cognitive and Linguistic Sciences and Department of Psychology, Brown Institute for Brain Science, Brown University, Providence, RI, 02912, USA

    M. J. Frank

  3. Department of Psychology and Applied Mathematics, University of Arizona, 1501 N. Campbell Av, Life Sciences North, #384, Tucson, AZ, 85724, USA

    Jean-Marc Fellous

Authors
  1. R. D. Samson
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. M. J. Frank
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jean-Marc Fellous
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jean-Marc Fellous.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Samson, R.D., Frank, M.J. & Fellous, JM. Computational models of reinforcement learning: the role of dopamine as a reward signal. Cogn Neurodyn 4, 91–105 (2010). https://doi.org/10.1007/s11571-010-9109-x

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11571-010-9109-x

Keywords

Search

Navigation