Abstract
Rationale
Throughout our lives we constantly assess the costs and benefits of the possible future outcomes of our actions and use this information to guide behavior. There is accumulating evidence that dopamine contributes to a fundamental component of this computation—how rewards are compared with the costs incurred when obtaining them.
Objective
We review the evidence for dopamine’s role in cost–benefit decision making and outline a simple mathematical framework in which to represent the interactions between rewards, costs, behavioral state and dopamine.
Conclusions
Dopamine’s effects on cost–benefit decision making can be modeled using simple utility–function curves. This approach provides a useful framework for modeling existing data and generating experimental hypotheses that can be objectively and quantitatively tested by observing choice behavior without the necessity to account for subjective psychological states such as pleasure or desire. We suggest that dopamine plays a key role in overcoming response costs and enabling high-effort behaviors. A particularly important anatomical site of this action is the core of the nucleus accumbens. Here, dopamine is able to modulate activity originating from the frontal cortical systems that also assess costs and rewards. Internal deprivation states (e.g., hunger and thirst) also help to energize goal-seeking behaviors, probably in part by their rich influence on dopamine, which can in turn modify decision making policies.
Similar content being viewed by others
References
Aberman JE, Salamone JD (1999) Nucleus accumbens dopamine depletions make rats more sensitive to high ratio requirements but do not impair primary food reinforcements. Neuroscience 92:545–552
Avanzi M, Uber E, Bonfa F (2004) Pathological gambling in two patients on dopamine replacement therapy for Parkinson’s disease. Neurol Sci 25:98–101
Balleine BW (2005) Neural bases of food-seeking: affect, arousal and reward in corticostriatolimbic circuits. Physiol Behav 86:717–730
Bassareo V, Tanda G, Petromilli P, Giua C, Di Chiara G (1996) Non-psychostimulant drugs of abuse and anxiogenic drugs activate with differential selectivity dopamine transmission in the nucleus accumbens and in the medial prefrontal cortex of the rat. Psychopharmacology (Berl) 124:293–299
Bautista LM, Tinbergen J, Kacelnik A (2001) To walk or to fly? How birds choose among foraging modes. Proc Natl Acad Sci USA 98:1089–1094
Bechara A, Damasio H, Damasio AR, Lee GP (1999) Different contributions of the human amygdala and ventromedial prefrontal cortex to decision-making. J Neurosci 19:5473–5481
Berlin HA, Rolls ET, Kischka U (2004) Impulsivity, time perception, emotion and reinforcement sensitivity in patients with orbitofrontal cortex lesions. Brain 127:1108–1126
Berridge KC (2000) Measuring hedonic impact in animals and infants: microstructure of affective taste reactivity patterns. Neurosci Biobehav Rev 24:173–198
Berridge KC, Robinson TE (1998) What is the role of dopamine in reward: hedonic impact, reward learning, or incentive salience? Brain Res Brain Res Rev 28:309–369
Berridge KC, Robinson TE (2003) Parsing reward. Trends Neurosci 26:507–513
Berridge KC, Venier IL, Robinson TE (1989) Taste reactivity analysis of 6-hydroxydopamine-induced aphagia: implications for arousal and anhedonia hypotheses of dopamine function. Behav Neurosci 103:36–45
Borgland SL, Taha SA, Sarti F, Fields HL, Bonci A (2006) Orexin A in the VTA is critical for the induction of synaptic plasticity and behavioral sensitization to cocaine. Neuron 49:589–601
Brog JS, Salyapongse A, Deutch AY, Zahm DS (1993) The patterns of afferent innervation of the core and shell in the “accumbens” part of the rat ventral striatum: immunohistochemical detection of retrogradely transported fluoro-gold. J Comp Neurol 338:255–278
Caine SB, Koob GF (1994) Effects of mesolimbic dopamine depletion on responding maintained by cocaine and food. J Exp Anal Behav 61:213–221
Cannon CM, Palmiter RD (2003) Reward without dopamine. J Neurosci 23:10827–10831
Cardinal RN, Robbins TW, Everitt BJ (2000) The effects of d-amphetamine, chlordiazepoxide, alpha-flupenthixol and behavioural manipulations on choice of signalled and unsignalled delayed reinforcement in rats. Psychopharmacology 152:362–375
Cardinal RN, Pennicott DR, Sugathapala CL, Robbins TW, Everitt BJ (2001) Impulsive choice induced in rats by lesions of the nucleus accumbens core. Science 292:2499–2501
Correa M, Carlson BB, Wisniecki A, Salamone JD (2002) Nucleus accumbens dopamine and work requirements on interval schedules. Behav Brain Res 137:179–187
Dayan P, Balleine BW (2002) Reward, motivation, and reinforcement learning. Neuron 36:285–298
Denk F, Walton ME, Jennings KA, Sharp T, Rushworth MFS, Bannerman DM (2005) Differential involvement of serotonin and dopamine systems in cost–benefit decisions about delay or effort. Psychopharmacology (Berl) 179:587–596
Deroche-Gamonet V, Belin D, Piazza PV (2004) Evidence for addiction-like behavior in the rat. Science 305:1014–1017
Dodd ML, Klos KJ, Bower JH, Geda YE, Josephs KA, Ahlskog JE (2005) Pathological gambling caused by drugs used to treat Parkinson disease. Arch Neurol 62:1377–1381
Figlewicz DP, Evans SB, Murphy J, Hoen M, Baskin DG (2003) Expression of receptors for insulin and leptin in the ventral tegmental area/substantia nigra (VTA/SN) of the rat. Brain Res 964:107–115
Fillenz M (2005) In vivo neurochemical monitoring and the study of behaviour. Neurosci Biobehav Rev 29:949–962
Fiorillo CD, Tobler PN, Schultz W (2003) Discrete coding of reward probability and uncertainty by dopamine neurons. Science 299:1898–1902
Ford CP, Mark GP, Williams JT (2006) Properties and opioid inhibition of mesolimbic dopamine neurons vary according to target location. J Neurosci 26:2788–2797
Fulton S, Richard D, Woodside B, Shizgal P (2002) Interaction of CRH and energy balance in the modulation of brain stimulation reward. Behav Neurosci 116:651–659
Hnasko TS, Sotak BN, Palmiter RD (2005) Morphine reward in dopamine-deficient mice. Nature 438:854–857
Hsu M, Bhatt M, Adolphs R, Tranel D, Camerer CF (2005) Neural systems responding to degrees of uncertainty in human decision-making. Science 310:1680–1683
Ikemoto S, Panksepp J (1996) Dissociations between appetitive and consummatory responses by pharmacological manipulations of reward-relevant brain regions. Behav Neurosci 110:331–345
Ikemoto S, Panksepp J (1999) The role of nucleus accumbens dopamine in motivated behavior: a unifying interpretation with special reference to reward-seeking. Brain Res Brain Res Rev 31:6–41
Jiang H, Betancourt L, Smith RG (2006) Ghrelin amplifies dopamine signaling by crosstalk involving formation of growth hormone secretagogue receptor/dopamine receptor subtype 1 heterodimers. Mol Endocrinol 20:1772–1785
Kacelnik A, Marsh B (2002) Cost can increase preference in starlings. Anim Behav 63:245–250
Kelley AE (2004) Ventral striatal control of appetitive motivation: role in ingestive behavior and reward-related learning. Neurosci Biobehav Rev 27:765–776
Kheramin S, Body S, Ho MY, Velazquez-Martinez DN, Bradshaw CM, Szabadi E, Deakin JF, Anderson IM (2004) Effects of orbital prefrontal cortex dopamine depletion on inter-temporal choice: a quantitative analysis. Psychopharmacology (Berl) 175:206–214
Kissinger PT, Hart JB, Adams, R.N. (1973) Voltammetry in brain tissue—a new neurophysiological measurement. Brain Res 55:209–213
Krugel U, Schraft T, Kittner H, Kiess W, Illes P (2003) Basal and feeding-evoked dopamine release in the rat nucleus accumbens is depressed by leptin. Eur J Pharmacol 482:185–187
Kuhnen CM, Knutson B (2005) The neural basis of financial risk taking. Neuron 47:763–770
Leeb K, Parker L, Eikelboom R (1991) Effects of pimozide on the hedonic properties of sucrose: analysis by the taste reactivity test. Pharmacol Biochem Behav 39:895–901
Ljungberg T, Apicella P, Schultz W (1992) Responses of monkey dopamine neurons during learning of behavioral reactions. J Neurophysiol 67:145–163
Long A, Platt M (2005) Decision making: the virtue of patience in primates. Curr Biol 15:R874–R876
Margolis EB, Lock H, Chefer VI, Shippenberg TS, Hjelmstad GO, Fields HL (2006) κ opioids selectively control dopaminergic neurons projecting to the prefrontal cortex. Proc Natl Acad Sci USA 103:2938–2942
Marinelli M, White FJ (2000) Enhanced vulnerability to cocaine self-administration is associated with elevated impulse activity of midbrain dopamine neurons. J Neurosci 20:8876–8885
Marsh B, Kacelnik A (2002) Framing effects and risky decisions in starlings. Proc Natl Acad Sci U S A 99:3352–3355
Matsushima T, Izawa E, Aoki N, Yanagihara S (2003) The mind through chick eyes: memory, cognition and anticipation. Zoolog Sci 20:395–408
McClure SM, Daw ND, Montague PR (2003) A computational substrate for incentive salience. Trends Neurosci 26:423–428
McClure SM, Laibson DI, Loewenstein G, Cohen JD (2004) Separate neural systems value immediate and delayed monetary rewards. Science 306:503–507
Mingote S, Weber SM, Ishiwari K, Correa M, Salamone JD (2005) Ratio and time requirements on operant schedules: effort-related effects of nucleus accumbens dopamine depletions. Eur J Neurosci 21:1749–1757
Mirenowicz J, Schultz W (1994) Importance of unpredictability for reward responses in primate dopamine neurons. J Neurophysiol 72:1024–1027
Mobini S, Body S, Ho MY, Bradshaw CM, Szabadi E, Deakin JF, Anderson IM (2002) Effects of lesions of the orbitofrontal cortex on sensitivity to delayed and probabilistic reinforcement. Psychopharmacology (Berl) 160:290–298
Montague PR, Dayan P, Sejnowski TJ (1996) A framework for mesencephalic dopamine systems based on predictive Hebbian learning. J Neurosci 16:1936–1947
Narita M, Nagumo Y, Hashimoto S, Narita M, Khotib J, Miyatake M, Sakurai T, Yanagisawa M, Nakamachi T, Shioda S, Suzuki (2006) Direct involvement of orexinergic systems in the activation of the mesolimbic dopamine pathway and related behaviors induced by morphine. J Neurosci 26:398–405
Nicola SM, Woodward Hopf F, Hjelmstad GO (2004) Contrast enhancement: a physiological effect of striatal dopamine? Cell Tissue Res 318:93–106
Niv Y, Daw ND, Dayan P (2005) How fast to work: response vigor, motivation and tonic dopamine. In: Weiss Y, Schölkopf B, Platt J (eds) Advances in neural information processing systems. MIT Press, Cambridge, MA, pp 1019–1026
Nowend KL, Arizzi M, Carlson BB, Salamone JD (2001) D1 or D2 antagonism in nucleus accumbens core or dorsomedial shell suppresses lever pressing for food but leads to compensatory increases in chow consumption. Pharmacol Biochem Behav 69:373–382
O’Donnell P (2003) Dopamine gating of forebrain neural ensembles. Eur J Neurosci 17:429–435
Pecina S, Berridge KC, Parker LA (1997) Pimozide does not shift palatability: separation of anhedonia from sensorimotor suppression by taste reactivity. Pharmacol Biochem Behav 58:801–811
Pecina S, Schulkin J, Berridge KC (2006) Nucleus accumbens corticotropin-releasing factor increases cue-triggered motivation for sucrose reward: paradoxical positive incentive effects in stress? BMC Biol 4:8
Phillips PEM, Wightman RM (2003) Critical guidelines for validation of the selectivity of in-vivo chemical microsensors. Trends Anal Chem 22:509–514
Phillips PEM, Stuber GD, Heien ML, Wightman RM, Carelli RM (2003) Subsecond dopamine release promotes cocaine seeking. Nature 422:614–618
Pompilio L, Kacelnik A, Behmer ST (2006) State-dependent learned valuation drives choice in an invertebrate. Science 311:1613–1615
Rebec GV, Christensen JR, Guerra C, Bardo MT (1997) Regional and temporal differences in real-time dopamine efflux in the nucleus accumbens during free-choice novelty. Brain Res 776:61–67
Rescorla RA, Wagner AR (1972) A theory of Pavlovian conditioning: variations in the effectiveness of reinforcement and nonreinforcement. In: Black AH, Prokasy WF (eds) Classical conditioning II: current research and theory. Appleton Century Crofts, New York, pp 64–99
Richards JB, Sabol KE, de Wit H (1999) Effects of methamphetamine on the adjusting amount procedure, a model of impulsive behavior in rats. Psychopharmacology (Berl) 146:432–439
Robinson DL, Phillips PEM, Budygin EA, Trafton BJ, Garris PA, Wightman RM (2001) Sub-second changes in accumbal dopamine during sexual behavior in male rats. Neuroreport 12:2549–2552
Robinson S, Sandstrom, S.M., Denenberg, V.H., Palmiter RD (2005) Distinguishing whether dopamine regulates liking, wanting, and/or learning about rewards. Behav Neurosci 119:5–15
Rogers RD, Everitt BJ, Baldacchino A, Blackshaw AJ, Swainson R, Wynne K, Baker NB, Hunter J, Carthy T, Booker E, London M, Deakin JF, Sahakian BJ, Robbins TW (1999) Dissociable deficits in the decision-making cognition of chronic amphetamine abusers, opiate abusers, patients with focal damage to prefrontal cortex, and tryptophan-depleted normal volunteers: evidence for monoaminergic mechanisms. Neuropsychopharmacology 20:322–339
Roitman MF, Stuber GD, Phillips PEM, Wightman RM, Carelli RM (2004) Dopamine operates as a subsecond modulator of food seeking. J Neurosci 24:1265–1271
Salamone JD (1996) The behavioral neurochemistry of motivation: methodological and conceptual issues in studies of the dynamic activity of nucleus accumbens dopamine. J Neurosci Methods 64:137–149
Salamone JD, Correa M (2002) Motivational views of reinforcement: implications for understanding the behavioral functions of nucleus accumbens dopamine. Behav Brain Res 137:3–25
Salamone JD, Mahan K, Rogers S (1993) Ventrolateral striatal dopamine depletions impair feeding and food handling in rats. Pharmacol Biochem Behav 44:605–610
Salamone JD, Cousins MS, Bucher S (1994) Anhedonia or anergia? Effects of haloperidol and nucleus accumbens dopamine depletion on instrumental response selection in a T-maze cost/benefit procedure. Behav Brain Res 65:221–229
Salamone JD, Wisniecki A, Carlson BB, Correa M (2001) Nucleus accumbens dopamine depletions make animals highly sensitive to high fixed ratio requirements but do not impair primary food reinforcement. Neuroscience 105:863–870
Salamone JD, Correa M, Mingote SM, Weber SM, Farrar AM (2006) Nucleus accumbens dopamine and the forebrain circuitry involved in behavioral activation and effort-related decision making: implications for understanding anergia and psychomotor slowing in depression. Curr Psychiatry Rev 2:267–280
Schultz W (1998) Predictive reward signal of dopamine neurons. J Neurophysiol 80:1–27
Schultz W (2006) Behavioral theories and the neurophysiology of reward. Annu Rev Psychol 57:87–115
Schultz W, Dickinson A (2000) Neuronal coding of prediction errors. Annu Rev Neurosci 23:473–500
Schultz W, Apicella P, Ljungberg T (1993) Responses of monkey dopamine neurons to reward and conditioned stimuli during successive steps of learning a delayed response task. J Neurosci 13:900–913
Schultz W, Dayan P, Montague PR (1997) A neural substrate of prediction and reward. Science 275:1593–1599
Schweimer J, Hauber W (2005) Involvement of the rat anterior cingulate cortex in control of instrumental responses guided by reward expectancy. Learn Mem 12:334–342
Schweimer J, Saft S, Hauber W (2005) Involvement of catecholamine neurotransmission in the rat anterior cingulate in effort-related decision making. Behav Neurosci 119:1687–1692
Sesack SR, Pickel VM (1992) Prefrontal cortical efferents in the rat synapse on unlabeled neuronal targets of catecholamine terminals in the nucleus accumbens septi and on dopamine neurons in the ventral tegmental area. J Comp Neurol 320:145–160
Sokolowski JD, Salamone JD (1998) The role of accumbens dopamine in lever pressing and response allocation: effects of 6-OHDA injected into core and dorsomedial shell. Pharmacol Biochem Behav 59:557–566
Stevens JR, Rosati AG, Ross KR, Hauser MD (2005) Will travel for food: spatial discounting in two new world monkeys. Curr Biol 15:1855–1860
Stuber GD, Wightman RM, Carelli RM (2005) Extinction of cocaine self-administration reveals functionally and temporally distinct dopaminergic signals in the nucleus accumbens. Neuron 46:661–669
Sutton RS, Barto AG (1998) Reinforcement learning: an introduction. MIT Press, Cambridge, MA
Swanson LW (1982) The projections of the ventral tegmental area and adjacent regions: a combined fluorescent retrograde tracer and immunofluorescence study in the rat. Brain Res Bull 9:321–353
Szarfman A, Doraiswamy PM, Tonning JM, Levine JG (2006) Association between pathologic gambling and parkinsonian therapy as detected in the Food and Drug Administration Adverse Event database. Arch Neurol 63:299–300
Tanaka SC, Doya K, Okada G, Ueda K, Okamoto Y, Yamawaki S (2004) Prediction of immediate and future rewards differentially recruits cortico-basal ganglia loops. Nat Neurosci 7:887–893
Tobler PN, Fiorillo CD, Schultz W (2005) Adaptive coding of reward value by dopamine neurons. Science 307:1642–1645
Ungless MA, Singh V, Crowder TL, Yaka R, Ron D, Bonci A (2003) Corticotropin-releasing factor requires CRF binding protein to potentiate NMDA receptors via CRF receptor 2 in dopamine neurons. Neuron 39:401–407
Vanderschuren LJ, Everitt BJ (2004) Drug seeking becomes compulsive after prolonged cocaine self-administration. Science 305:1017–1019
Wade TR, de Wit H, Richards JB (2000) Effects of dopaminergic drugs on delayed reward as a measure of impulsive behavior in rats. Psychopharmacology (Berl) 150:90–101
Waelti P, Dickinson A, Schultz W (2001) Dopamine responses comply with basic assumptions of formal learning theory. Nature 412:43–48
Wakabayashi KT, Fields HL, Nicola SM (2004) Dissociation of the role of nucleus accumbens dopamine in responding to reward-predictive cues and waiting for reward. Behav Brain Res 154:19–30
Walton ME, Bannerman DM, Rushworth MF (2002) The role of rat medial frontal cortex in effort-based decision making. J Neurosci 22:10996–11003
Walton ME, Bannerman DM, Alterescu K, Rushworth MF (2003) Functional specialization within medial frontal cortex of the anterior cingulate for evaluating effort-related decisions. J Neurosci 23:6475–6479
Walton ME, Croxson PL, Rushworth MF, Bannerman DM (2005) The mesocortical dopamine projection to anterior cingulate cortex plays no role in guiding effort-related decisions. Behav Neurosci 119:323–328
Walton ME, Kennerley SW, Bannerman DM, Phillips PEM, Rushworth MF (2006) Weighing up the benefits of work: behavioural and neural analyses of effort-related decision making. Neural Netw 19:1302–1314
Wellman PJ, Davis KW, Nation JR (2005) Augmentation of cocaine hyperactivity in rats by systemic ghrelin. Regul Pept 125:151–154
Wilson C, Nomikos GG, Collu M, Fibiger HC (1995) Dopaminergic correlates of motivated behavior: importance of drive. J Neurosci 15:5169–5178
Winstanley CA, Theobald DE, Cardinal RN, Robbins TW (2004) Contrasting roles of basolateral amygdala and orbitofrontal cortex in impulsive choice. J Neurosci 24:4718–4722
Winstanley CA, Theobald DE, Dalley JW, Robbins TW (2005) Interactions between serotonin and dopamine in the control of impulsive choice in rats: therapeutic implications for impulse control disorders. Neuropsychopharmacology 30:669–682
Winstanley CA, Theobald DE, Dalley JW, Cardinal RN, Robbins, T.W. (2006) Double dissociation between serotonergic and dopaminergic modulation of medial prefrontal and orbitofrontal cortex during a test of impulsive choice. Cereb Cortex 16:106–114
Wise RA (2004) Dopamine, learning and motivation. Nat Rev Neurosci 5:483–494
Wise RA, Spindler J, de Wit H, Gerberg GJ (1978) Neuroleptic-induced “anhedonia” in rats: pimozide blocks reward quality of food. Science 201:262–264
Yun IA, Wakabayashi KT, Fields HL, Nicola SM (2004) The ventral tegmental area is required for the behavioral and nucleus accumbens neuronal firing responses to incentive cues. J Neurosci 24:2923–2933
Zhou QY, Palmiter RD (1995) Dopamine-deficient mice are severely hypoactive, adipsic, and aphagic. Cell 83:1197–1209
Zigman JM, Jones JE, Lee CE, Saper CB, Elmquist JK (2006) Expression of ghrelin receptor mRNA in the rat and the mouse brain. J Comp Neurol 494:528–548
Acknowledgements
The authors would like to thank John Salamone for his intellectual input at the inception of this work. Support was provided from pilot project grant 5 P01 DA015916-04 (NIH/NIDA; PI: Chavkin) to PEMP. MEW was supported by a Human Frontier Science Program Short-term Fellowship.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Phillips, P.E.M., Walton, M.E. & Jhou, T.C. Calculating utility: preclinical evidence for cost–benefit analysis by mesolimbic dopamine. Psychopharmacology 191, 483–495 (2007). https://doi.org/10.1007/s00213-006-0626-6
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00213-006-0626-6