Experimental Brain Research

, Volume 212, Issue 4, pp 593–601 | Cite as

Inter-individual discount factor differences in reward prediction are topographically associated with caudate activation

  • Keiichi Onoda
  • Yasumasa Okamoto
  • Yoshihiko Kunisato
  • Siori Aoyama
  • Kazuhiro Shishida
  • Go Okada
  • Saori C. Tanaka
  • Nicolas Schweighofer
  • Shuhei Yamaguchi
  • Kenji Doya
  • Shigeto Yamawaki
Research Article

Abstract

In general, humans tend to devalue a delayed reward. Such delay discounting is a theoretical and computational concept in which the discount factor influences the time scale of the trade-off between delay of reward and amount of reward. The discount factor relies on the individual’s ability to evaluate the future reward. Using functional magnetic resonance imaging, we investigated brain mechanisms for reward valuation at different individual discount factors in a delayed reward choice task. In the task, participants were required to select small/immediate or large/delayed rewards to maximize the total reward over time. The discount factor for each participant individually was calculated from the behavioral data based on an exponential discounting model. The estimated value of a future reward increases as the expected delivery approaches, so the time course of these estimated values was computed based on each individual’s discount factor; each was entered into the regression analysis as an explanatory (independent) variable. After the region of interest was narrowed anatomically to the caudate, a peak coordinate was detected in each individual. A correlation analysis revealed that the location of the peak along the dorsal–ventral axis in the right caudate was positively correlated with the discount factor. This implies that individuals who showed a larger discount factor had peak activations in a more dorsal part of the right caudate associated with future reward prediction. This evidence also suggests that a higher ability to delay reward prediction might be related to activation of the more dorsal caudate.

Keywords

Delayed reward Delay discounting Caudate Individual difference fMRI 

References

  1. Ainslie G (1975) Specious reward: a behavioral theory of impulsiveness and impulse control. Psychol Bull 82:463–496PubMedCrossRefGoogle Scholar
  2. Ashburner J (2007) A fast diffeomorphic image registration algorithm. Neuroimage 38:95–113. doi: 10.1016/j.neuroimage.2007.07.007 PubMedCrossRefGoogle Scholar
  3. Ballard K, Knutson B (2009) Dissociable neural representations of future reward magnitude and delay during temporal discounting. Neuroimage 45:143–150. doi: 10.1016/j.neuroimage.2008.11.004 PubMedCrossRefGoogle Scholar
  4. Botvinick MM, Huffstetler S, McGuire JT (2009) Effort discounting in human nucleus accumbens. Cogn Affect Behav Neurosci 9:16–27. doi: 10.3758/CABN.9.1.16 PubMedCrossRefGoogle Scholar
  5. Breiter HC, Aharon I, Kahneman D, Dale A, Shizgal P (2001) Functional imaging of neural responses to expectancy and experience of monetary gains and losses. Neuron 30:619–639PubMedCrossRefGoogle Scholar
  6. Bromberg-Martin ES, Hikosaka O (2009) Midbrain dopamine neurons signal preference for advance information about upcoming rewards. Neuron 63:119–126. doi: 10.1016/j.neuron.2009.06.009 PubMedCrossRefGoogle Scholar
  7. Cardinal RN, Winstanley CA, Robbins TW, Everitt BJ (2004) Limbic corticostriatal systems and delayed reinforcement. Ann NY Acad Sci 1021:33–50. doi: 10.1196/annals.1308.004 PubMedCrossRefGoogle Scholar
  8. Cummings JL (1993) Frontal-subcortical circuits and human behavior. Arch Neurol 50:873–880PubMedGoogle Scholar
  9. Doya K (2002) Metalearning and neuromodulation. Neural Netw 15:495–506PubMedCrossRefGoogle Scholar
  10. Evenden J (1999) Impulsivity: a discussion of clinical and experimental findings. J Psychopharmacol 13:180–192PubMedCrossRefGoogle Scholar
  11. Figner B, Knoch D, Johnson EJ, Krosch AR, Lisanby SH, Fehr E, Weber EU (2010) Lateral prefrontal cortex and self-control in intertemporal choice. Nat Neurosci 13:538–539. doi: 10.1038/nn.2516 PubMedCrossRefGoogle Scholar
  12. Green L, Myerson J, McFadden E (1997) Rate of temporal discounting decreases with amount of reward. Mem Cognit 25:715–723PubMedCrossRefGoogle Scholar
  13. Gregorios-Pippas L, Tobler PN, Schultz W (2009) Short-term temporal discounting of reward value in human ventral striatum. J Neurophysiol 101:1507–1523. doi: 10.1152/jn.90730.2008 PubMedCrossRefGoogle Scholar
  14. Hariri AR, Brown SM, Williamson DE, Flory JD, de Wit H, Manuck SB (2006) Preference for immediate over delayed rewards is associated with magnitude of ventral striatal activity. J Neurosci 26:13213–13217. doi: 10.1523/JNEUROSCI.3446-06.2006 PubMedCrossRefGoogle Scholar
  15. Horn NR, Dolan M, Elliott R, Deakin JF, Woodruff PW (2003) Response inhibition and impulsivity: an fMRI study. Neuropsychologia 41:1959–1966PubMedCrossRefGoogle Scholar
  16. Huettel SA, Stowe CJ, Gordon EM, Warner BT, Platt ML (2006) Neural signatures of economic preferences for risk and ambiguity. Neuron 49:765–775. doi: 10.1016/j.neuron.2006.01.024 PubMedCrossRefGoogle Scholar
  17. Johnson MW, Bickel WK (2002) Within-subject comparison of real and hypothetical money rewards in delay discounting. J Exp Anal Behav 77:129–146. doi: 10.1901/jeab.2002.77-129 PubMedCrossRefGoogle Scholar
  18. Kable JW, Glimcher PW (2007) The neural correlates of subjective value during intertemporal choice. Nat Neurosci 10:1625–1633. doi: 10.1038/nn2007 PubMedCrossRefGoogle Scholar
  19. Kalivas PW, Volkow ND (2005) The neural basis of addiction: a pathology of motivation and choice. Am J Psychiatry 162:1403–1413. doi: 10.1176/appi.ajp.162.8.1403 PubMedCrossRefGoogle Scholar
  20. Killeen PR (2009) An additive-utility model of delay discounting. Psychol Rev 116:602–619. doi: 10.1037/a0016414 PubMedCrossRefGoogle Scholar
  21. Knutson B, Cooper JC (2005) Functional magnetic resonance imaging of reward prediction. Curr Opin Neurol 18:411–417PubMedCrossRefGoogle Scholar
  22. Knutson B, Taylor J, Kaufman M, Peterson R, Glover G (2005) Distributed neural representation of expected value. J Neurosci 25:4806–4812. doi: 10.1523/JNEUROSCI.0642-05.2005 PubMedCrossRefGoogle Scholar
  23. Koopmans TC (1960) Stationary ordinal utility and impatience. Econometrica 28:287–309CrossRefGoogle Scholar
  24. Laibson D (1997) Golden eggs and hyperbolic discounting. Q J Econ 112:443–477CrossRefGoogle Scholar
  25. Lancaster K (1963) An axiomatic theory of consumer time preference. Int Econ Rev 4:221–231CrossRefGoogle Scholar
  26. Madden GJ, Bickel WK, Jacobs EA (1999) Discounting of delayed rewards in opioid-dependent outpatients: exponential or hyperbolic discounting functions? Exp Clin Psychopharmacol 7:284–293PubMedCrossRefGoogle Scholar
  27. Maldjian JA, Laurienti PJ, Kraft RA, Burdette JH (2003) An automated method for neuroanatomic and cytoarchitectonic atlas-based interrogation of fMRI data sets. Neuroimage 19:1233–1239PubMedCrossRefGoogle Scholar
  28. Marco-Pallares J, Mohammadi B, Samii A, Munte TF (2010) Brain activations reflect individual discount rates in intertemporal choice. Brain Res 1320:123–129. doi: 10.1016/j.brainres.2010.01.025 PubMedCrossRefGoogle Scholar
  29. Mazur JE (2001) Hyperbolic value addition and general models of animal choice. Psychol Rev 108:96–112PubMedCrossRefGoogle Scholar
  30. Mazur JE (2007) Rats’ choices between one and two delayed reinforcers. Learn Behav 35:169–176PubMedCrossRefGoogle Scholar
  31. Mazur JE, Biondi DR (2009) Delay-amount tradeoffs in choices by pigeons and rats: hyperbolic versus exponential discounting. J Exp Anal Behav 91:197–211. doi: 10.1901/jeab.2009.91-197 PubMedCrossRefGoogle Scholar
  32. McClure SM, Laibson DI, Loewenstein G, Cohen JD (2004) Separate neural systems value immediate and delayed monetary rewards. Science 306:503–507. doi: 10.1126/science.1100907 PubMedCrossRefGoogle Scholar
  33. McClure SM, Ericson KM, Laibson DI, Loewenstein G, Cohen JD (2007) Time discounting for primary rewards. J Neurosci 27:5796–5804. doi: 10.1523/JNEUROSCI.4246-06.2007 PubMedCrossRefGoogle Scholar
  34. McKerchar TL, Green L, Myerson J, Pickford TS, Hill JC, Stout SC (2009) A comparison of four models of delay discounting in humans. Behav Process 81:256–259. doi: 10.1016/j.beproc.2008.12.017 CrossRefGoogle Scholar
  35. Myerson J, Green L (1995) Discounting of delayed rewards: models of individual choice. J Exp Anal Behav 64:263–276PubMedCrossRefGoogle Scholar
  36. O’Doherty JP, Deichmann R, Critchley HD, Dolan RJ (2002) Neural responses during anticipation of a primary taste reward. Neuron 33:815–826PubMedCrossRefGoogle Scholar
  37. O’Doherty JP, Dayan P, Friston K, Critchley H, Dolan RJ (2003) Temporal difference models and reward-related learning in the human brain. Neuron 38:329–337PubMedCrossRefGoogle Scholar
  38. O’Doherty J, Dayan P, Schultz J, Deichmann R, Friston K, Dolan RJ (2004) Dissociable roles of ventral and dorsal striatum in instrumental conditioning. Science 304:452–454. doi: 10.1126/science.1094285 PubMedCrossRefGoogle Scholar
  39. Phelps ES, Pollak RA (1968) On second-best national saving and game-equilibrium growth. Rev Econ Stud 35:185–199CrossRefGoogle Scholar
  40. Pine A, Seymour B, Roiser JP, Bossaerts P, Friston KJ, Curran HV, Dolan RJ (2009) Encoding of marginal utility across time in the human brain. J Neurosci 29:9575–9581. doi: 10.1523/JNEUROSCI.1126-09.2009 PubMedCrossRefGoogle Scholar
  41. Reynolds B, Schiffbauer R (2004) Measuring state changes in human delay discounting: an experiential discounting task. Behav Process 67:343–356. doi: 10.1016/j.beproc.2004.06.003 Google Scholar
  42. Samuelson PA (1937) A note on measurement of utility. Rev Econ Stud 4:155–161CrossRefGoogle Scholar
  43. Schweighofer N, Shishida K, Han CE, Okamoto Y, Tanaka SC, Yamawaki S, Doya K (2006) Humans can adopt optimal discounting strategy under real-time constraints. PLoS Comput Biol 2:e152. doi: 10.1371/journal.pcbi.0020152 PubMedCrossRefGoogle Scholar
  44. Schweighofer N, Tanaka SC, Doya K (2007) Serotonin and the evaluation of future rewards: theory, experiments, and possible neural mechanisms. Ann N Y Acad Sci 1104:289–300. doi: 10.1196/annals.1390.011 PubMedCrossRefGoogle Scholar
  45. Schweighofer N, Bertin M, Shishida K, Okamoto Y, Tanaka SC, Yamawaki S, Doya K (2008) Low-serotonin levels increase delayed reward discounting in humans. J Neurosci 28:4528–4532. doi: 10.1523/JNEUROSCI.4982-07.2008 PubMedCrossRefGoogle Scholar
  46. Sopher B, Sheth A (2005) A deeper look at hyperbolic discounting. Theory Decis 60:219–255CrossRefGoogle Scholar
  47. Sutton RS, Barto AG (1998) Reinforcement learning. MIT press, CambridgeGoogle Scholar
  48. Tanaka SC, Doya K, Okada G, Ueda K, Okamoto Y, Yamawaki S (2004) Prediction of immediate and future rewards differentially recruits cortico-basal ganglia loops. Nat Neurosci 7:887–893. doi: 10.1038/nn1279 PubMedCrossRefGoogle Scholar
  49. Tanaka SC, Schweighofer N, Asahi S, Shishida K, Okamoto Y, Yamawaki S, Doya K (2007) Serotonin differentially regulates short- and long-term prediction of rewards in the ventral and dorsal striatum. PLoS One 2:e1333. doi: 10.1371/journal.pone.0001333 PubMedCrossRefGoogle Scholar
  50. Wittmann M, Paulus MP (2008) Decision making, impulsivity and time perception. Trends Cogn Sci 12:7–12. doi: 10.1016/j.tics.2007.10.004 PubMedCrossRefGoogle Scholar
  51. Wittmann M, Leland DS, Paulus MP (2007) Time and decision making: differential contribution of the posterior insular cortex and the striatum during a delay discounting task. Exp Brain Res 179:643–653. doi: 10.1007/s00221-006-0822-y PubMedCrossRefGoogle Scholar
  52. Xu L, Liang ZY, Wang K, Li S, Jiang T (2009) Neural mechanism of intertemporal choice: from discounting future gains to future losses. Brain Res 1261:65–74. doi: 10.1016/j.brainres.2008.12.061 PubMedCrossRefGoogle Scholar

Copyright information

© Springer-Verlag 2011

Authors and Affiliations

  • Keiichi Onoda
    • 1
  • Yasumasa Okamoto
    • 2
  • Yoshihiko Kunisato
    • 2
  • Siori Aoyama
    • 2
  • Kazuhiro Shishida
    • 2
  • Go Okada
    • 2
  • Saori C. Tanaka
    • 3
  • Nicolas Schweighofer
    • 4
  • Shuhei Yamaguchi
    • 1
  • Kenji Doya
    • 5
  • Shigeto Yamawaki
    • 2
  1. 1.Department of NeurologyShimane UniversityShimaneJapan
  2. 2.Department of Psychiatry and Neurosciences, Division of Frontier Graduate School of Biomedical SciencesHiroshima UniversityHiroshimaJapan
  3. 3.Institute of Social and Economic ResearchOsaka UniversityOsakaJapan
  4. 4.Department of Physical Therapy and BiokinesiologyUniversity of Southern CaliforniaLos AngelesUSA
  5. 5.Department of Computational NeurobiologyAdvanced Telecommunication Research Institute InternationalKyotoJapan

Personalised recommendations