Abstract
Midbrain dopamine neurons are activated when reward is greater than predicted, and this error signal could teach target neurons both the value of reward and when it will occur. We used the dopamine error signal to measure how the expectation of reward was distributed over time. Animals were trained with fixed-duration intervals of 1–16 s between conditioned stimulus onset and reward. In contrast to the weak responses that have been observed after short intervals (1–2 s), activations to reward increased steeply and linearly with the logarithm of the interval. Results with varied stimulus-reward intervals suggest that the neural expectation was substantial after just half an interval had elapsed. Thus, the neural expectation of reward in these experiments was not highly precise and the precision declined sharply with interval duration. The neural precision of expectation appeared to be at least qualitatively similar to the precision of anticipatory licking behavior.
Similar content being viewed by others
References
Buhusi, C.V. & Meck, W.H. What makes us tick? Functional and neural mechanisms of interval timing. Nat. Rev. Neurosci. 6, 755–765 (2005).
Mauk, M.D. & Buonomano, D.V. The neural basis of temporal processing. Annu. Rev. Neurosci. 27, 307–340 (2004).
Rao, S.M., Mayer, A.R. & Harrington, D.L. The evolution of brain activation during temporal processing. Nat. Neurosci. 4, 317–323 (2001).
Gibbon, J., Malapani, C., Dale, C.L. & Gallistel, C.R. Toward a neurobiology of temporal cognition: advances and challenges. Curr. Opin. Neurobiol. 7, 170–184 (1997).
Meck, W.H. Neuropharmacology of timing and time perception. Brain Res. Cogn. Brain Res. 3, 227–242 (1996).
Matell, M.S. & Meck, W.H. Cortico-striatal circuits and interval timing: coincidence detection of oscillatory processes. Brain Res. Cogn. Brain Res. 21, 139–170 (2004).
Houk, J., Adams, J. & Barto, A. A model of how the basal ganglia generate and use neural signals that predict reinforcement. Models of Information Processing in the Basal Ganglia (eds Houk, J., Davis, J. & Beiser, D.) 249–270 (MIT Press, Cambridge, Massachusetts, 1995).
Montague, P.R., Dayan, P. & Sejnowski, T.J. A framework for mesencephalic dopamine systems based on predictive Hebbian learning. J. Neurosci. 16, 1936–1947 (1996).
Schultz, W., Dayan, P. & Montague, R.R. A neural substrate of prediction and reward. Science 275, 1593–1599 (1997).
Berns, G.S. & Sejnowski, T.J. A computational model of how the basal ganglia produce sequences. J. Cogn. Neurosci. 10, 108–121 (1998).
Brown, J., Bullock, D. & Grossberg, S. How the basal ganglia use parallel excitatory and inhibitory learning pathways to selectively respond to unexpected rewarding cues. J. Neurosci. 19, 10502–10511 (1999).
Contreras-Vidal, J.L. & Schultz, W. A predictive reinforcement model of dopamine neurons for learning approach behavior. J. Comput. Neurosci. 6, 191–214 (1999).
Suri, R.E. & Schultz, W. A neural network model with dopamine-like reinforcement signal that learns a spatial delayed response task. Neuroscience 91, 871–890 (1999).
Daw, N.D., Courville, A.C. & Touretzky, D.S. Representation and timing in theories of the dopamine system. Neural Comput. 18, 1637–1677 (2006).
Schultz, W., Apicella, P. & Ljungberg, T. Responses of monkey dopamine neurons to reward and conditioned stimuli during successive steps of learning a delayed response task. J. Neurosci. 13, 900–913 (1993).
Hollerman, J.R. & Schultz, W. Dopamine neurons report an error in the temporal prediction of reward during learning. Nat. Neurosci. 1, 304–309 (1998).
Schultz, W. The predictive reward signal of dopamine neurons. J. Neurophysiol. 80, 1–27 (1998).
Fiorillo, C.D., Tobler, P.N. & Schultz, W. Discrete coding of reward probability and uncertainty by dopamine neurons. Science 299, 1898–1902 (2003).
Satoh, T., Nakai, S., Sato, T. & Kimura, M. Correlated coding of motivation and outcome of decision by dopamine neurons. J. Neurosci. 23, 9913–9923 (2003).
Nakahara, H., Itho, H., Kawagoe, R., Takikawa, Y. & Hikosaka, O. Dopamine neurons can represent context-dependent prediction error. Neuron 41, 269–280 (2004).
Morris, G., Arkadir, D., Nevet, A., Vaadia, E. & Bergman, H. Coincident, but distinct, messages of midbrain dopamine and striatal tonically active neurons. Neuron 43, 133–143 (2004).
Pan, W.X., Schmidt, R., Wickens, J.R. & Hyland, B.I. Dopamine cells respond to predicted events during classical conditioning: evidence for eligibility traces in the reward-learning network. J. Neurosci. 25, 6235–6242 (2005).
Tobler, P.N., Fiorillo, C.D. & Schultz, W. Adaptive coding of reward value by dopamine neurons. Science 307, 1642–1645 (2005).
Bayer, H.M. & Glimcher, P.W. Midbrain dopamine neurons encode a quantitative reward prediction error signal. Neuron 47, 129–141 (2005).
Rakitin, B.C. et al. Scalar expectancy theory and peak-interval timing in humans. J. Exp. Psychol. Anim. Behav. Process. 24, 15–33 (1998).
Kacelnik, A. & Brito e Abreu, F. Risky choice and Weber's Law. J. Theor. Biol. 194, 289–298 (1998).
Janssen, P. & Shadlen, M.N. A representation of the hazard rate of elapsed time in macaque area LIP. Nat. Neurosci. 8, 234–241 (2005).
Ghose, G.M. & Maunsell, J.H.R. Attentional modulation in visual cortex depends on task-timing. Nature 419, 616–620 (2002).
Bateson, M. & Kacelnik, A. Accuracy of memory for amount in the foraging starling, Sturnus vulgaris. Anim. Behav. 50, 431–443 (1995).
Komura, Y. et al. Retrospective and prospective coding for predicted reward in the sensory thalamus. Nature 412, 546–549 (2001).
Brody, C.D., Hernandez, A., Zanos, A. & Romo, R. Timing and neural encoding of somatosensory parametric working memory in macaque prefrontal cortex. Cereb. Cortex 13, 1196–1207 (2003).
Leon, M.I. & Shadlen, M.N. Representation of time by neurons in the posterior parietal cortex of the macaque. Neuron 38, 317–327 (2003).
Renoult, L.Roux. S. & Riehle, A. Time is a rubberband: neuronal activity in monkey motor cortex in relation to time estimation. Eur. J. Neurosci. 23, 3098–3108 (2006).
O'Reilly, R.C., Frank, M.J., Hazy, T.E. & Watz, B. PVLV: the primary value and learned value Pavlovian learning algorithm. Behav. Neurosci. 121, 31–49 (2007).
Grossberg, S. & Schmajuk, N.A. Neural dynamics of adaptive timing and temporal discrimination during associative learning. Neural Netw. 2, 79–102 (1989).
Medina, J.F., Nores, W.L. & Mauk, M.D. Inhibition of climbing fibers is a signal for the extinction of conditioned eyelid responses. Nature 416, 330–333 (2002).
Schultz, W. & Romo, R. Responses of nigrostriatal dopamine neurons to high-intensity somatosensory stimulation in the anesthetized monkey. J. Neurophysiol. 57, 201–217 (1987).
Nieder, A. & Miller, E.K. Coding of cognitive magnitude: compressed scaling of numerical information in the primate prefrontal cortex. Neuron 37, 149–157 (2003).
Acknowledgements
This work was supported by grants from the Human Frontiers Science Program (C.D.F.), the Howard Hughes Medical Institute (W.T.N.), the US National Institutes of Health (EY 05603, W.T.N.), the Swiss National Science Funds (W.S.) and the Wellcome Trust (W.S.).
Author information
Authors and Affiliations
Contributions
C.D.F. conducted the experiments, analyzed the data and developed the mathematical model of dopamine responses. C.D.F. and W.S. designed the experiments. C.D.F. wrote the manuscript with feedback from W.T.N. and W.S.
Corresponding author
Supplementary information
Supplementary Text and Figures
Supplementary Figures 1–4, Supplementary Methods and Supplementary Results (PDF 3905 kb)
Rights and permissions
About this article
Cite this article
Fiorillo, C., Newsome, W. & Schultz, W. The temporal precision of reward prediction in dopamine neurons. Nat Neurosci 11, 966–973 (2008). https://doi.org/10.1038/nn.2159
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1038/nn.2159
- Springer Nature America, Inc.
This article is cited by
-
Neural inhibition as implemented by an actor-critic model involves the human dorsal striatum and ventral tegmental area
Scientific Reports (2024)
-
Dopaminergic prediction errors in the ventral tegmental area reflect a multithreaded predictive model
Nature Neuroscience (2023)
-
Blocking D2/D3 dopamine receptors in male participants increases volatility of beliefs when learning to trust others
Nature Communications (2023)
-
Efficient coding of cognitive variables underlies dopamine response and choice behavior
Nature Neuroscience (2022)
-
Functional architecture of executive control and associated event-related potentials in macaques
Nature Communications (2022)