Abstract
Recent findings in animals have challenged the traditional view of the cerebellum solely as the site of motor control, suggesting that the cerebellum may also be important for learning to predict reward from trial-and-error feedback. Yet, evidence for the role of the cerebellum in reward learning in humans is lacking. Moreover, open questions remain about which specific aspects of reward learning the cerebellum may contribute to. Here we address this gap through an investigation of multiple forms of reward learning in individuals with cerebellum dysfunction, represented by cerebellar ataxia cases. Nineteen participants with cerebellar ataxia and 57 age- and sex-matched healthy controls completed two separate tasks that required learning about reward contingencies from trial-and-error. To probe the selectivity of reward learning processes, the tasks differed in their underlying structure: while one task measured incremental reward learning ability alone, the other allowed participants to use an alternative learning strategy based on episodic memory alongside incremental reward learning. We found that individuals with cerebellar ataxia were profoundly impaired at reward learning from trial-and-error feedback on both tasks, but retained the ability to learn to predict reward based on episodic memory. These findings provide evidence from humans for a specific and necessary role for the cerebellum in incremental learning of reward associations based on reinforcement. More broadly, the findings suggest that alongside its role in motor learning, the cerebellum likely operates in concert with the basal ganglia to support reinforcement learning from reward.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Data Availability
All code used to analyze the data in this study may be found here: https://github.com/boomsbloom/ataxia-rl. Data is available upon request from the corresponding authors.
References
Raymond JL, Lisberger SG, Mauk MD. The cerebellum: a neuronal learning machine? Science. 1996;272:1126–31.
Llinás R, Welsh JP. On the cerebellum and motor learning. Curr Opin Neurobiol. 1993;3:958–65.
Ito M, Itō M. The cerebellum and neural control. 1984;(Raven Press)
Marr D. A theory of cerebellar cortex. J Physiol. 1969;202:437–70.
Doya K. What are the computations of the cerebellum, the basal ganglia and the cerebral cortex? Neural Netw. 1999;12:961–74.
Wolpert DM, Miall RC, Kawato M. Internal models in the cerebellum. Trends Cogn Sci. 1998;2:338–47.
Raymond JL, Medina JF. Computational principles of supervised learning in the cerebellum. Annu Rev Neurosci. 2018;41:233–53.
Caligiore D, Arbib MA, Miall RC, Baldassarre G. The super-learning hypothesis: integrating learning processes across cortex, cerebellum and basal ganglia. Neurosci Biobehav Rev. 2019;100:19–34.
Hull C. Prediction signals in the cerebellum: beyond supervised motor learning. eLife. 2020;9:e54073.
Sendhilnathan N, Goldberg ME. The mid-lateral cerebellum is necessary for reinforcement learning. 2020. http://biorxiv.org/lookup/doi/10.1101/2020.03.20.000190
Sendhilnathan N, Semework M, Goldberg ME, Ipata AE. Neural correlates of reinforcement learning in mid-lateral cerebellum. Neuron. 2020;106:188-198.e5.
Sendhilnathan N, Ipata A, Goldberg ME. Mid-lateral cerebellar complex spikes encode multiple independent reward-related signals during reinforcement learning. Nat Commun. 2021;12:6475.
Larry N, Yarkoni M, Lixenberg A, Joshua M. Cerebellar climbing fibers encode expected reward size. eLife. 2019;8:e46870.
Carta I, Chen CH, Schott AL, Dorizan S, Khodakhah K. Cerebellar modulation of the reward circuitry and social behavior. Science. 2019;363:eaav0581.
Heffley W, Hull C. Classical conditioning drives learned reward prediction signals in climbing fibers across the lateral cerebellum. eLife. 2019;8:e46764.
Wagner MJ, Kim TH, Savall J, Schnitzer MJ, Luo L. Cerebellar granule cells encode the expectation of reward. Nature. 2017;544:96–100.
Heffley W, et al. Coordinated cerebellar climbing fiber activity signals learned sensorimotor predictions. Nat Neurosci. 2018;21:1431–41.
Kostadinov D, Beau M, Blanco-Pozo M, Häusser M. Predictive and reactive reward signals conveyed by climbing fiber inputs to cerebellar Purkinje cells. Nat Neurosci. 2019;22:950–62.
Ohmae S, Medina JF. Climbing fibers encode a temporal-difference prediction error during cerebellar learning in mice. Nat Neurosci. 2015;18:1798–803.
Therrien AS, Wolpert DM, Bastian AJ. Effective reinforcement learning following cerebellar damage requires a balance between exploration and motor noise. Brain J Neurol. 2016;139:101–14.
Doya K. Complementary roles of basal ganglia and cerebellum in learning and motor control. Curr Opin Neurobiol. 2000;10:732–9.
King M, Hernandez-Castillo CR, Poldrack RA, Ivry RB, Diedrichsen J. Functional boundaries in the human cerebellum revealed by a multi-domain task battery. Nat Neurosci. 2019;22:1371–8.
Volkow ND, et al. Expectation enhances the regional brain metabolic and the reinforcing effects of stimulants in cocaine abusers. J Neurosci Off J Soc Neurosci. 2003;23:11461–8.
Grant S, et al. Activation of memory circuits during cue-elicited cocaine craving. Proc Natl Acad Sci U S A. 1996;93:12040–5.
Ramnani N, Elliott R, Athwal BS, Passingham RE. Prediction error for free monetary reward in the human prefrontal cortex. Neuroimage. 2004;23:777–86.
Sutton RS, Barto AG. Reinforcement learning: an introduction. 352.
Houk JC, Adams JL, Barto AG. A model of how the basal ganglia generate and use neural signals that predict reinforcement. in Models of information processing in the basal ganglia 249–270 (The MIT Press, 1995).
Rescorla RA, Wagner AR. 3 A theory of Pavlovian conditioning : variations in the effectiveness of reinforcement and nonreinforcement. in 1972
Schultz W, Dayan P, Montague PR. A Neural substrate of prediction and reward. Science. 1997;275:1593–9.
Kuo S-H. Ataxia. Contin Minneap Minn. 2019;25:1036–54.
Duncan K, Semmler A, Shohamy D. Modulating the use of multiple memory systems in value-based decisions with contextual novelty. J Cogn Neurosci. 2019;1–13. https://doi.org/10.1162/jocn_a_01447
Nicholas J, Daw ND, Shohamy D. Uncertainty alters the balance between incremental learning and episodic memory. eLife. 2022;11:e81679.
Hariri AR. The emerging importance of the cerebellum in broad risk for psychopathology. Neuron. 2019;102:17–20.
Bellebaum C, Daum I. Cerebellar involvement in executive control. Cerebellum. 2007;6:184–92.
Beuriat P-A et al. A new insight on the role of the cerebellum for executive functions and emotion processing in adults. Front Neurol. 2020;11
Mannarelli D, et al. The cerebellum modulates attention network functioning: evidence from a cerebellar transcranial direct current stimulation and attention network test study. Cerebellum. 2019;18:457–68.
Litman L, Robinson J, Abberbock T. TurkPrime.com: a versatile crowdsourcing data acquisition platform for the behavioral sciences. Behav Res Methods. 2017;49:433–42.
Hoffman MD, Gelman A. The no-U-turn sampler: adaptively setting path lengths in Hamiltonian Monte Carlo. 31.
Team SD. Stan Reference Manual.
Vehtari A, Gelman A, Gabry J. Practical Bayesian model evaluation using leave-one-out cross-validation and WAIC. Stat Comput. 2017;27:1413–32.
Kalman RE. A new approach to linear filtering and prediction problems. J Basic Eng. 1960;82:35–45.
Nassar MR, Wilson RC, Heasly B, Gold JI. An approximately Bayesian delta-rule model explains the dynamics of belief updating in a changing environment. J Neurosci. 2010;30:12366–78.
Collins AGE, Frank MJ. How much of reinforcement learning is working memory, not reinforcement learning? A behavioral, computational, and neurogenetic analysis. Eur J Neurosci. 2012;35:1024–35.
Yoo AH, Collins AGE. How working memory and reinforcement learning are intertwined: a cognitive, neural, and computational perspective. J Cogn Neurosci. 2022;34:551–68.
Hoche F, Guell X, Vangel MG, Sherman JC, Schmahmann JD. The cerebellar cognitive affective/Schmahmann syndrome scale. Brain. 2018;141:248–70.
Chirino-Pérez A, et al. Mapping the cerebellar cognitive affective syndrome in patients with chronic cerebellar strokes. Cerebellum. 2022;21:208–18.
McDougle SD et al. Continuous manipulation of mental representations is compromised in cerebellar degeneration. Brain J Neurol. 2022;awac072. https://doi.org/10.1093/brain/awac072.
Buckner RL. The cerebellum and cognitive function: 25 years of insight from anatomy and neuroimaging. Neuron. 2013;80:807–15.
Koziol LF, et al. Consensus paper: The cerebellum’s role in movement and cognition. Cerebellum Lond Engl. 2014;13:151–77.
Alexander MP, Gillingham S, Schweizer T, Stuss DT. Cognitive impairments due to focal cerebellar injuries in adults. Cortex J Devoted Study Nerv Syst Behav. 2012;48:980–90.
Amokrane N, Lin C-YR, Desai NA, Kuo S-H. The impact of compulsivity and impulsivity in cerebellar ataxia: a case series. Tremor Hyperkinetic Mov. 10;43
Buckner RL, Krienen FM, Castellanos A, Diaz JC, Yeo BTT. The organization of the human cerebellum estimated by intrinsic functional connectivity. J Neurophysiol. 2011;106:2322–45.
Middleton FA, Strick PL. Cerebellar projections to the prefrontal cortex of the primate. J Neurosci. 2001;21:700–12.
Amokrane N, et al. Impulsivity in cerebellar ataxias: testing the cerebellar reward hypothesis in humans. Mov Disord. 2020;35:1491–3.
Chen TX, et al. Impulsivity trait profiles in patients with cerebellar ataxia and Parkinson disease. Neurology. 2022;99:e176–86.
Thoma P, Bellebaum C, Koch B, Schwarz M, Daum I. The cerebellum is involved in reward-based reversal learning. Cerebellum. 2008;7:433.
Rustemeier M, Koch B, Schwarz M, Bellebaum C. Processing of positive and negative feedback in patients with cerebellar lesions. Cerebellum Lond Engl. 2016;15:425–38.
McDougle SD, et al. Credit assignment in movement-dependent reinforcement learning. Proc Natl Acad Sci. 2016;113:6797–802.
Caligiore D, et al. Consensus paper: Towards a systems-level view of cerebellar function: the interplay between cerebellum, basal ganglia, and cortex. Cerebellum. 2017;16:203–29.
Funding
J.N. was supported by the NSF Graduate Research Fellowship (1644869). S.H.K. was supported by NINDS R01NS104423, NINDS R01 NS118179, NINDS R01 NS124854, and National Ataxia Foundation. D.S. was supported by an NSF CRCNS award (1822619), NIMH R01 MH121093 and the Kavli Foundation.
Author information
Authors and Affiliations
Contributions
J.N., C.R.L., L.M.K., M.K.P., S.K., and D.S. designed the study. C.A., N.D., and C.R.L. collected data from cerebellar ataxia participants. J.N. collected data from healthy control participants. J.N. analyzed the data and prepared all figures. J.N. and C.A. wrote the main manuscript text. All authors reviewed the manuscript.
Corresponding authors
Ethics declarations
Ethical Approval
Informed consent from all participants in the study was obtained with approval from the Columbia University Institutional Review Board.
Competing Interests
The authors declare no competing interests.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Supplementary Information
Below is the link to the electronic supplementary material.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Nicholas, J., Amlang, C., Lin, CY.R. et al. The Role of the Cerebellum in Learning to Predict Reward: Evidence from Cerebellar Ataxia. Cerebellum (2023). https://doi.org/10.1007/s12311-023-01633-2
Accepted:
Published:
DOI: https://doi.org/10.1007/s12311-023-01633-2