Skip to main content
SpringerLink
Log in

Reinforcement Motor Learning After Cerebellar Damage Is Related to State Estimation

  • Research
  • Published:
The Cerebellum Aims and scope Submit manuscript

Abstract

Recent work showed that individuals with cerebellar degeneration could leverage intact reinforcement learning (RL) to alter their movement. However, there was marked inter-individual variability in learning, and the factors underlying it were unclear. Cerebellum-dependent sensory prediction may contribute to RL in motor contexts by enhancing body state estimates, which are necessary to solve the credit-assignment problem. The objective of this study was to test the relationship between the predictive component of state estimation and RL in individuals with cerebellar degeneration. Individuals with cerebellar degeneration and neurotypical control participants completed two tasks: an RL task that required them to alter the angle of reaching movements and a state estimation task that tested the somatosensory perception of active and passive movement. The state estimation task permitted the calculation of the active benefit shown by each participant, which is thought to reflect the cerebellum-dependent predictive component of state estimation. We found that the cerebellar and control groups showed similar magnitudes of learning with reinforcement and active benefit on average, but there was substantial variability across individuals. Using multiple regression, we assessed potential predictors of RL. Our analysis included active benefit, somatosensory acuity, clinical ataxia severity, movement variability, movement speed, and age. We found a significant relationship in which greater active benefit predicted better learning with reinforcement in the cerebellar, but not the control group. No other variables showed significant relationships with learning. Overall, our results support the hypothesis that the integrity of sensory prediction is a strong predictor of RL after cerebellar damage.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

Data Availability

All experimental data and analysis scripts are available for download at https://osf.io/s86bp/?view_only=c85f779a6f734825be1950dd54145fe0

References

  1. Glickstein M, Strata P, Voogd J. Cerebellum: history. Neuroscience. 2009;162:549–59.

    Article  CAS  PubMed  Google Scholar 

  2. Wolpert DM, Miall RC, Kawato M. Internal models in the cerebellum. Trends Cogn Sci. 1998;2:338–47.

    Article  CAS  PubMed  Google Scholar 

  3. Wolpert DM, Ghahramani Z. Computational principles of movement neuroscience. Nat Neurosci. 2000;3:1212–7.

    Article  CAS  PubMed  Google Scholar 

  4. Krakauer JW, Pine ZM, Ghilardi MF, Ghez C. Learning of visuomotor transformations for vectorial planning of reaching trajectories. J Neurosci. 2000;20:8916–24.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  5. Shadmehr R, Mussa-Ivaldi FA. Adaptive representation of dynamics during learning of a motor task. J Neurosci. 1994;14:3208–24.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  6. Bastian AJ, Martin TA, Keating JG, Thach WT. Cerebellar ataxia: abnormal control of interaction torques across multiple joints. J Neurophysiol. 1996;76:492–509.

    Article  CAS  PubMed  Google Scholar 

  7. Day BL, Thompson PD, Harding AE, Marsden CD. Influence of vision on upper limb reaching movements in patients with cerebellar ataxia. Brain. 1998;121(Pt 2):357–72.

    Article  PubMed  Google Scholar 

  8. Bhanpuri NH, Okamura AM, Bastian AJ. Predicting and correcting ataxia using a model of cerebellar function. Brain. 2014;137:1931–44.

    Article  PubMed  PubMed Central  Google Scholar 

  9. Weiner MJ, Hallett M, Funkenstein HH. Adaptation to lateral displacement of vision in patients with lesions of the central nervous system. Neurology. 1983;33:766–72.

    Article  CAS  PubMed  Google Scholar 

  10. Martin TA, Keating JG, Goodkin HP, Bastian AJ, Thach WT. Throwing while looking through prisms: I. Focal olivocerebellar lesions impair adaptation. Brain. 1996;119:1183–98.

    Article  PubMed  Google Scholar 

  11. Maschke M, Gomez CM, Ebner TJ, Konczak J. Hereditary cerebellar ataxia progressively impairs force adaptation during goal-directed arm movements. J Neurophysiol. 2004;91:230–8.

    Article  PubMed  Google Scholar 

  12. Morton SM, Bastian AJ. Prism adaptation during walking generalizes to reaching and requires the cerebellum. J Neurophysiol. 2004;92:2497–509.

    Article  PubMed  Google Scholar 

  13. Smith MA, Shadmehr R. Intact ability to learn internal models of arm dynamics in Huntington’s disease but not cerebellar degeneration. J Neurophysiol. 2005;93:2809–21.

    Article  PubMed  Google Scholar 

  14. Tseng Y, Diedrichsen J, Krakauer JW, Shadmehr R, Bastian AJ. Sensory prediction errors drive cerebellum-dependent adaptation of reaching. J Neurophysiol. 2007;98:54–62.

    Article  PubMed  Google Scholar 

  15. Izawa J, Criscimagna-Hemminger SE, Shadmehr R. Cerebellar contributions to reach adaptation and learning sensory consequences of action. J Neurosci. 2012;32:4230–9.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  16. Schlerf JE, Xu J, Klemfuss NM, Griffiths TL, Ivry RB. Individuals with cerebellar degeneration show similar adaptation deficits with large and small visuomotor errors. J Neurophysiol. 2013;109:1164–73.

    Article  PubMed  Google Scholar 

  17. Ilg W, Bastian AJ, Boesch S, Burciu RG, Celnik P, Claaßen J, et al. Consensus paper: management of degenerative cerebellar disorders. Cerebellum. 2014;13:248–68.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  18. Morton SM, Bastian AJ. Can rehabilitation help ataxia? Neurology. 2009;73:1818–9.

    Article  PubMed  Google Scholar 

  19. Ilg W, Timmann D. Gait ataxia—specific cerebellar influences and their rehabilitation. Mov Disord. 2013;28:1566–75.

    Article  PubMed  Google Scholar 

  20. Haith AM, Krakauer JW. Model-based and model-free mechanisms of human motor learning. Adv Exp Med Biol. 2013;782:1–21.

    Article  PubMed  PubMed Central  Google Scholar 

  21. Krakauer JW, Hadjiosif AM, Xu J, Wong AL, Haith AM. Motor learning. Compr Physiol. 2019;9:613–63.

    Article  PubMed  Google Scholar 

  22. Therrien AS, Wolpert DM, Bastian AJ. Effective reinforcement learning following cerebellar damage requires a balance between exploration and motor noise. Brain. 2016;139:101–14.

    Article  PubMed  Google Scholar 

  23. McDougle SD, Boggess MJ, Crossley MJ, Parvin D, Ivry RB, Taylor JA. Credit assignment in movement-dependent reinforcement learning. PNAS. 2016;113:6797–802.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  24. Uehara S, Mawase F, Celnik P. Learning similar actions by reinforcement or sensory-prediction errors rely on distinct physiological mechanisms. Cereb Cortex. 2018;28:3478–90.

    Article  PubMed  Google Scholar 

  25. Niv Y. Reinforcement learning in the brain. J Math Psychol. 2009;53:139–54.

    Article  Google Scholar 

  26. Sutton RS, Barto AG. Reinforcement learning: an introduction. Cambridge, Massachusetts: The MIT Press; 1998.

    Google Scholar 

  27. Therrien AS, Statton MA, Bastian AJ. Reinforcement signaling can be used to reduce elements of cerebellar reaching ataxia. Cerebellum. 2021;20:62–73.

    Article  PubMed  PubMed Central  Google Scholar 

  28. Taylor JA, Ivry RB. Cerebellar and prefrontal cortex contributions to adaptation, strategies, and reinforcement learning. In: Ramnani N, editor. Prog Brain Res. Elsevier; 2014. p. 217–53.

    Google Scholar 

  29. Stengel RF. Optimal control and estimation. Dover Publications; 1994.

    Google Scholar 

  30. Wolpert DM, Ghahramani Z, Jordan MI. An internal model for sensorimotor integration. Science. 1995;269:1880–2.

    Article  CAS  PubMed  Google Scholar 

  31. Paillard J, Brouchon M. A proprioceptive contribution to the spatial encoding of position cues for ballistic movements. Brain Res. 1974;71:273–84.

    Article  CAS  PubMed  Google Scholar 

  32. Adamovich SV, Berkinblit MB, Fookson O, Poizner H. Pointing in 3D space to remembered targets. I. Kinesthetic versus visual target presentation. J Neurophysiol. 1998;79:2833–46.

    Article  CAS  PubMed  Google Scholar 

  33. Fuentes CT, Bastian AJ. Where is your arm? Variations in proprioception across space and tasks. J Neurophysiol. 2010;103:164–71.

    Article  PubMed  Google Scholar 

  34. Bhanpuri NH, Okamura AM, Bastian AJ. Active force perception depends on cerebellar function. J Neurophysiol. 2012;107:1612–20.

    Article  PubMed  Google Scholar 

  35. Bhanpuri NH, Okamura AM, Bastian AJ. Predictive modeling by the cerebellum improves proprioception. J Neurosci. 2013;33:14301–6.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  36. Weeks HM, Therrien AS, Bastian AJ. Proprioceptive localization deficits in people with cerebellar damage. Cerebellum. 2017;16:427–37.

    Article  PubMed  PubMed Central  Google Scholar 

  37. Weeks HM, Therrien AS, Bastian AJ. The cerebellum contributes to proprioception during motion. J Neurophysiol. 2017;118:693–702.

    Article  PubMed  PubMed Central  Google Scholar 

  38. Zimmet AM, Cao D, Bastian AJ, Cowan NJ. Cerebellar patients have intact feedback control that can be leveraged to improve reaching. Elife. 2020;9:e53246.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  39. Holland P, Codol O, Galea JM. Contribution of explicit processes to reinforcement-based motor learning. J Neurophysiol. 2018;119:2241–55.

    Article  PubMed  PubMed Central  Google Scholar 

  40. Codol O, Holland PJ, Galea JM. The relationship between reinforcement and explicit control during visuomotor adaptation. Sci Rep. 2018;8:9121.

    Article  PubMed  PubMed Central  Google Scholar 

  41. Butcher PA, Ivry RB, Kuo S-H, Rydz D, Krakauer JW, Taylor JA. The cerebellum does more than sensory prediction error-based learning in sensorimotor adaptation tasks. J Neurophysiol. 2017;118:1622–36.

    Article  PubMed  PubMed Central  Google Scholar 

  42. Tsay JS, Schuck L, Ivry RB. Cerebellar degeneration impairs strategy discovery but not strategy recall. Cerebellum. 2022; Online ahead of print; https://doi.org/10.1007/s12311-022-01500-6.

  43. Wong AL, Marvel CL, Taylor JA, Krakauer JW. Can patients with cerebellar disease switch learning mechanisms to reduce their adaptation deficits? Brain. 2019;142:662–73.

    Article  PubMed  PubMed Central  Google Scholar 

  44. Brand M, Schiebener J. Interactions of age and cognitive functions in predicting decision making under risky conditions over the life span. J Clin Exp Neuropsychol. 2013;35:9–23.

    Article  PubMed  Google Scholar 

  45. Eppinger B, Nystrom LE, Cohen JD. Reduced sensitivity to immediate reward during decision-making in older than younger adults. PLoS One. 2012;7:e36953.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  46. Weller JA, King ML, Figner B, Denburg NL. Information use in risky decision making: do age differences depend on affective context? Psychol Aging. 2019;34:1005–20.

    Article  PubMed  PubMed Central  Google Scholar 

  47. Trouillas P, Takayanagi T, Hallett M, Currier RD, Subramony SH, Wessel K, et al. International Cooperative Ataxia Rating Scale for pharmacological assessment of the cerebellar syndrome. J Neurol Sci. 1997;145:205–11.

    Article  CAS  PubMed  Google Scholar 

  48. Gibo TL, Criscimagna-Hemminger SE, Okamura AM, Bastian AJ. Cerebellar motor learning: are environment dynamics more important than error size? J Neurophysiol. 2013;110:322–33.

    Article  PubMed  PubMed Central  Google Scholar 

  49. Wichmann FA, Hill NJ. The psychometric function: I. Fitting, sampling, and goodness of fit. Percept Psychophys. 2001;63:1293–313.

    Article  CAS  PubMed  Google Scholar 

  50. Mawase F, Uehara S, Bastian AJ, Celnik P. Motor learning enhances use-dependent plasticity. J Neurosci. 2017;37:2673–85.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  51. van Mastrigt NM, Tsay JS, Wang T, Avraham G, Abram SJ, van der Kooij K, et al. Implicit reward-based motor learning. Exp Brain Res. 2023;

  52. Forano M, Franklin DW. Reward actively engages both implicit and explicit components in dual force field adaptation. bioRxiv. 2023:2023.08.09.552587.

  53. Vassiliadis P, Derosiere G, Duque J. Beyond motor noise: considering other causes of impaired reinforcement learning in cerebellar patients. eNeuro. 2019;6 ENEURO.0458-18.2019

  54. Hull C. Prediction signals in the cerebellum: beyond supervised motor learning. eLife. 2020;9:e54073.

    Article  PubMed  PubMed Central  Google Scholar 

  55. Yoshida J, Oñate M, Khatami L, Vera J, Nadim F, Khodakhah K. Cerebellar contributions to the basal ganglia influence motor coordination, reward processing, and movement vigor. J Neurosci. 2022;42:8406–15.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  56. Sendhilnathan N, Ipata A, Goldberg ME. Mid-lateral cerebellar complex spikes encode multiple independent reward-related signals during reinforcement learning. Nat Commun. 2021;12:6475.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  57. Larry N, Yarkoni M, Lixenberg A, Joshua M. Cerebellar climbing fibers encode expected reward size. Elife. 2019;8:e46870.

    Article  PubMed  PubMed Central  Google Scholar 

  58. Thoma P, Bellebaum C, Koch B, Schwarz M, Daum I. The cerebellum is involved in reward-based reversal learning. Cerebellum. 2008;7:433–43.

    Article  PubMed  Google Scholar 

  59. Cardoso CD, Branco LD, Cotrena C, Kristensen CH, Schneider Bakos DD, Fonseca RP. The impact of frontal and cerebellar lesions on decision making: evidence from the Iowa Gambling Task. Front Neurosci. 2014;8:61.

    Article  PubMed  PubMed Central  Google Scholar 

  60. Nicholas J, Amlang C, Lin C-YR, Montaser-Kouhsari L, Desai N, Pan M-K, et al. The role of the cerebellum in learning to predict reward: evidence from cerebellar ataxia. bioRxiv. 2022:2022.11.04.515251.

  61. Lixenberg A, Yarkoni M, Botschko Y, Joshua M. Encoding of eye movements explains reward-related activity in cerebellar simple spikes. J Neurophysiol. 2020;123:786–99.

    Article  PubMed  PubMed Central  Google Scholar 

  62. Powell DJ, Marder E, Nusbaum MP. Perturbation-specific responses by two neural circuits generating similar activity patterns. Curr Biol. 2021;31:4831–4838.e4.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  63. Izawa J, Shadmehr R. Learning from sensory and reward prediction errors during motor adaptation. PLoS Comput Biol. 2011;7:e1002012.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  64. Cashaback JGA, McGregor HR, Mohatarem A, Gribble PL. Dissociating error-based and reinforcement-based loss functions during sensorimotor learning. PLoS Comput Biol. 2017;13:e1005623.

    Article  PubMed  PubMed Central  Google Scholar 

Download references

Funding

This work was supported by pilot project funding from the Moss Rehabilitation Research Institute (MRRI) Peer Review Committee and start-up funding from the MRRI awarded to AST.

Author information

Authors and Affiliations

  1. Moss Rehabilitation Research Institute, Medical Arts Building, Suite 100, 50 Township Line Rd, Elkins Park, PA, USA

    Christopher M. White, Evan C. Snow & Amanda S. Therrien

  2. Department of Rehabilitation Medicine, Thomas Jefferson University, Philadelphia, PA, USA

    Amanda S. Therrien

Authors
  1. Christopher M. White
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Evan C. Snow
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Amanda S. Therrien
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

AST conceived and designed the research, analyzed data, prepared figures, interpreted results, and wrote the manuscript. AST and ECS programmed the experimental tasks. CW and ECS performed the experiments and assisted with data analysis.

Corresponding author

Correspondence to Amanda S. Therrien.

Ethics declarations

Ethical Approval and Consent to Participate

The study was approved by the Einstein Healthcare Network and Jefferson Health Institutional Review Boards. All participants gave written informed consent to participate and to publish the data collected.

Competing Interests

The authors declare that they have no competing interests.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

White, C.M., Snow, E.C. & Therrien, A.S. Reinforcement Motor Learning After Cerebellar Damage Is Related to State Estimation. Cerebellum (2023). https://doi.org/10.1007/s12311-023-01615-4

Download citation

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s12311-023-01615-4

Keywords

Search

Navigation