Advertisement

Hippocampal contributions to value-based learning: Converging evidence from fMRI and amnesia

  • Daniela J. PalomboEmail author
  • Scott M. Hayes
  • Allison G. Reid
  • Mieke Verfaellie
Special Issue/Reward Systems, Cognition,and Emotion

Abstract

Recent evidence suggests that the human hippocampus—known primarily for its involvement in episodic memory—plays a role in a host of motivationally relevant behaviors, including some forms of value-based decision-making. However, less is known about the role of the hippocampus in value-based learning. Such learning is typically associated with a striatal system, yet a small number of studies, both in human and nonhuman species, suggest hippocampal engagement. It is not clear, however, whether this engagement is necessary for such learning. In the present study, we used both functional MRI (fMRI) and lesion-based neuropsychological methods to clarify hippocampal contributions to value-based learning. In Experiment 1, healthy participants were scanned while learning value-based contingencies (whether players in a “game” win money) in the context of a probabilistic learning task. Here, we observed recruitment of the hippocampus, in addition to the expected ventral striatal (nucleus accumbens) activation that typically accompanies such learning. In Experiment 2, we administered this task to amnesic patients with medial temporal lobe damage and to healthy controls. Amnesic patients, including those with damage circumscribed to the hippocampus, failed to acquire value-based contingencies, thus confirming that hippocampal engagement is necessary for task performance. Control experiments established that this impairment was not due to perceptual demands or memory load. Future research is needed to clarify the mechanisms by which the hippocampus contributes to value-based learning, but these findings point to a broader role for the hippocampus in goal-directed behaviors than previously appreciated.

Keywords

hippocampus reward value amnesia 

Notes

Supplementary material

13415_2018_687_MOESM1_ESM.docx (312 kb)
ESM 1 (DOCX 312 kb)

References

  1. Adcock, R. A., Thangavel, A., Whitfield-Gabrieli, S., Knutson, B., & Gabrieli, J. D. (2006). Reward-motivated learning: Mesolimbic activation precedes memory formation. Neuron, 50(3), 507–517.CrossRefGoogle Scholar
  2. Baker, S., Vieweg, P., Gao, F., Gilboa, A., Wolbers, T., Black, S. E., & Rosenbaum, R. S. (2016). The human dentate gyrus plays a necessary role in discriminating new memories. Current Biology, 26(19), 2629–2634.CrossRefGoogle Scholar
  3. Ballard, I. C., Wagner, A. D., & McClure, S. M. (2018). Hippocampal pattern separation supports reinforcement learning.  https://doi.org/10.1101/293332
  4. Beckmann, C. F., Jenkinson, M., & Smith, S. M. (2003). General multilevel linear modeling for group analysis in fMRI. NeuroImage, 20(2), 1052–1063.CrossRefGoogle Scholar
  5. Bornstein, A. M., Khaw, M. W., Shohamy, D., & Daw, N. D. (2017). Reminders of past choices bias decisions for reward in humans. Nature Communications, 8, 15958.  https://doi.org/10.1038/ncomms15958 CrossRefGoogle Scholar
  6. Callan, D. E., & Schweighofer, N. (2008). Positive and negative modulation of word learning by reward anticipation. Human Brain Mapping, 29(2), 237–249.CrossRefGoogle Scholar
  7. Castel, A. D., Farb, N. A., & Craik, F. I. (2007). Memory for general and specific value information in younger and older adults: Measuring the limits of strategic control. Memory & Cognition, 35(4), 689–700.CrossRefGoogle Scholar
  8. Cohen, N. J., Poldrack, R. A., & Eichenbaum, H. (1997). Memory for items and memory for relations in the procedural/declarative memory framework. Memory, 5(1/2), 131–178.CrossRefGoogle Scholar
  9. Davidow, J. Y., Foerde, K., Galvan, A., & Shohamy, D. (2016). An upside to reward sensitivity: The hippocampus supports enhanced reinforcement learning in adolescence. Neuron, 92(1), 93–99.CrossRefGoogle Scholar
  10. Delgado, M. R., Miller, M. M., Inati, S., & Phelps, E. A. (2005). An fMRI study of reward-related probability learning. NeuroImage, 24(3), 862–873.CrossRefGoogle Scholar
  11. Delgado, M. R., Nystrom, L. E., Fissell, C., Noll, D. C., & Fiez, J. A. (2000). Tracking the hemodynamic responses to reward and punishment in the striatum. Journal of Neurophysiology, 84(6), 3072–3077.CrossRefGoogle Scholar
  12. Dickerson, K. C., & Delgado, M. R. (2015). Contributions of the hippocampus to feedback learning. Cognitive, Affective, & Behavioral Neuroscience, 15(4), 861–877.CrossRefGoogle Scholar
  13. Dickerson, K. C., Li, J., & Delgado, M. R. (2011). Parallel contributions of distinct human memory systems during probabilistic learning. NeuroImage, 55(1), 266–276.CrossRefGoogle Scholar
  14. Duncan, K., Doll, B. B., Daw, N. D., & Shohamy, D. (2018). More than the sum of its parts: A role for the hippocampus in configural reinforcement learning. Neuron, 98(3), 645–657.CrossRefGoogle Scholar
  15. Eichenbaum, H., Yonelinas, A. P., & Ranganath, C. (2007). The medial temporal lobe and recognition memory. Annual Reviews of Neuroscience, 30, 123–152.CrossRefGoogle Scholar
  16. Eklund, A., Nichols, T. E., & Knutsson, H. (2016). Cluster failure: Why fMRI inferences for spatial extent have inflated false-positive rates. Proceedings of the National Academy of Sciences, 113(28), 7900–7905.CrossRefGoogle Scholar
  17. Fera, F., Passamonti, L., Herzallah, M. M., Myers, C. E., Veltri, P., Morganti, G., … Gluck, M. A. (2014). Hippocampal BOLD response during category learning predicts subsequent performance on transfer generalization. Human Brain Mapping, 35(7), 3122–3131.CrossRefGoogle Scholar
  18. Floresco, S. B. (2007). Dopaminergic regulation of limbic-striatal interplay. Journal of Psychiatry & Neuroscience 32(6), 400–411.Google Scholar
  19. Foerde, K., Race, E., Verfaellie, M., & Shohamy, D. (2013). A role for the medial temporal lobe in feedback-driven learning: Evidence from amnesia. Journal of Neuroscience, 33(13), 5698–5704.CrossRefGoogle Scholar
  20. Foerde, K., & Shohamy, D. (2011). Feedback timing modulates brain systems for learning in humans. Journal of Neuroscience, 31(37), 13157-13167.CrossRefGoogle Scholar
  21. Gluck, M. A., Ermita, B. R., Oliver, L. M., & Myers, C. E. (1997). Extending models of hippocampal function in animal conditioning to human amnesia. Memory, 5(1/2), 179–212.CrossRefGoogle Scholar
  22. Gluck, M. A., Shohamy, D., & Myers, C. (2002). How do people solve the “weather prediction” task?: Individual variability in strategies for probabilistic category learning. Learning & Memory, 9(6), 408–418.CrossRefGoogle Scholar
  23. Groenewegen, H. J., Vermeulen-Van der Zee, E., te Kortschot, A., & Witter, M. P. (1987). Organization of the projections from the subiculum to the ventral striatum in the rat: A study using anterograde transport of Phaseolus vulgaris leucoagglutinin. Neuroscience, 23(1), 103–120.CrossRefGoogle Scholar
  24. Hopkins, R. O., Myers, C. E., Shohamy, D., Grossman, S., & Gluck, M. (2004). Impaired probabilistic category learning in hypoxic subjects with hippocampal damage. Neuropsychologia, 42(4), 524–535.CrossRefGoogle Scholar
  25. Howard, M. W., & Eichenbaum, H. (2015). Time and space in the hippocampus. Brain Research, 1621, 345–354.CrossRefGoogle Scholar
  26. Jenkinson, M., Bannister, P., Brady, M., & Smith, S. (2002). Improved optimization for the robust and accurate linear registration and motion correction of brain images. NeuroImage, 17(2), 825–841.CrossRefGoogle Scholar
  27. Kan, I. P., Giovanello, K. S., Schnyer, D. M., Makris, N., & Verfaellie, M. (2007). Role of the medial temporal lobes in relational memory: Neuropsychological evidence from a cued recognition paradigm. Neuropsychologia, 45(11), 2589-2597.CrossRefGoogle Scholar
  28. Kelley, A. E., & Domesick, V. B. (1982). The distribution of the projection from the hippocampal formation to the nucleus accumbens in the rat: An anterograde- and retrograde-horseradish peroxidase study. Neuroscience, 7(10), 2321–2335.CrossRefGoogle Scholar
  29. Knowlton, B. J., Squire, L. R., & Gluck, M. A. (1994). Probabilistic classification learning in amnesia. Learning & Memory, 1(2), 106–120.Google Scholar
  30. Lee, H., Ghim, J. W., Kim, H., Lee, D., & Jung, M. (2012). Hippocampal neural correlates for values of experienced events. Journal of Neuroscience, 32(43), 15053–15065.CrossRefGoogle Scholar
  31. Leutgeb, J. K., Leutgeb, S., Moser, M. B., & Moser, E. I. (2007). Pattern separation in the dentate gyrus and CA3 of the hippocampus. Science, 315(5814), 961–966.CrossRefGoogle Scholar
  32. Li, J., Delgado, M. R., & Phelps, E. A. (2011). How instructed knowledge modulates the neural systems of reward learning. Proceedings of the National Academy of Sciences, 108(1), 55–60.CrossRefGoogle Scholar
  33. Lighthall, N. R., Pearson, J. M., Huettel, S. A., & Cabeza, R. (2018). Feedback-based learning in aging: Contributions and trajectories of change in striatal and hippocampal systems. Journal of Neuroscience, 38(39), 8453–8462.CrossRefGoogle Scholar
  34. Lisman, J. E., & Grace, A. A. (2005). The hippocampal-VTA loop: Controlling the entry of information into long-term memory. Neuron, 46(5), 703–713.CrossRefGoogle Scholar
  35. Loh, E., Kumaran, D., Koster, R., Berron, D., Dolan, R., & Duzel, E. (2016). Context-specific activation of hippocampus and SN/VTA by reward is related to enhanced long-term memory for embedded objects. Neurobiology of Learning and Memory, 134(Pt. A), 65–77.CrossRefGoogle Scholar
  36. Madan, C. R., Fujiwara, E., Gerson, B. C., & Caplan, J. B. (2012). High reward makes items easier to remember, but harder to bind to a new temporal context. Frontiers in Integrative Neuroscience, 6, 61.CrossRefGoogle Scholar
  37. Mather, M., & Schoeke, A. (2011). Positive outcomes enhance incidental learning for both younger and older adults. Frontiers in Neuroscience, 5, 129.CrossRefGoogle Scholar
  38. Moeller, S., Yacoub, E., Olman, C. A., Auerbach, E., Strupp, J., Harel, N., & Uğurbil K. (2010). Multiband multislice GE-EPI at 7 tesla, with 16-fold acceleration using partial parallel imaging with application to high spatial and temporal whole-brain fMRI. Magnetic Resonance in Medicine, 63(5), 1144–1153.CrossRefGoogle Scholar
  39. Murty, V. P., & Adcock, R. A. (2014). Enriched encoding: Reward motivation organizes cortical networks for hippocampal detection of unexpected events. Cerebral Cortex, 24(8), 2160–2168.CrossRefGoogle Scholar
  40. Murty, V. P., LaBar, K. S., & Adcock, R. A. (2016). Distinct medial temporal networks encode surprise during motivation by reward versus punishment. Neurobiology of Learning and Memory, 134(Pt. A), 55–64.CrossRefGoogle Scholar
  41. Palombo, D. J., Di Lascio, J. M., Howard, M. W., & Verfaellie, M. (2019,). Medial temporal lobe amnesia is associated with a deficit in recovering temporal context. Journal of Cognitive Neuroscience , 31(2), 236-248.Google Scholar
  42. Palombo, D. J., Keane, M. M., & Verfaellie, M. (2015). How does the hippocampus shape decisions? Neurobiology of Learning and Memory, 125, 93–97.CrossRefGoogle Scholar
  43. Palombo, D. J., & Verfaellie, M. (2017). Hippocampal contributions to memory for time: Evidence from neuropsychological studies. Current Opinion in Behavioral Sciences, 17, 107–113.CrossRefGoogle Scholar
  44. Poldrack, R. A., Clark, J., Pare-Blagoev, E. J., Shohamy, D., Creso Moyano, J., Myers, C., & Cluck, M. A. (2001). Interactive memory systems in the human brain. Nature, 414(6863), 546–550.CrossRefGoogle Scholar
  45. Poldrack, R. A., & Packard, M. G. (2003). Competition among multiple memory systems: Converging evidence from animal and human brain studies. Neuropsychologia, 41(3), 245–251.CrossRefGoogle Scholar
  46. Poppenk, J., Evensmoen, H. R., Moscovitch, M., & Nadel, L. (2013). Long-axis specialization of the human hippocampus. Trends in Cognitive Sciences, 17(5), 230–240.CrossRefGoogle Scholar
  47. Preuschoff, K., Bossaerts, P., & Quartz, S. R. (2006). Neural differentiation of expected reward and risk in human subcortical structures. Neuron, 51(3), 381–390.CrossRefGoogle Scholar
  48. Pruim, R. H., Mennes, M., van Rooij, D., Llera, A., Buitelaar, J. K., & Beckmann, C. F. (2015). ICA-AROMA: A robust ICA-based strategy for removing motion artifacts from fMRI data. NeuroImage, 112, 267–277.CrossRefGoogle Scholar
  49. Roiser, J. P., Linden, D. E., Gorno-Tempinin, M. L., Moran, R. J., Dickerson, B. C., & Grafton, S. T. (2016). Minimum statistical standards for submissions to Neuroimage: Clinical. NeuroImage: Clinical, 12, 1045–1047.Google Scholar
  50. Schonberg, T., O’Doherty, J. P., Joel, D., Inzelberg, R., Segev, Y., & Daw, N. D. (2010). Selective impairment of prediction error signaling in human dorsolateral but not ventral striatum in Parkinson’s disease patients: Evidence from a model-based fMRI study. NeuroImage, 49(1), 772–781.CrossRefGoogle Scholar
  51. Sheldon, S., & Levine, B. (2016). The role of the hippocampus in memory and mental construction. Annals of the New York Academy of Sciences, 1369(1), 76–92.CrossRefGoogle Scholar
  52. Sheldon, S., Romero, K., & Moscovitch, M. (2013). Medial temporal lobe amnesia impairs performance on a free association task. Hippocampus, 23(5), 405–412.CrossRefGoogle Scholar
  53. Shohamy, D., & Adcock, R. A. (2010). Dopamine and adaptive memory. Trends in Cognitive Sciences, 14(10), 464–472.CrossRefGoogle Scholar
  54. Shohamy, D., Myers, C. E., Hopkins, R. O., Sage, J., & Gluck, M. A. (2009). Distinct hippocampal and basal ganglia contributions to probabilistic learning and reversal. Journal of Cognitive Neuroscience, 21(9), 1821–1833.CrossRefGoogle Scholar
  55. Shohamy, D., Myers, C. E., Kalanithi, J., & Gluck, M. A. (2008). Basal ganglia and dopamine contributions to probabilistic category learning. Neuroscience and Biobehavioral Reviews, 32(2), 219–236.CrossRefGoogle Scholar
  56. Shohamy, D., & Turk-Browne, N. B. (2013). Mechanisms for widespread hippocampal involvement in cognition. Journal of Experimental Psychology: General, 142(4), 1159–1170.CrossRefGoogle Scholar
  57. Spaniol, J., Schain, C., & Bowen, H. J. (2014). Reward-enhanced memory in younger and older adults. The Journals of Gerontology. Series B, Psychological Sciences and Social Sciences, 69(5), 730–740.CrossRefGoogle Scholar
  58. Squire, L. R. (2004). Memory systems of the brain: a brief history and current perspective. Neurobiology of Learning and Memory, 82(3), 171–177.CrossRefGoogle Scholar
  59. Stachenfeld, K. L., Botvinick, M. M., & Gershman, S. J. (2017). The hippocampus as a predictive map. Nature Neuroscience, 20(11), 1643–1653.CrossRefGoogle Scholar
  60. Strange, B. A., Witter, M. P., Lein, E. S., & Moser, E. I. (2014). Functional organization of the hippocampal longitudinal axis. Nature Reviews Neuroscience, 15(10), 655–669.CrossRefGoogle Scholar
  61. Wechsler, D. (1997). Wechsler Adult Intelligence Scale–Third Edition: Administration and scoring manual. San Antonio, TX: Harcourt Assessment.Google Scholar
  62. Wimmer, G. E., Daw, N. D., & Shohamy, D. (2012). Generalization of value in reinforcement learning by humans. Eur J Neurosci, 35(7), 1092-1104.CrossRefGoogle Scholar
  63. Wittmann, B. C., Bunzeck, N., Dolan, R. J., & Duzel, E. (2007). Anticipation of novelty recruits reward system and hippocampus while promoting recollection. NeuroImage, 38(1), 194–202.CrossRefGoogle Scholar
  64. Woo, C. W., Krishnan, A., & Wager, T. D. (2014). Cluster-extent based thresholding in fMRI analyses: Pitfalls and recommendations. NeuroImage, 91, 412–419.CrossRefGoogle Scholar
  65. Woolrich, M. W., Behrens, T. E., Beckmann, C. F., Jenkinson, M., & Smith, S. M. (2004). Multilevel linear modelling for fMRI group analysis using Bayesian inference. Neuroimage, 21(4), 1732-1747.CrossRefGoogle Scholar
  66. Woolrich, M. W., Ripley, B. D., Brady, M., & Smith, S. M. (2001). Temporal autocorrelation in univariate linear modeling of fMRI data. NeuroImage, 14(6), 1370–1386.CrossRefGoogle Scholar

Copyright information

© This is a U.S. government work and its text is not subject to copyright protection in the United States; however, its text may be subject to foreign copyright protection 2019

Authors and Affiliations

  1. 1.VA Boston Healthcare SystemBrocktonUSA
  2. 2.Department of PsychiatryBoston University School of MedicineBostonUSA
  3. 3.Department of PsychologyUniversity of British ColumbiaVancouverCanada
  4. 4.Department of PsychologyThe Ohio State UniversityColumbusUSA

Personalised recommendations