Advertisement

Cognitive, Affective, & Behavioral Neuroscience

, Volume 19, Issue 6, pp 1458–1466 | Cite as

The importance of agency in human reward processing

  • Cameron D. Hassall
  • Greg Hajcak
  • Olave E. KrigolsonEmail author
Article

Abstract

Converging evidence suggests that reinforcement learning (RL) signals exist within the human brain and that they play a role in the modification of behaviour. According to RL theory, prediction errors are used to update values associated with actions and/or predictive cues, thus facilitate decision-making. For example, the reward positivity—a feedback-sensitive component of the event-related brain potential (ERP)—is thought to index an RL prediction error. An unresolved question, however, is whether or not action is required to elicit a reward positivity. Reinforcement learning theory would predict that the reward positivity should diminish or disappear in the absence of action, but evidence for this claim is conflicting. To investigate the impact of cue, choice, and action on the amplitude of the reward positivity, we altered a two-armed bandit task by systematically removing these factors. The reward positivity was greatly reduced or absent in the altered versions of the task. This result highlights the key role of agency in producing learning signals, such as the reward positivity.

Keywords

Agency Reward positivity Reinforcement learning 

Notes

Acknowledgements

This research was supported by the University of Victoria Neuroeducation Network (first author) and the Natural Sciences and Engineering Research Council of Canada (third author).

References

  1. Begleiter, H., Porjesz, B., Chou, C. L., & Aunon, J. I. (1983). P3 and Stimulus Incentive Value. Psychophysiology, 20(1), 95–101. Google Scholar
  2. Bellebaum, C., Kobza, S., Thiele, S., & Daum, I. (2010). It Was Not MY Fault: Event-Related Brain Potentials in Active and Observational Learning from Feedback. Cerebral Cortex, 20(12), 2874–2883.  https://doi.org/10.1093/cercor/bhq038 CrossRefPubMedGoogle Scholar
  3. Brainard, D. H. (1997). The psychophysics toolbox. Spatial Vision, 10, 433–436.CrossRefGoogle Scholar
  4. Bress, J. N., Foti, D., Kotov, R., Klein, D. N., & Hajcak, G. (2013). Blunted neural response to rewards prospectively predicts depression in adolescent girls. Psychophysiology, 50(1), 74–81.CrossRefGoogle Scholar
  5. Carrillo-de-la-Peña, M. T., & Cadaveira, F. (2000). The effect of motivational instructions on P300 amplitude. Neurophysiologie Clinique/Clinical Neurophysiology, 30(4), 232–239.CrossRefGoogle Scholar
  6. Collins, A. G. E., & Frank, M. J. (2018). Within- and across-trial dynamics of human EEG reveal cooperative interplay between reinforcement learning and working memory. Proceedings of the National Academy of Sciences, 115(10), 2502–2507.CrossRefGoogle Scholar
  7. Cumming, G. (2014). The New Statistics: Why and How. Psychological Science, 25(1), 7–29.  https://doi.org/10.1177/0956797613504966 CrossRefPubMedGoogle Scholar
  8. Daw, N. D., Niv, Y., & Dayan, P. (2005). Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control. Nature Neuroscience, 8(12), 1704–1711.  https://doi.org/10.1038/nn1560 CrossRefPubMedGoogle Scholar
  9. Delorme, A., & Makeig, S. (2004). EEGLAB: an open source toolbox for analysis of single-trial EEG dynamics including independent component analysis. Journal of Neuroscience Methods, 134(1), 9–21.  https://doi.org/10.1016/J.JNEUMETH.2003.10.009 CrossRefPubMedGoogle Scholar
  10. Donkers, F. C. L., Nieuwenhuis, S., & van Boxtel, G. J. M. (2005). Mediofrontal negativities in the absence of responding. Cognitive Brain Research, 25(3), 777–787.  https://doi.org/10.1016/j.cogbrainres.2005.09.007 CrossRefPubMedGoogle Scholar
  11. Dunning, J. P., & Hajcak, G. (2007). Error-related negativities elicited by monetary loss and cues that predict loss. Neuroreport, 18(17), 1875–1878.  https://doi.org/10.1097/WNR.0b013e3282f0d50b CrossRefPubMedGoogle Scholar
  12. Fischer, A. G., & Ullsperger, M. (2013). Real and Fictive Outcomes Are Processed Differently but Converge on a Common Adaptive Mechanism. Neuron, 79(6), 1243–1255.CrossRefGoogle Scholar
  13. Goldstein, R. Z., Cottone, L. A., Jia, Z., Maloney, T., Volkow, N. D., & Squires, N. K. (2006). The effect of graded monetary reward on cognitive event-related potentials and behavior in young healthy adults. International Journal of Psychophysiology, 62(2), 272–279.CrossRefGoogle Scholar
  14. Haggard, P. (2017). Sense of agency in the human brain. Nature Reviews Neuroscience, 18(4), 196–207.  https://doi.org/10.1038/nrn.2017.14 CrossRefPubMedGoogle Scholar
  15. Holroyd, C. B., & Coles, M. G. H. (2002). The neural basis of human error processing: Reinforcement learning, dopamine, and the error-related negativity. Psychological Review, 109(4), 679–709.  https://doi.org/10.1037//0033-295X.109.4.679 CrossRefPubMedGoogle Scholar
  16. Holroyd, C. B., Krigolson, O. E., & Lee, S. (2011). Reward positivity elicited by predictive cues. Neuroreport, 22(5), 249–252.  https://doi.org/10.1097/WNR.0b013e328345441d CrossRefPubMedGoogle Scholar
  17. Johansson, P., Hall, L., Sikström, S., & Olsson, A. (2005). Failure to Detect Mismatches Between Intention and Outcome in a Simple Decision Task. Science, 310(5745), 116–119.  https://doi.org/10.1126/science.1111709 CrossRefPubMedGoogle Scholar
  18. Jung, T.-P., Makeig, S., Humphries, C., Lee, T.-W., McKeown, M. J., Iragui, V., & Sejnowski, T. J. (2000). Removing electroencephalographic artifacts by blind source separation. Psychophysiology, 37(2), 163–178.Google Scholar
  19. Kleih, S. C., Nijboer, F., Halder, S., & Kübler, A. (2010). Motivation modulates the P300 amplitude during brain–computer interface use. Clinical Neurophysiology, 121(7), 1023–1031.Google Scholar
  20. Krigolson, O. E. (2017). Event-related brain potentials and the study of reward processing: Methodological considerations. International Journal of Psychophysiology.  https://doi.org/10.1016/j.ijpsycho.2017.11.007 CrossRefGoogle Scholar
  21. Krigolson, O. E., Hassall, C. D., & Handy, T. C. (2013). How We Learn to Make Decisions: Rapid Propagation of Reinforcement Learning Prediction Errors in Humans. Journal of Cognitive Neuroscience, 26(3), 635–644.  https://doi.org/10.1162/jocn_a_00509 CrossRefPubMedGoogle Scholar
  22. Krigolson, O. E., & Holroyd, C. B. (2007). Predictive information and error processing: The role of medial-frontal cortex during motor control. Psychophysiology, 44(4), 586–595.  https://doi.org/10.1111/j.1469-8986.2007.00523.x CrossRefPubMedGoogle Scholar
  23. Lakens, D. (2013). Calculating and reporting effect sizes to facilitate cumulative science: a practical primer for t-tests and ANOVAs. Frontiers in Psychology, 4.  https://doi.org/10.3389/fpsyg.2013.00863
  24. Langer, E. J. (1975). The illusion of control. Journal of Personality and Social Psychology, 32(2), 311–328.CrossRefGoogle Scholar
  25. Loftus, G. R., & Masson, M. E. (1994). Using confidence intervals in within-subject designs. Psychonomic Bulletin & Review, 1(4), 476–490.CrossRefGoogle Scholar
  26. Martin, L. E., & Potts, G. F. (2011). Medial Frontal Event Related Potentials and Reward Prediction: Do Responses Matter? Brain and Cognition, 77(1), 128–134.  https://doi.org/10.1016/j.bandc.2011.04.001 CrossRefPubMedPubMedCentralGoogle Scholar
  27. Masson, M. E., & Loftus, G. R. (2003). Using confidence intervals for graphically based data interpretation. Canadian Journal of Experimental Psychology/Revue Canadienne de Psychologie Expérimentale, 57(3), 203.CrossRefGoogle Scholar
  28. Meadows, C. C., Gable, P. A., Lohse, K. R., & Miller, M. W. (2016). The effects of reward magnitude on reward processing: An averaged and single trial event-related potential study. Biological Psychology, 118, 154–160.CrossRefGoogle Scholar
  29. Miltner, W. H., Braun, C. H., & Coles, M. G. (1997). Event-related brain potentials following incorrect feedback in a time-estimation task: Evidence for a “generic” neural system for error detection. Journal of Cognitive Neuroscience, 9(6), 788–798.Google Scholar
  30. Mühlberger, C., Angus, D. J., Jonas, E., Harmon-Jones, C., & Harmon-Jones, E. (2017). Perceived control increases the reward positivity and stimulus preceding negativity. Psychophysiology, 54(2), 310–322.  https://doi.org/10.1111/psyp.12786 CrossRefPubMedGoogle Scholar
  31. Mulligan, E. M., Flynn, H., & Hajcak, G. (2018). Neural response to reward and psychosocial risk factors independently predict antenatal depressive symptoms. Biological Psychology. In press, corrected proof.Google Scholar
  32. O’Doherty, J. P., Cockburn, J., & Pauli, W. M. (2017). Learning, Reward, and Decision Making. Annual Review of Psychology, 68(1), 73–100.  https://doi.org/10.1146/annurev-psych-010416-044216 CrossRefPubMedGoogle Scholar
  33. Oberg, S. A. K., Christie, G. J., & Tata, M. S. (2011). Problem gamblers exhibit reward hypersensitivity in medial frontal cortex during gambling. Neuropsychologia, 49(13), 3768–3775.  https://doi.org/10.1016/j.neuropsychologia.2011.09.037 CrossRefPubMedGoogle Scholar
  34. Olejnik, S., & Algina, J. (2003). Generalized eta and omega squared statistics: measures of effect size for some common research designs. Psychological Methods, 8(4), 434–447.  https://doi.org/10.1037/1082-989X.8.4.434 CrossRefPubMedGoogle Scholar
  35. Pavlov, P. I. (2010). Conditioned reflexes: An investigation of the physiological activity of the cerebral cortex. Annals of Neurosciences, 17(3), 136–141. (Original work published in 1927)  https://doi.org/10.5214/ans.0972-7531.1017309 CrossRefPubMedGoogle Scholar
  36. Pelli, D. G. (1997). The VideoToolbox software for visual psychophysics: transforming numbers into movies. Spatial Vision, 10(4), 437–442.  https://doi.org/10.1163/156856897X00366 CrossRefPubMedGoogle Scholar
  37. Polich, J. (2007). Updating P300: An integrative theory of P3a and P3b. Clinical Neurophysiology, 118(10), 2128–2148.  https://doi.org/10.1016/j.clinph.2007.04.019 CrossRefPubMedPubMedCentralGoogle Scholar
  38. Proudfit, G. H. (2015). The reward positivity: From basic research on reward to a biomarker for depression. Psychophysiology, 52(4), 449–459.CrossRefGoogle Scholar
  39. Rescorla, R. A., & Wagner, A. R. (1972). A theory of Pavlovian conditioning: Variations in the effectiveness of reinforcement and nonreinforcement. Classical Conditioning II: Current Research and Theory, 2, 64–99.Google Scholar
  40. Sambrook, T. D., & Goslin, J. (2015). A neural reward prediction error revealed by a meta-analysis of ERPs using great grand averages. Psychological Bulletin, 141(1), 213–235.  https://doi.org/10.1037/bul0000006 CrossRefPubMedGoogle Scholar
  41. Schmitt, H., Ferdinand, N. K., & Kray, J. (2015). The influence of monetary incentives on context processing in younger and older adults: an event-related potential study. Cognitive, Affective, & Behavioral Neuroscience, 15(2), 416–434.CrossRefGoogle Scholar
  42. Sutton, R. S., & Barto, A. G. (1998). Reinforcement Learning: An Introduction. MIT Press, Cambridge.Google Scholar
  43. Thorndike, E. (2017). Animal Intelligence: Experimental Studies. New York: Routledge. (Original work published in 1911)CrossRefGoogle Scholar
  44. Walsh, M. M., & Anderson, J. R. (2012). Learning from experience: Event-related potential correlates of reward processing, neural adaptation, and behavioral choice. Neuroscience & Biobehavioral Reviews, 36(8), 1870–1884.  https://doi.org/10.1016/j.neubiorev.2012.05.008 CrossRefGoogle Scholar
  45. Yeung, N., Botvinick, M. M., & Cohen, J. D. (2004). The Neural Basis of Error Detection: Conflict Monitoring and the Error-Related Negativity. Psychological Review, 111(4), 931–959.  https://doi.org/10.1037/0033-295X.111.4.931 CrossRefPubMedGoogle Scholar
  46. Yeung, N., Holroyd, C. B., & Cohen, J. D. (2005). ERP Correlates of Feedback and Reward Processing in the Presence and Absence of Response Choice. Cerebral Cortex, 15(5), 535–544.  https://doi.org/10.1093/cercor/bhh153 CrossRefPubMedGoogle Scholar
  47. Yeung, N., & Sanfey, A. G. (2004). Independent Coding of Reward Magnitude and Valence in the Human Brain. Journal of Neuroscience, 24(28), 6258–6264.  https://doi.org/10.1523/JNEUROSCI.4537-03.2004 CrossRefPubMedPubMedCentralGoogle Scholar

Copyright information

© The Psychonomic Society, Inc. 2019

Authors and Affiliations

  1. 1.Centre for Biomedical ResearchUniversity of VictoriaVictoriaCanada
  2. 2.Department of Psychology and Biomedical SciencesFlorida State UniversityTallahasseeUSA

Personalised recommendations