Cognitive, Affective, & Behavioral Neuroscience

, Volume 14, Issue 4, pp 1167–1183 | Cite as

Physiological and behavioral signatures of reflective exploratory choice

  • A. Ross Otto
  • W. Bradley Knox
  • Arthur B. Markman
  • Bradley C. Love
Article

Abstract

Physiological arousal, a marker of emotional response, has been demonstrated to accompany human decision making under uncertainty. Anticipatory emotions have been portrayed as basic and rapid evaluations of chosen actions. Instead, could these arousal signals stem from a “cognitive” assessment of value that utilizes the full environment structure, as opposed to merely signaling a coarse, reflexive assessment of the possible consequences of choices? Combining an exploration–exploitation task, computational modeling, and skin conductance measurements, we find that physiological arousal manifests a reflective assessment of the benefit of the chosen action, mirroring observed behavior. Consistent with the level of computational sophistication evident in these signals, a follow-up experiment demonstrates that anticipatory arousal is modulated by current environment volatility, in accordance with the predictions of our computational account. Finally, we examine the cognitive costs of the exploratory choice behavior these arousal signals accompany by manipulating concurrent cognitive demand. Taken together, these results demonstrate that the arousal that accompanies choice under uncertainty arises from a more reflective and “cognitive” assessment of the chosen action’s consequences than has been revealed previously.

Keywords

Decision-making Reward Reinforcement learning Emotion Arousal 

References

  1. Badre, D., Doll, B. B., Long, N. M., & Frank, M. J. (2012). Rostrolateral prefrontal cortex and individual differences in uncertainty-driven exploration. Neuron, 73(3), 595–607.PubMedCentralPubMedCrossRefGoogle Scholar
  2. Bechara, A., Damasio, H., Damasio, A. R., & Lee, G. P. (1999). Different contributions of the human amygdala and ventromedial prefrontal cortex to decision-making. Journal of Neuroscience, 19(13), 5473–5481.PubMedGoogle Scholar
  3. Bechara, A., Tranel, D., Damasio, H., & Damasio, A. R. (1996). Failure to respond autonomically to anticipated future outcomes following damage to prefrontal cortex. Cerebral Cortex, 6(2), 215–225.PubMedCrossRefGoogle Scholar
  4. Benedek, M., & Kaernbach, C. (2010). A continuous measure of phasic electrodermal activity. Journal of Neuroscience Methods, 190(1), 80–91.PubMedCentralPubMedCrossRefGoogle Scholar
  5. Blanco, N. J., Otto, A. R., Maddox, W. T., Beevers, C. G., & Love, B. C. (2013). The influence of depression symptoms on exploratory decision-making. Cognition, 129(3), 563–568.PubMedCrossRefGoogle Scholar
  6. Botvinick, M. M., & Rosen, Z. B. (2009). Anticipation of cognitive demand during decision-making. Psychological Research Psychologische Forschung, 73(6), 835–842.CrossRefGoogle Scholar
  7. Busemeyer, J. R., & Stout, J. C. (2002). A contribution of cognitive decision models to clinical assessment: Decomposing performance on the Bechara gambling task. Psychological Assessment, 14(3), 253–262.PubMedCrossRefGoogle Scholar
  8. Camille, N., Coricelli, G., Sallet, J., Pradat-Diehl, P., Duhamel, J.-R., & Sirigu, A. (2004). The involvement of the orbitofrontal cortex in the experience of regret. Science, 304(5674), 1167–1170.PubMedCrossRefGoogle Scholar
  9. Cassandra, A., Littman, M. L., & Zhang, N. L. (1997). Incremental pruning: A simple, fast, exact method for partially observable Markov decision processes. In Proceedings of the Thirteenth Conference on Uncertainty in Artificial Intelligence, 54–61.Google Scholar
  10. Cohen, J. D., McClure, S. M., & Yu, A. J. (2007). Should I stay or should I go? How the human brain manages the trade-off between exploitation and exploration. Philosophical Transactions of the Royal Society, B: Biological Sciences, 362(1481), 933–942.PubMedCentralCrossRefGoogle Scholar
  11. Critchley, H. D. (2005). Neural mechanisms of autonomic, affective, and cognitive integration. The Journal of Comparative Neurology, 493(1), 154–166.PubMedCrossRefGoogle Scholar
  12. Critchley, H. D., Elliott, R., Mathias, C. J., & Dolan, R. J. (2000). Neural activity relating to generation and representation of galvanic skin conductance responses: A functional magnetic resonance imaging study. Journal of Neuroscience, 20(8), 3033–3040.PubMedGoogle Scholar
  13. Critchley, H. D., Mathias, C. J., & Dolan, R. J. (2001). Neural activity in the human brain relating to uncertainty and arousal during anticipation. Neuron, 29(2), 537–545.PubMedCrossRefGoogle Scholar
  14. Critchley, H. D., Tang, J., Glaser, D., Butterworth, B., & Dolan, R. J. (2005). Anterior cingulate activity during error and autonomic response. NeuroImage, 27(4), 885–895.PubMedCrossRefGoogle Scholar
  15. Damasio, A. (1994). Descartes’ error: Emotion, reason, and the human brain. New York: Putnam.Google Scholar
  16. Daw, N. D., Gershman, S. J., Seymour, B., Dayan, P., & Dolan, R. J. (2011). Model-based influences on humans’ choices and striatal prediction errors. Neuron, 69(6), 1204–1215.PubMedCentralPubMedCrossRefGoogle Scholar
  17. Daw, N. D., Niv, Y., & Dayan, P. (2005). Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control. Nature Neuroscience, 8(12), 1704–1711.PubMedCrossRefGoogle Scholar
  18. Daw, N. D., O’Doherty, J. P., Dayan, P., Seymour, B., & Dolan, R. J. (2006). Cortical substrates for exploratory decisions in humans. Nature, 441, 876–879.PubMedCentralPubMedCrossRefGoogle Scholar
  19. Dolan, R. J. (2002). Emotion, cognition, and behavior. Science, 298(5596), 1191–1194.PubMedCrossRefGoogle Scholar
  20. Dunn, B. D., Dalgleish, T., & Lawrence, A. D. (2006). The somatic marker hypothesis: A critical evaluation. Neuroscience & Biobehavioral Reviews, 30(2), 239–271.CrossRefGoogle Scholar
  21. Fellows, L. K. (2007). Advances in understanding ventromedial prefrontal function the accountant joins the executive. Neurology, 68(13), 991–995.PubMedCrossRefGoogle Scholar
  22. Fellows, L. K., & Farah, M. J. (2005). Different underlying impairments in decision-making following ventromedial and dorsolateral frontal lobe damage in humans. Cerebral Cortex, 15(1), 58–63.PubMedCrossRefGoogle Scholar
  23. Figner, B., Mackinlay, R. J., Wilkening, F., & Weber, E. U. (2009). Affective and deliberative processes in risky choice: Age differences in risk taking in the Columbia Card Task. Journal of Experimental Psychology Learning, Memory, and Cognition, 35(3), 709–730.PubMedCrossRefGoogle Scholar
  24. Foerde, K., Knowlton, B. J., & Poldrack, R. A. (2006). Modulation of competing memory systems by distraction. Proceedings of the National Academy of Sciences, 103(31), 11778–11783.CrossRefGoogle Scholar
  25. Gershman, S. J., Markman, A. B., & Otto, A. R. (2014). Retrospective revaluation in sequential decision making: A tale of two systems. Journal of Experimental Psychology: General, 143(1), 182–194.Google Scholar
  26. Gilzenrat, M., Nieuwenhuis, S., Jepma, M., & Cohen, J. (2010). Pupil diameter tracks changes in control state predicted by the adaptive gain theory of locus coeruleus function. Cognitive, Affective, & Behavioral Neuroscience, 10(2), 252–269.CrossRefGoogle Scholar
  27. Gläscher, J., Daw, N., Dayan, P., & O’Doherty, J. P. (2010). States versus rewards: Dissociable neural prediction error signals underlying model-based and model-free reinforcement learning. Neuron, 66(4), 585–595.PubMedCentralPubMedCrossRefGoogle Scholar
  28. Hajcak, G., McDonald, N., & Simons, R. F. (2003). To err is autonomic: Error-related brain potentials, ANS activity, and post-error compensatory behavior. Psychophysiology, 40(6), 895–903.PubMedCrossRefGoogle Scholar
  29. Hampton, A. N., Bossaerts, P., & O’Doherty, J. P. (2006). The role of the ventromedial prefrontal cortex in abstract state-based inference during decision making in humans. Journal of Neuroscience, 26(32), 8360–8367.PubMedCrossRefGoogle Scholar
  30. Hare, T. A., Camerer, C. F., & Rangel, A. (2009). Self-control in decision-making involves modulation of the vmPFC valuation system. Science, 324(5927), 646–648.PubMedCrossRefGoogle Scholar
  31. Humphries, M. D., Khamassi, M., & Gurney, K. (2012). Dopaminergic control of the exploration-exploitation trade-off via the basal ganglia. Frontiers in Decision Neuroscience, 6, 9.Google Scholar
  32. Jepma, M., & Nieuwenhuis, S. (2011). Pupil diameter predicts changes in the exploration–exploitation trade-off: Evidence for the adaptive gain theory. Journal of Cognitive Neuroscience, 23(7), 1587–1596.PubMedCrossRefGoogle Scholar
  33. Kaelbling, L. P., Littman, M. L., & Cassandra, A. R. (1998). Planning and acting in partially observable stochastic domains. Artificial Intelligence, 101(1–2), 99–134.CrossRefGoogle Scholar
  34. Keramati, M., Dezfouli, A., & Piray, P. (2011). Speed/accuracy trade-off between the habitual and the goal-directed processes. PLoS Computational Biology, 7(5), e1002055.PubMedCentralPubMedCrossRefGoogle Scholar
  35. Knox, W. B., Otto, A. R., Stone, P. H., & Love, B. C. (2012). The nature of belief-directed exploratory choice by human decision-makers. Frontiers in Psychology, 2, 398.PubMedCentralPubMedCrossRefGoogle Scholar
  36. Lang, P. J., Greenwald, M. K., Bradley, M. M., & Hamm, A. O. (1993). Looking at pictures: Affective, facial, visceral, and behavioral reactions. Psychophysiology, 30(3), 261–273.PubMedCrossRefGoogle Scholar
  37. Ledoux, J. (1996). The emotional brain: The mysterious underpinnings of emotional life. New York: Simon and Schuster.Google Scholar
  38. Loewenstein, G. F., Weber, E. U., Hsee, C. K., & Welch, N. (2001). Risk as feelings. Psychological Bulletin, 127(2), 267–286.PubMedCrossRefGoogle Scholar
  39. Lovibond, P. F. (2003). Causal beliefs and conditioned responses: Retrospective revaluation induced by experience and by instruction. Journal of Experimental Psychology: Learning, Memory, and Cognition, 29(1), 97–106.PubMedGoogle Scholar
  40. Martin, L. N., & Delgado, M. R. (2011). The influence of emotion regulation on decision-making under risk. Journal of Cognitive Neuroscience, 23(9), 2569–2581.PubMedCentralPubMedCrossRefGoogle Scholar
  41. Mellers, B. A., Schwartz, A., Ho, K., & Ritov, I. (1997). Decision affect theory: Emotional reactions to the outcomes of risky options. Psychological Science, 8(6), 423–429.CrossRefGoogle Scholar
  42. Mitchell, D. G. V. (2011). The nexus between decision making and emotion regulation: A review of convergent neurocognitive substrates. Behavioural Brain Research, 217(1), 215–231.PubMedCrossRefGoogle Scholar
  43. Nagai, Y., Critchley, H., Featherstone, E., Trimble, M., & Dolan, R. (2004). Activity in ventromedial prefrontal cortex covaries with sympathetic skin conductance level: A physiological account of a “default mode” of brain function. NeuroImage, 22(1), 243–251.PubMedCrossRefGoogle Scholar
  44. Nassar, M. R., Rumsey, K. M., Wilson, R. C., Parikh, K., Heasly, B., & Gold, J. I. (2012). Rational regulation of learning dynamics by pupil-linked arousal systems. Nature Neuroscience, 15(7), 1040–1046.PubMedCentralPubMedCrossRefGoogle Scholar
  45. Nicolle, A., Klein-Flügge, M. C., Hunt, L. T., Vlaev, I., Dolan, R. J., & Behrens, T. E. J. (2012). An agent independent axis for executed and modeled choice in medial prefrontal cortex. Neuron, 75(6), 1114–1121.PubMedCentralPubMedCrossRefGoogle Scholar
  46. Öhman, A., & Soares, J. J. F. (1994). “Unconscious anxiety”: Phobic responses to masked stimuli. Journal of Abnormal Psychology, 103(2), 231–240.PubMedCrossRefGoogle Scholar
  47. Olsson, A., & Phelps, E. A. (2004). Learned fear of “unseen” faces after pavlovian, observational, and instructed fear. Psychological Science, 15(12), 822–828.PubMedCrossRefGoogle Scholar
  48. Otto, A. R., Gershman, S. J., Markman, A. B., & Daw, N. D. (2013a). The curse of planning dissecting multiple reinforcement-learning systems by taxing the central executive. Psychological Science, 24(5), 751–761.PubMedCrossRefGoogle Scholar
  49. Otto, A. R., Markman, A. B., Gureckis, T. M., & Love, B. C. (2010). Regulatory fit and systematic exploration in a dynamic decision-making environment. Journal of Experimental Psychology: Learning, Memory, and Cognition, 36(3), 797–804.PubMedGoogle Scholar
  50. Otto, A. R., Raio, C. M., Chiang, A., Phelps, E. A., & Daw, N. D. (2013b). Working-memory capacity protects model-based learning from stress. Proceedings of the National Academy of Sciences, 110(52), 20941–20946.Google Scholar
  51. Otto, A. R., Taylor, E. G., & Markman, A. B. (2011). There are at least two kinds of probability matching: Evidence from a secondary task. Cognition, 118(2), 274–279.PubMedCrossRefGoogle Scholar
  52. Preuschoff, K., Hart, B., & Einhäuser, W. (2011). Pupil dilation signals surprise: Evidence for noradrenaline’s role in decision making. Frontiers in Decision Neuroscience, 5, 115.Google Scholar
  53. Rolls, E. T. (1999). The brain and emotion. Oxford: Oxford University Press.Google Scholar
  54. Rushworth, M. F. S., Noonan, M. P., Boorman, E. D., Walton, M. E., & Behrens, T. E. (2011). Frontal cortex and reward-guided learning and decision-making. Neuron, 70(6), 1054–1069.PubMedCrossRefGoogle Scholar
  55. Schonberg, T., Fox, C. R., & Poldrack, R. A. (2011). Mind the gap: Bridging economic and naturalistic risk-taking with cognitive neuroscience. Trends in Cognitive Sciences, 15(1), 11–19.PubMedCentralPubMedCrossRefGoogle Scholar
  56. Schwarz, G. (1978). Estimating the dimension of a model. The Annals of Statistics, 6(2), 461–464.CrossRefGoogle Scholar
  57. Sokol-Hessner, P., Hsu, M., Curley, N. G., Delgado, M. R., Camerer, C. F., & Phelps, E. A. (2009). Thinking like a trader selectively reduces individuals’ loss aversion. Proceedings of the National Academy of Sciences, 106(13), 5035–5040.CrossRefGoogle Scholar
  58. Studer, B., & Clark, L. (2011). Place your bets: Psychophysiological correlates of decision-making under risk. Cognitive, Affective, & Behavioral Neuroscience, 11(2), 144–158.CrossRefGoogle Scholar
  59. Sutton, R. S., & Barto, A. G. (1998). Reinforcement Learning. Cambridge, MA: MIT Press.Google Scholar
  60. Suzuki, A., Hirota, A., Takasawa, N., & Shigemasu, K. (2003). Application of the somatic marker hypothesis to individual differences in decision making. Biological Psychology, 65(1), 81–88.PubMedCrossRefGoogle Scholar
  61. Tomb, I., Hauser, M., Deldin, P., & Caramazza, A. (2002). Do somatic markers mediate decisions on the gambling task? Nature Neuroscience, 5(11), 1103–1104.PubMedCrossRefGoogle Scholar
  62. Tversky, A., & Kahneman, D. (1992). Advances in prospect theory: Cumulative representation of uncertainty. Journal of Risk and Uncertainty, 5(4), 297–323.CrossRefGoogle Scholar
  63. Whitney, P., Hinson, J. M., Wirick, A., & Holben, H. (2007). Somatic responses in behavioral inhibition. Cognitive, Affective, & Behavioral Neuroscience, 7(1), 37–43.CrossRefGoogle Scholar
  64. Worthy, D. A., Maddox, W. T., & Markman, A. B. (2007). Regulatory fit effects in a choice task. Psychonomic Bulletin & Review, 14(6), 1125–1132.CrossRefGoogle Scholar
  65. Worthy, D. A., Hawthorne, M. J., & Otto, A. R. (2013). Heterogeneity of strategy use in the Iowa gambling task: A comparison of win-stay/lose-shift and reinforcement learning models. Psychonomic Bulletin & Review, 20(2), 364–371.Google Scholar
  66. Wunderlich, K., Dayan, P., & Dolan, R. J. (2012). Mapping value based planning and extensively trained choice in the human brain. Nature Neuroscience. Retrieved from http://www.ncbi.nlm.nih.gov/pubmed/22406551
  67. Yechiam, E., & Busemeyer, J. R. (2005). Comparison of basic assumptions embedded in learning models for experience-based decision making. Psychonomic Bulletin & Review, 12(3), 387–402.CrossRefGoogle Scholar
  68. Zajonc, R. B. (1984). On the primacy of affect. American Psychologist, 39(2), 117–123.CrossRefGoogle Scholar
  69. Zeithamova, D., & Maddox, W. T. (2006). Dual-task interference in perceptual category learning. Memory & Cognition, 34, 387–398.CrossRefGoogle Scholar

Copyright information

© Psychonomic Society, Inc. 2014

Authors and Affiliations

  • A. Ross Otto
    • 1
  • W. Bradley Knox
    • 2
  • Arthur B. Markman
    • 3
  • Bradley C. Love
    • 4
  1. 1.Center for Neural ScienceNew York UniversityNew YorkUSA
  2. 2.Massachusetts Institute of TechnologyCambridgeUSA
  3. 3.University of Texas at AustinAustinUSA
  4. 4.University College LondonLondonUK

Personalised recommendations