Skip to main content
SpringerLink
Log in

Audiovisual Representations of Valence: a Cross-study Perspective

  • RESEARCH ARTICLE
  • Published:
Affective Science Aims and scope Submit manuscript

Abstract

Hedonic valence describes the pleasantness or unpleasantness of psychological states elicited by stimuli and is conceived as a fundamental building block of emotional experience. Multivariate pattern analysis approaches contribute to the study of valence representation by allowing identification of valence from distributed patterns of activity. However, the issue of construct validity arises in that there is always a possibility that classification results from a single study are driven by factors other than valence, such as the idiosyncrasies of the stimuli. In this work, we identify valence across participants from six different fMRI studies that used auditory, visual, or audiovisual stimuli, thus increasing the likelihood that classification is driven by valence and not by the specifics of the experimental paradigm of a particular study. The studies included a total of 93 participants and differed on stimuli, task, trial duration, number of participants, and scanner parameters. In a leave-one-study-out cross-validation procedure, we trained the classifiers on fMRI data from five studies and predicted valence, positive or negative, for each of the participants in the left-out study. Whole-brain classification demonstrated a reliable distinction between positive and negative valence states (72% accuracy). In a searchlight analysis, the representation of valence was localized to the right postcentral and supramarginal gyri, left superior frontal and middle frontal cortices, and right pregenual anterior cingulate and superior medial frontal cortices. The demonstrated cross-study classification of valence enhances the construct validity and generalizability of the findings from the combined studies.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

References

  • Abelson, R. P., & Sermat, V. (1962). Multidimensional scaling of facial expressions. Journal of Experimental Psychology, 63(6), 546–554.

    PubMed  Google Scholar 

  • Baker, M. (2016). Reproducibility crisis? Nature, 533(26), 353–366.

    Google Scholar 

  • Barrett, L. F. (2017). The theory of constructed emotion: An active inference account of interoception and categorization. Social Cognitive and Affective Neuroscience, 12(11), 1833.

    PubMed  PubMed Central  Google Scholar 

  • Barrett, L. F., & Bliss-Moreau, E. (2009). Affect as a psychological primitive. Advances in Experimental Social Psychology, 41, 167–218.

    PubMed  PubMed Central  Google Scholar 

  • Baucom, L. B., Wedell, D. H., Wang, J., Blitzer, D. N., & Shinkareva, S. V. (2012). Decoding the neural representation of affective states. Neuroimage, 59(1), 718–727.

    PubMed  Google Scholar 

  • Bennett, C. M., & Miller, M. B. (2013). fMRI reliability: Influences of task and experimental design. Cognitive, Affective, & Behavioral Neuroscience, 13(4), 690–702.

    Google Scholar 

  • Bigand, E., Filipic, S., & Lalitte, P. (2005). The time course of emotional responses to music. Annals of the New York Academy of Sciences, 1060(1), 429–437.

    PubMed  Google Scholar 

  • Bowring, A., Maumet, C., & Nichols, T. E. (2019). Exploring the impact of analysis software on task fMRI results. Human Brain Mapping, 40(11), 3362–3384.

    PubMed  PubMed Central  Google Scholar 

  • Bradley, M. M., & Lang, P. J. (1999). Affective norms for English words (ANEW): Instruction manual and affective ratings: Technical report C-1, the center for research in psychophysiology. University of Florida.

  • Bradley, M. M., & Lang, P. J. (2000). Affective reactions to acoustic stimuli. Psychophysiology, 37(2), 204–215.

    PubMed  Google Scholar 

  • Bradley, M. M., & Lang, P. J. (2007). The International Affective Digitized Sounds (; IADS-2): Affective ratings of sounds and instruction manual. University of Florida, Gainesville, FL, Tech. Rep. B-3.

  • Bush, L. E. (1973). Individual differences multidimensional scaling of adjectives denoting feelings. Journal of Personality and Social Psychology, 25(1), 50–57.

    PubMed  Google Scholar 

  • Bzdok, D., Langner, R., Schilbach, L., Engemann, D. A., Laird, A. R., Fox, P. T., & Eickhoff, S. (2013). Segregation of the human medial prefrontal cortex in social cognition. Frontiers in Human Neuroscience, 7, 232.

    PubMed  PubMed Central  Google Scholar 

  • Chikazoe, J., Lee, D. H., Kriegeskorte, N., & Anderson, A. K. (2014). Population coding of affect across stimuli, modalities and individuals. Nature Neuroscience, 17(8), 1114–1122.

    PubMed  PubMed Central  Google Scholar 

  • Costafreda, S. G., Khanna, A., Mourao-Miranda, J., & Fu, C. H. (2009). Neural correlates of sad faces predict clinical remission to cognitive behavioural therapy in depression. Neuroreport, 20(7), 637–641.

    PubMed  Google Scholar 

  • Damasio, A. R. (1996). The somatic marker hypothesis and the possible functions of the prefrontal cortex. Philosophical Transactions of the Royal Society of London. Series B: Biological Sciences, 351(1346), 1413–1420.

  • Dixon, M. L., Thiruchselvam, R., Todd, R., & Christoff, K. (2017). Emotion and the prefrontal cortex: An integrative review. Psychological Bulletin, 143(10), 1033–1081.

    PubMed  Google Scholar 

  • Egner, T., Etkin, A., Gale, S., & Hirsch, J. (2007). Dissociable neural systems resolve conflict from emotional versus nonemotional distracters. Cerebral Cortex, 18(6), 1475–1484.

    PubMed  Google Scholar 

  • Etkin, A., Egner, T., & Kalisch, R. (2011). Emotional processing in anterior cingulate and medial prefrontal cortex. Trends in Cognitive Sciences, 15(2), 85–93.

    PubMed  Google Scholar 

  • Etkin, A., Egner, T., Peraza, D. M., Kandel, E. R., & Hirsch, J. (2006). Resolving emotional conflict: A role for the rostral anterior cingulate cortex in modulating activity in the amygdala. Neuron, 51(6), 871–882.

    PubMed  Google Scholar 

  • Gabrieli, J. D., Ghosh, S. S., & Whitfield-Gabrieli, S. (2015). Prediction as a humanitarian and pragmatic contribution from human cognitive neuroscience. Neuron, 85(1), 11–26.

    PubMed  PubMed Central  Google Scholar 

  • Gao, C., Weber, C. E., & Shinkareva, S. V. (2019). The brain basis of audiovisual affective processing: Evidence from a coordinate-based activation likelihood estimation meta-analysis. Cortex., 120, 66–77.

    PubMed  Google Scholar 

  • Gao, C., Weber, C. E., Wedell, D. H., & Shinkareva, S. V. (2020). An fMRI study of affective congruence across visual and auditory modalities. Journal of Cognitive Neuroscience, 32(7), 1251–1262.

    PubMed  Google Scholar 

  • Gao, C., Wedell, D. H., Green, J. J., Jia, X., Mao, X., Guo, C., & Shinkareva, S. V. (2018). Temporal dynamics of audiovisual affective processing. Biological Psychology, 139, 59–72.

    PubMed  Google Scholar 

  • Gao, C., Wedell, D. H., Kim, J., Weber, C. E., & Shinkareva, S. V. (2018). Modelling audiovisual integration of affect from videos and music. Cognition and Emotion, 32(3), 516–529.

    PubMed  Google Scholar 

  • Habes, I., Krall, S. C., Johnston, S., Yuen, K., Healy, D., Goebel, R., et al. (2013). Pattern classification of valence in depression. NeuroImage: clinical, 2, 675–683.

    Google Scholar 

  • Hayes, W. M., & Wedell, D. H. (2020). Modeling the role of feelings in the Iowa Gambling Task. Decision, 7(1), 67–89.

    Google Scholar 

  • Just, M. A., Pan, L., Cherkassky, V. L., McMakin, D. L., Cha, C., Nock, M. K., & Brent, D. (2017). Machine learning of neural representations of suicide and emotion concepts identifies suicidal youth. Nature Human Behaviour, 1(12), 911–919.

    PubMed  PubMed Central  Google Scholar 

  • Kay, K., Rokem, A., Winawer, J., Dougherty, R., & Wandell, B. (2013). GLMdenoise: A fast, automated technique for denoising task-based fMRI data. Frontiers in Neuroscience, 7, 247.

    PubMed  PubMed Central  Google Scholar 

  • Kim, J., Shinkareva, S. V., & Wedell, D. H. (2017). Representations of modality-general valence for videos and music derived from fMRI data. NeuroImage, 148, 42–54.

    PubMed  Google Scholar 

  • Kim, J., Wang, J., Wedell, D. H., & Shinkareva, S. V. (2016). Identifying core affect in individuals from fMRI responses to dynamic naturalistic audiovisual stimuli. PLoS One, 11(9), e0161589.

    PubMed  PubMed Central  Google Scholar 

  • Kim, J., Weber, C. E., Gao, C., Schulteis, S., Wedell, D. H., & Shinkareva, S. V. (2020). A study in affect: Predicting valence from fMRI data. Neuropsychologia, 107473.

  • Klasen, M., Kenworthy, C. A., Mathiak, K. A., Kircher, T. T., & Mathiak, K. (2011). Supramodal representation of emotions. The Journal of Neuroscience, 31(38), 13635–13643. https://doi.org/10.1523/jneurosci.2833-11.2011.

    Article  PubMed  PubMed Central  Google Scholar 

  • Kragel, P. A., Kano, M., Van Oudenhove, L., Ly, H. G., Dupont, P., Rubio, A., et al. (2018). Generalizable representations of pain, cognitive control, and negative emotion in medial frontal cortex. Nature Neuroscience, 21(2), 283–289. https://doi.org/10.1038/s41593-017-0051-7.

    Article  PubMed  PubMed Central  Google Scholar 

  • Kragel, P. A., & LaBar, K. S. (2016). Decoding the nature of emotion in the brain. Trends in Cognitive Sciences, 20(6), 444–455.

    PubMed  PubMed Central  Google Scholar 

  • Kreifelts, B., Ethofer, T., Grodd, W., Erb, M., & Wildgruber, D. (2007). Audiovisual integration of emotional signals in voice and face: An event-related fMRI study. Neuroimage, 37(4), 1445–1456.

    PubMed  Google Scholar 

  • Kriegeskorte, N., Goebel, R., & Bandettini, P. (2006). Information-based functional brain mapping. Proceedings of the National Academy of Sciences, 103(10), 3863–3868.

    Google Scholar 

  • Kulkarni, B., Bentley, D. E., Elliott, R., Youell, P., & Jones, A. K. P. (2005). Attention to pain localization and unpleasantness discriminates the functions of the medial and lateral pain systems. European Journal of Neuroscience, 21(11), 3133–3142.

    PubMed  Google Scholar 

  • Lang, P. J. (1995). The emotion probe: Studies of motivation and attention. American Psychologist, 50(5), 372–385.

    PubMed  Google Scholar 

  • Lang, P. J. (2005). International Affective Picture System (IAPS): Affective ratings of pictures and instruction manual. Technical Report.

  • Lepping, R. J., Atchley, R. A., Chrysikou, E., Martin, L. E., Clair, A. A., Ingram, R. E., Simmons, W. K., & Savage, C. R. (2016). Neural processing of emotional musical and nonmusical stimuli in depression. PLoS One, 11(6), e0156859.

    PubMed  PubMed Central  Google Scholar 

  • Li, B. J., Bailenson, J. N., Pines, A., Greenleaf, W. J., & Williams, L. M. (2017). A public database of immersive VR videos with corresponding ratings of arousal, valence, and correlations between head movements and self report measures. Frontiers in Psychology, 8, 2116.

    PubMed  PubMed Central  Google Scholar 

  • Lindquist, K. A., Satpute, A. B., Wager, T. D., Weber, J., & Barrett, L. F. (2015). The brain basis of positive and negative affect: Evidence from a meta-analysis of the human neuroimaging literature. Cerebral Cortex, 26(5), 1910–1922.

    PubMed  Google Scholar 

  • Marusak, H., Thomason, M., Peters, C., Zundel, C., Elrahal, F., & Rabinak, C. (2016). You say ‘prefrontal cortex’and I say ‘anterior cingulate’: Meta-analysis of spatial overlap in amygdala-to-prefrontal connectivity and internalizing symptomology. Translational Psychiatry, 6(11), e944–e944.

    PubMed  PubMed Central  Google Scholar 

  • Miskovic, V., & Anderson, A. (2018). Modality general and modality specific coding of hedonic valence. Current Opinion in Behavioral Sciences, 19, 91–97.

    PubMed  PubMed Central  Google Scholar 

  • Munafò, M. R., Nosek, B. A., Bishop, D. V., Button, K. S., Chambers, C. D., Du Sert, N. P., et al. (2017). A manifesto for reproducible science. Nature Human Behaviour, 1(1), 1–9.

    Google Scholar 

  • Niedenthal, P. M. (2007). Embodying emotion. science, 316(5827), 1002–1005.

    PubMed  Google Scholar 

  • Olofsson, J. K., Nordin, S., Sequeira, H., & Polich, J. (2008). Affective picture processing: An integrative review of ERP findings. Biological Psychology, 77(3), 247–265.

    PubMed  Google Scholar 

  • Ontivero-Ortega, M., Lage-Castellanos, A., Valente, G., Goebel, R., & Valdes-Sosa, M. (2017). Fast Gaussian Naïve Bayes for searchlight classification analysis. Neuroimage, 163, 471–479.

    PubMed  Google Scholar 

  • Peelen, M. V., Atkinson, A. P., & Vuilleumier, P. (2010). Supramodal representations of perceived emotions in the human brain. Journal of Neuroscience, 30(30), 10127–10134.

    PubMed  Google Scholar 

  • Pereira, F., & Botvinick, M. (2011). Information mapping with pattern classifiers: A comparative study. Neuroimage, 56(2), 476–496.

    PubMed  Google Scholar 

  • Pereira, F., Mitchell, T., & Botvinick, M. (2009). Machine learning classifiers and fMRI: A tutorial overview. Neuroimage, 45(1), S199–S209.

    PubMed  Google Scholar 

  • Poldrack, R. A., Baker, C. I., Durnez, J., Gorgolewski, K. J., Matthews, P. M., Munafò, M. R., Nichols, T. E., Poline, J. B., Vul, E., & Yarkoni, T. (2017). Scanning the horizon: Towards transparent and reproducible neuroimaging research. Nature Reviews Neuroscience, 18(2), 115–126.

    PubMed  PubMed Central  Google Scholar 

  • Poldrack, R. A., Halchenko, Y. O., & Hanson, S. J. (2009). Decoding the large-scale structure of brain function by classifying mental states across individuals. Psychological Science, 20(11), 1364–1372.

    PubMed  PubMed Central  Google Scholar 

  • Poldrack, R. A., Huckins, G., & Varoquaux, G. (2020). Establishment of best practices for evidence for prediction: A review. JAMA Psychiatry, 77(5), 534–540.

    PubMed  PubMed Central  Google Scholar 

  • Raizada, R. D., & Lee, Y.-S. (2013). Smoothness without smoothing: Why Gaussian naive Bayes is not naive for multi-subject searchlight studies. PLoS One, 8(7), e69566.

    PubMed  PubMed Central  Google Scholar 

  • Roberts, J. S., & Wedell, D. H. (1994). Context effects on similarity judgments of multidimensional stimuli: Inferring the structure of the emotion space. Journal of Experimental Social Psychology, 30, 1–38.

    Google Scholar 

  • Russell, J. A. (2003). Core affect and the psychological construction of emotion. Psychological Review, 110(1), 145–172.

    PubMed  Google Scholar 

  • Russell, J. A., & Bullock, M. (1985). Multidimensional scaling of emotional facial expressions: Similarity from preschoolers to adults. Journal of Personality and Social Psychology, 48(5), 1290–1298.

    Google Scholar 

  • Satpute, A. B., & Lindquist, K. A. (2019). The default mode network’s role in discrete emotion. Trends in Cognitive Sciences, 23(10), 851–864.

    PubMed  PubMed Central  Google Scholar 

  • Shackman, A. J., Salomons, T. V., Slagter, H. A., Fox, A. S., Winter, J. J., & Davidson, R. J. (2011). The integration of negative affect, pain and cognitive control in the cingulate cortex. Nature Reviews Neuroscience, 12(3), 154–167.

    PubMed  PubMed Central  Google Scholar 

  • Sharot, T., & Garrett, N. (2016). Forming beliefs: Why valence matters. Trends in Cognitive Sciences, 20(1), 25–33.

    PubMed  Google Scholar 

  • Shinkareva, S. V., Wang, J., Kim, J., Facciani, M. J., Baucom, L. B., & Wedell, D. H. (2014). Representations of modality-specific affective processing for visual and auditory stimuli derived from functional magnetic resonance imaging data. Human Brain Mapping, 35(7), 3558–3568.

    PubMed  Google Scholar 

  • Skerry, A. E., & Saxe, R. (2014). A common neural code for perceived and inferred emotion. Journal of Neuroscience, 34(48), 15997–16008.

    PubMed  Google Scholar 

  • Todd, M. T., Nystrom, L. E., & Cohen, J. D. (2013). Confounds in multivariate pattern analysis: Theory and rule representation case study. Neuroimage, 77, 157–165.

    PubMed  Google Scholar 

  • Wang, Q., Cagna, B., Chaminade, T., & Takerkart, S. (2020). Inter-subject pattern analysis: A straightforward and powerful scheme for group-level MVPA. NeuroImage, 204, 116205.

    PubMed  Google Scholar 

  • Weber, C. E., Shinkareva, S. V., Kim, J., Gao, C., & Wedell, D. H. (2020). Evaluative conditioning of affective valence. Social Cognition, 38(2), 97–118.

    Google Scholar 

  • Yuen, K. S., Johnston, S. J., De Martino, F., Sorger, B., Formisano, E., Linden, D. E., & Goebel, R. (2012). Pattern classification predicts individuals’ responses to affective stimuli. Translational Neuroscience, 3(3), 278–287.

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Psychology, Institute for Mind and Brain, University of South Carolina, Columbia, SC, 29201, USA

    Svetlana V. Shinkareva, Chuanji Gao & Douglas Wedell

Authors
  1. Svetlana V. Shinkareva
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Chuanji Gao
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Douglas Wedell
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Svetlana V. Shinkareva.

Ethics declarations

Funding

This study was funded in part by the College of Arts and Sciences Faculty Research Initiative at the University of South Carolina.

Data Availability

https://osf.io/q4ry3/

Conflict of Interest

The authors declare that they have no conflict of interest.

Ethics Approval

This study was approved by the Institutional Review Board of the University of South Carolina.

Informed Consent

Written informed consent was obtained from all participants.

Consent to Publish

No identifying information is published.

Additional information

Handling Editor: Kristen Lindquist

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Shinkareva, S.V., Gao, C. & Wedell, D. Audiovisual Representations of Valence: a Cross-study Perspective. Affec Sci 1, 237–246 (2020). https://doi.org/10.1007/s42761-020-00023-9

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s42761-020-00023-9

Keywords

Search

Navigation