Abstract
The brain functional mechanisms underlying emotional changes have been primarily studied based on the traditional task design with discrete and simple stimuli. However, the brain state transitions when exposed to continuous and naturalistic stimuli with rich affection variations remain poorly understood. This study proposes a dynamic hyperalignment algorithm (dHA) to functionally align the inter-subject neural activity. The hidden Markov model (HMM) was used to study how the brain dynamics responds to emotion during long-time movie-viewing activity. The results showed that dHA significantly improved inter-subject consistency and allowed more consistent temporal HMM states across participants. Afterward, grouping the emotions in a clustering dendrogram revealed a hierarchical grouping of the HMM states. Further emotional sensitivity and specificity analyses of ordered states revealed the most significant differences in happiness and sadness. We then compared the activation map in HMM states during happiness and sadness and found significant differences in the whole brain, but strong activation was observed during both in the superior temporal gyrus, which is related to the early process of emotional prosody processing. A comparison of the inter-network functional connections indicates unique functional connections of the memory retrieval and cognitive network with the cerebellum network during happiness. Moreover, the persistent bilateral connections among salience, cognitive, and sensorimotor networks during sadness may reflect the interaction between high-level cognitive networks and low-level sensory networks. The main results were verified by the second session of the dataset. All these findings enrich our understanding of the brain states related to emotional variation during naturalistic stimuli.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Availability of Data and Material
Code Availability
References
An, S., Han, X., Wu, B., Shi, Z., Marks, M., Wang, S., et al. (2018). Neural activation in response to the two sides of emotion. Neuroscience Letters, 684, 140–144. https://doi.org/10.1016/j.neulet.2018.07.011
Baumgartner, T., Kai, L., Schmidt, C. F., & Jäncke, L. (2006). The emotional power of music: How music enhances the feeling of affective pictures. Brain Research, 1075(1), 151–164.
Ben-Yakov, A., & Henson, R. N. (2018). The Hippocampal Film Editor: Sensitivity and Specificity to Event Boundaries in Continuous Experience. Journal of Neuroscience, 38(47), 10057–10068. https://doi.org/10.1523/JNEUROSCI.0524-18.2018
Boldt, R., Malinen, S., Seppa, M., Tikka, P., Savolainen, P., Hari, R., et al. (2013). Listening to an audio drama activates two processing networks, one for all sounds another exclusively for speech. PLoS ONE, 8(5), e64489. https://doi.org/10.1371/journal.pone.0064489
Cabral, J., Vidaurre, D., Marques, P., Magalhaes, R., Silva Moreira, P., Miguel Soares, J., et al. (2017). Cognitive performance in healthy older adults relates to spontaneous switching between states of functional connectivity during rest. Scientific Reports, 7(1), 5135. https://doi.org/10.1038/s41598-017-05425-7
Chan, H. Y., Smidts, A., Schoots, V. C., Sanfey, A. G., & Boksem, M. A. S. (2020). Decoding dynamic affective responses to naturalistic videos with shared neural patterns. NeuroImage, 216, 116618. https://doi.org/10.1016/j.neuroimage.2020.116618
Chen, P. A., Jolly, E., Cheong, J. H., & Chang, L. J. (2020). Intersubject representational similarity analysis reveals individual variations in affective experience when watching erotic movies. NeuroImage, 216, 116851. https://doi.org/10.1016/j.neuroimage.2020.116851
Craik, A., He, Y., & Contreras-Vidal, J. L. (2019). Deep learning for electroencephalogram (EEG) classification tasks: A review. Journal of Neural Engineering, 16(3), 031001. https://doi.org/10.1088/1741-2552/ab0ab5
Di, X., & Biswal, B. B. (2020). Intersubject consistent dynamic connectivity during natural vision revealed by functional MRI. NeuroImage, 216, 116698. https://doi.org/10.1016/j.neuroimage.2020.116698
Erhardt, E. B., Allen, E. A., Wei, Y., Eichele, T., & Calhoun, V. D. (2012). SimTB a simulation toolbox for fMRI data under a model of spatiotemporal separability. NeuroImage, 59(4), 4160–4167.
Feilong, M., Nastase, S. A., Guntupalli, J. S., & Haxby, J. V. (2018). Reliable individual differences in fine-grained cortical functional architecture. NeuroImage, 183, 375–386. https://doi.org/10.1016/j.neuroimage.2018.08.029
Fitzgerald, P. B., Laird, A. R., Maller, J., & Daskalakis, Z. J. (2008). A meta-analytic study of changes in brain activation in depression. Human Brain Mapping, 29(6), 683–695. https://doi.org/10.1002/hbm.20426
Goldin, P. R., McRae, K., Ramel, W., & Gross, J. J. (2008). The neural bases of emotion regulation: Reappraisal and suppression of negative emotion. Biological Psychiatry, 63(6), 577–586. https://doi.org/10.1016/j.biopsych.2007.05.031
Guntupalli, J. S., Hanke, M., Halchenko, Y. O., Connolly, A. C., Ramadge, P. J., & Haxby, J. V. (2016). A Model of Representational Spaces in Human Cortex. Cerebral Cortex, 26(6), 2919–2934. https://doi.org/10.1093/cercor/bhw068
Hanke, M., Adelhofer, N., Kottke, D., Iacovella, V., Sengupta, A., Kaule, F. R., et al. (2016). A studyforrest extension, simultaneous fMRI and eye gaze recordings during prolonged natural stimulation. Scientific Data, 3, 160092. https://doi.org/10.1038/sdata.2016.92
Haxby, J. V., Connolly, A. C., & Guntupalli, J. S. (2014). Decoding neural representational spaces using multivariate pattern analysis. Annual Review of Neuroscience, 37, 435–456. https://doi.org/10.1146/annurev-neuro-062012-170325
Haxby, J. V., Guntupalli, J. S., Connolly, A. C., Halchenko, Y. O., Conroy, B. R., Gobbini, M. I., et al. (2011). A common high-dimensional model of the representational space in human ventral temporal cortex. Neuron, 72(2), 404–416. https://doi.org/10.1016/j.neuron.2011.08.026
Haxby, J. V., Guntupalli, J. S., Nastase, S. A., & Feilong, M. (2020). Hyperalignment: Modeling shared information encoded in idiosyncratic cortical topographies. eLife, 9, e56601. https://doi.org/10.7554/eLife.56601
He, Z., Li, Z., Yang, F., Wang, L., Li, J., Zhou, C., et al. (2020). Advances in Multimodal Emotion Recognition Based on Brain-Computer Interfaces. Brain Sciences, 10(10), 687. https://doi.org/10.3390/brainsci10100687
Hunyadi, B., Woolrich, M. W., Quinn, A. J., Vidaurre, D., & De Vos, M. (2019). A dynamic system of brain networks revealed by fast transient EEG fluctuations and their fMRI correlates. NeuroImage, 185, 72–82. https://doi.org/10.1016/j.neuroimage.2018.09.082
Jaaskelainen, I. P., Sams, M., Glerean, E., & Ahveninen, J. (2021). Movies and narratives as naturalistic stimuli in neuroimaging. NeuroImage, 224, 117445. https://doi.org/10.1016/j.neuroimage.2020.117445
Jeong, J. W., Diwadkar, V. A., Chugani, C. D., Sinsoongsud, P., Muzik, O., Behen, M. E., et al. (2011). Congruence of happy and sad emotion in music and faces modifies cortical audiovisual activation. NeuroImage, 54(4), 2973–2982. https://doi.org/10.1016/j.neuroimage.2010.11.017
Jiahui, G., Feilong, M., di Oleggio Castello, M. V., Guntupalli, J. S., Chauhan, V., Haxby, J. V., et al. (2020). Predicting individual face-selective topography using naturalistic stimuli. NeuroImage, 216, 116458. https://doi.org/10.1016/j.neuroimage.2019.116458
Labs, A., Reich, T., Schulenburg, H., Boennen, M., Mareike, G., Golz, M., et al. (2015). Portrayed emotions in the movie "Forrest Gump". F1000Research, 4, 92. https://doi.org/10.12688/f1000research.6230.1
Lee, C. S., Aly, M., & Baldassano, C. (2021). Anticipation of temporally structured events in the brain. eLife, 10, e64972. https://doi.org/10.7554/eLife.64972
Lettieri, G., Handjaras, G., Ricciardi, E., Leo, A., Papale, P., Betta, M., et al. (2019). Emotionotopy in the human right temporo-parietal cortex. Nature Communications, 10(1), 5568. https://doi.org/10.1038/s41467-019-13599-z
Lichev, V., Sacher, J., Ihme, K., Rosenberg, N., Quirin, M., Lepsien, J., et al. (2015). Automatic emotion processing as a function of trait emotional awareness: An fMRI study. Social Cognitive and Affective Neuroscience, 10(5), 680–689. https://doi.org/10.1093/scan/nsu104
Lindquist, K. A., Satpute, A. B., Wager, T. D., Weber, J., & Barrett, L. F. (2016). The Brain Basis of Positive and Negative Affect: Evidence from a Meta-Analysis of the Human Neuroimaging Literature. Cerebral Cortex, 26(5), 1910–1922. https://doi.org/10.1093/cercor/bhv001
Liu, X., Zhen, Z., Yang, A., Bai, H., & Liu, J. (2019). A manually denoised audio-visual movie watching fMRI dataset for the studyforrest project. Scientific Data, 6(1), 295. https://doi.org/10.1038/s41597-019-0303-3
Meer, J. N. V., Breakspear, M., Chang, L. J., Sonkusare, S., & Cocchi, L. (2020). Movie viewing elicits rich and reliable brain state dynamics. Nature Communications, 11(1), 5004. https://doi.org/10.1038/s41467-020-18717-w
Moraczewski, D., Chen, G., & Redcay, E. (2018). Inter-subject synchrony as an index of functional specialization in early childhood. Scientific Reports, 8(1), 2252. https://doi.org/10.1038/s41598-018-20600-0
Nastase, S. A., Liu, Y. F., Hillman, H., Norman, K. A., & Hasson, U. (2020). Leveraging shared connectivity to aggregate heterogeneous datasets into a common response space. NeuroImage, 217, 116865. https://doi.org/10.1016/j.neuroimage.2020.116865
Nummenmaa, L., Glerean, E., Viinikainen, M., Jaaskelainen, I. P., Hari, R., & Sams, M. (2012). Emotions promote social interaction by synchronizing brain activity across individuals. Proceedings of the National Academy of Sciences, 109(24), 9599–9604. https://doi.org/10.1073/pnas.1206095109
Ochsner, K. N., Ray, R. D., Cooper, J. C., Robertson, E. R., Chopra, S., Gabrieli, J. D., et al. (2004). For better or for worse: Neural systems supporting the cognitive down- and up-regulation of negative emotion. NeuroImage, 23(2), 483–499. https://doi.org/10.1016/j.neuroimage.2004.06.030
Power, J. D., Cohen, A. L., Nelson, S. M., Wig, G. S., Barnes, K. A., Church, J. A., et al. (2011). Functional network organization of the human brain. Neuron, 72(4), 665–678. https://doi.org/10.1016/j.neuron.2011.09.006
Quinn, A. J., Vidaurre, D., Abeysuriya, R., Becker, R., Nobre, A. C., & Woolrich, M. W. (2018). Task-Evoked Dynamic Network Analysis Through Hidden Markov Modeling. Frontiers in Neuroscience, 12, 603. https://doi.org/10.3389/fnins.2018.00603
Ramirez, F. M., Revsine, C., & Merriam, E. P. (2020). What do across-subject analyses really tell us about neural coding? Neuropsychologia, 143, 107489. https://doi.org/10.1016/j.neuropsychologia.2020.107489
Raz, G., Touroutoglou, A., Wilson-Mendenhall, C., Gilam, G., Lin, T., Gonen, T., et al. (2016). Functional connectivity dynamics during film viewing reveal common networks for different emotional experiences. Cognitive Affective and Behavioral Neuroscience, 16(4), 709–723. https://doi.org/10.3758/s13415-016-0425-4
Redcay, E., & Moraczewski, D. (2020). Social cognition in context: A naturalistic imaging approach. NeuroImage, 216, 116392. https://doi.org/10.1016/j.neuroimage.2019.116392
Richardson, H., Lisandrelli, G., Riobueno-Naylor, A., & Saxe, R. (2018). Development of the social brain from age three to twelve years. Nature Communications, 9(1), 1027. https://doi.org/10.1038/s41467-018-03399-2
Rocca, R., Coventry, K. R., Tylen, K., Staib, M., Lund, T. E., & Wallentin, M. (2020). Language beyond the language system: Dorsal visuospatial pathways support processing of demonstratives and spatial language during naturalistic fast fMRI. NeuroImage, 216, 116128. https://doi.org/10.1016/j.neuroimage.2019.116128
Saarimaki, H. (2021). Naturalistic Stimuli in Affective Neuroimaging: A Review. Frontiers in Human Neuroscience, 15, 675068. https://doi.org/10.3389/fnhum.2021.675068
Saarimaki, H., Gotsopoulos, A., Jaaskelainen, I. P., Lampinen, J., Vuilleumier, P., Hari, R., et al. (2016). Discrete Neural Signatures of Basic Emotions. Cerebral Cortex, 26(6), 2563–2573. https://doi.org/10.1093/cercor/bhv086
Sachs, M. E., Habibi, A., Damasio, A., & Kaplan, J. T. (2020). Dynamic intersubject neural synchronization reflects affective responses to sad music. NeuroImage, 218, 116512. https://doi.org/10.1016/j.neuroimage.2019.116512
Shappell, H., Caffo, B. S., Pekar, J. J., & Lindquist, M. A. (2019). Improved state change estimation in dynamic functional connectivity using hidden semi-Markov models. NeuroImage, 191, 243–257. https://doi.org/10.1016/j.neuroimage.2019.02.013
Simony, E., & Chang, C. (2020). Analysis of stimulus-induced brain dynamics during naturalistic paradigms. NeuroImage, 216, 116461. https://doi.org/10.1016/j.neuroimage.2019.116461
Stevner, A. B. A., Vidaurre, D., Cabral, J., Rapuano, K., Nielsen, S. F. V., Tagliazucchi, E., et al. (2019). Discovery of key whole-brain transitions and dynamics during human wakefulness and non-REM sleep. Nature Communications, 10(1), 1035. https://doi.org/10.1038/s41467-019-08934-3
Vanderwal, T., Eilbott, J., & Castellanos, F. X. (2019). Movies in the magnet: Naturalistic paradigms in developmental functional neuroimaging. Developmental Cognitive Neuroscience, 36, 100600. https://doi.org/10.1016/j.dcn.2018.10.004
Vidaurre, D., Abeysuriya, R., Becker, R., Quinn, A. J., Alfaro-Almagro, F., Smith, S. M., et al. (2018a). Discovering dynamic brain networks from big data in rest and task. NeuroImage, 180, 646–656. https://doi.org/10.1016/j.neuroimage.2017.06.077
Vidaurre, D., Hunt, L. T., Quinn, A. J., Hunt, B. A. E., Brookes, M. J., Nobre, A. C., et al. (2018b). Spontaneous cortical activity transiently organises into frequency specific phase-coupling networks. Nature Communications, 9(1), 2987. https://doi.org/10.1038/s41467-018-05316-z
Vidaurre, D., Myers, N. E., Stokes, M., Nobre, A. C., & Woolrich, M. W. (2019). Temporally Unconstrained Decoding Reveals Consistent but Time-Varying Stages of Stimulus Processing. Cerebral Cortex, 29(2), 863–874. https://doi.org/10.1093/cercor/bhy290
Vidaurre, D., Quinn, A. J., Baker, A. P., Dupret, D., Tejero-Cantero, A., & Woolrich, M. W. (2016). Spectrally resolved fast transient brain states in electrophysiological data. NeuroImage, 126, 81–95. https://doi.org/10.1016/j.neuroimage.2015.11.047
Vidaurre, D., Smith, S. M., & Woolrich, M. W. (2017). Brain network dynamics are hierarchically organized in time. Proceedings of the National Academy of Sciences, 114(48), 12827–12832. https://doi.org/10.1073/pnas.1705120114
Wang, J., Wang, X., Xia, M., Liao, X., Evans, A., & Yong, H. (2015). GRETNA: A graph theoretical network analysis toolbox for imaging connectomics. Frontiers in Human Neuroscience, 9(386), 386.
Wang, X.-W., Nie, D., & Lu, B.-L. (2014). Emotional state classification from EEG data using machine learning approach. Neurocomputing, 129, 94–106. https://doi.org/10.1016/j.neucom.2013.06.046
Warnick, R., Guindani, M., Erhardt, E., Allen, E., Calhoun, V., & Vannucci, M. (2018). A Bayesian Approach for Estimating Dynamic Functional Network Connectivity in fMRI Data. Journal of the American Statistical Association, 113(521), 134–151. https://doi.org/10.1080/01621459.2017.1379404
Wittfoth, M., Schroder, C., Schardt, D. M., Dengler, R., Heinze, H. J., & Kotz, S. A. (2010). On emotional conflict: Interference resolution of happy and angry prosody reveals valence-specific effects. Cerebral Cortex, 20(2), 383–392. https://doi.org/10.1093/cercor/bhp106
Xia, M., Wang, J., Yong, H., & Peter, C. (2013). BrainNet Viewer: A Network Visualization Tool for Human Brain Connectomics. Plos One, 8(7), e68910.
Xiao, X., Zhou, Y., Liu, J., Ye, Z., Yao, L., Zhang, J., et al. (2020). Individual-specific and shared representations during episodic memory encoding and retrieval. NeuroImage, 217, 116909. https://doi.org/10.1016/j.neuroimage.2020.116909
Yankouskaya, A., & Sui, J. (2021). Self-Positivity or Self-Negativity as a Function of the Medial Prefrontal Cortex. Brain Sciences, 11(2), 264. https://doi.org/10.3390/brainsci11020264
Young, C. B., Raz, G., Everaerd, D., Beckmann, C. F., Tendolkar, I., Hendler, T., et al. (2017). Dynamic Shifts in Large-Scale Brain Network Balance As a Function of Arousal. Journal of Neuroscience, 37(2), 281–290. https://doi.org/10.1523/JNEUROSCI.1759-16.2016
Zhang, G., Cai, B., Zhang, A., Stephen, J. M., Wilson, T. W., Calhoun, V. D., et al. (2020). Estimating Dynamic Functional Brain Connectivity With a Sparse Hidden Markov Model. IEEE Transactions on Medical Imaging, 39(2), 488–498. https://doi.org/10.1109/TMI.2019.2929959
Zhang, G. Y., & Liu, X. (2021). Investigation of functional brain network reconfiguration during exposure to naturalistic stimuli using graph-theoretical analysis. Journal of Neural Engineering, 18(5), 056027. https://doi.org/10.1088/1741-2552/ac20e7
Funding
The research leading to these results has received funding from the National Natural Science Foundation of China (No. 61876126).
Author information
Authors and Affiliations
Contributions
Chenhao Tan performed data analysis and wrote the original manuscript. Xin Liu contributed to data preprocessing. Gaoyan Zhang proposed the data analysis method and revised the manuscript.
Corresponding author
Ethics declarations
Ethics Approval
The studyforrest dataset provides the required ethics and consent needed for study and dissemination, such that no further institutional review board (IRB) approval is required.
Conflicts of Interest/Competing Interests
The authors declare that there are no conflicts of interest regarding the publication of this paper.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Supplementary Information
Below is the link to the electronic supplementary material.
Rights and permissions
About this article
Cite this article
Tan, C., Liu, X. & Zhang, G. Inferring Brain State Dynamics Underlying Naturalistic Stimuli Evoked Emotion Changes With dHA-HMM. Neuroinform 20, 737–753 (2022). https://doi.org/10.1007/s12021-022-09568-5
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s12021-022-09568-5