Abstract
Emotions are thought to be stored in long-term memory based on features as a memory unit of objects. Previous studies suggest that verbal features hierarchically represent emotions, such as sadness. Although emotion is associated with verbal and nonverbal features, it is not clear how these features interrelate in the memory representation of emotion. We hypothesized that some nonverbal features hierarchically function as “superordinate” or “subordinate” in sensory-perceptual dimensions representing emotions corresponding with verbal features. We focused on sadness as a target emotion and conducted a picture-word congruency judgment experiment. This study used 16 bodily-expression photographs (four-level trunk inclinations × four-level hand-arm postures) as sensory-perceptual features and 32 crying-related words as verbal features that prototypically represent sadness. Forty-eight participants assessed congruency concerning all combinations between bodily-expression pictures and crying-related words. Using a network graph analysis, bodily expressions were classified into three communities based on congruence ratings: the dominant bodily-expression community showed the highest verbal congruency composed of around-eye and overall-face hand-arm postures, irrespective of trunk inclination. Within this dominant community, trunk inclination subordinately modulated picture-word congruency according to bodily-activation properties. These findings support a feature-based representation of emotion through corresponding hierarchical structures between verbal and nonverbal features.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Data Availability
The data that support the findings of this study are available from the corresponding author upon reasonable request.
References
Armour, C., Fried, E. I., Deserno, M. K., Tsai, J., & Pietrzak, R. H. (2017). A network analysis of DSM-5 posttraumatic stress disorder symptoms and correlates in U.S. military veterans. Journal of Anxiety Disorders, 45, 49–59. https://doi.org/10.1016/j.janxdis.2016.11.008
Atkinson, A. P., Dittrich, W. H., Gemmell, A. J., & Young, A. W. (2004). Emotion perception from dynamic and static body expressions in point-light and full-light displays. Perception, 33(6), 717–746. https://doi.org/10.1068/p5096
Barabási, A.-L. (2013). Network science. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, 371(1987), 20120375. https://doi.org/10.1098/rsta.2012.0375
Barrett, L. F. (2006). Solving the emotion paradox: Categorization and the experience of emotion. Personality and Social Psychology Review, 10(1), 20–46. https://doi.org/10.1207/s15327957pspr1001_2
Borghi, A. M., Binkofski, F., Castelfranchi, C., Cimatti, F., Scorolli, C., & Tummolini, L. (2017). The challenge of abstract concepts. Psychological Bulletin, 143(3), 263–292. https://doi.org/10.1037/bul0000089
Costa, L. d. F., Rodrigues, F. A., Travieso, G., & Villas Boas, P. R. (2007). Characterization of complex networks: A survey of measurements. Advances in Physics, 56(1), 167–242. https://doi.org/10.1080/00018730601170527
Dreyer, F. R., & Pulvermüller, F. (2018). Abstract semantics in the motor system? – An event-related fMRI study on passive reading of semantic word categories carrying abstract emotional and mental meaning. Cortex, 100, 52–70. https://doi.org/10.1016/j.cortex.2017.10.021
Fortunato, S. (2010). Community detection in graphs. Physics Reports, 486(3–5), 75–174. https://doi.org/10.1016/j.physrep.2009.11.002
Harpaintner, M., Trumpp, N. M., & Kiefer, M. (2018). The semantic content of abstract concepts: A property listing study of 296 abstract words. Frontiers in Psychology, 9, 1748. https://doi.org/10.3389/fpsyg.2018.01748
Hoemann, K., Xu, F., & Barrett, L. F. (2019). Emotion words, emotion concepts, and emotional development in children: A constructionist hypothesis. Developmental Psychology, 55(9), 1830–1849. https://doi.org/10.1037/dev0000686
Hoenig, K., Sim, E.-J., Bochev, V., Herrnberger, B., & Kiefer, M. (2008). Conceptual flexibility in the human brain: Dynamic recruitment of semantic maps from visual, motor, and motion-related areas. Journal of Cognitive Neuroscience, 20(10), 1799–1814. https://doi.org/10.1162/jocn.2008.20123
Inose, H. (2007). Translating Japanese onomatopoeia and mimetic words. Translation Research Projects, 97–116.
Izard, C. E. (1991). The psychology of emotions. Plenum Press.
Kiefer, M., & Harpaintner, M. (2020). Varieties of abstract concepts and their grounding in perception or action. Open Psychology, 2(1), 119–137. https://doi.org/10.1515/psych-2020-0104
Kiefer, M., & Pulvermüller, F. (2012). Conceptual representations in mind and brain: Theoretical developments, current evidence and future directions. Cortex, 48(7), 805–825. https://doi.org/10.1016/j.cortex.2011.04.006
Kleinsmith, A., & Bianchi-Berthouze, N. (2013). Affective body expression perception and recognition: A survey. IEEE Transactions on Affective Computing, 4(1), 15–33. https://doi.org/10.1109/T-AFFC.2012.16
Kosslyn, S. M., Ganis, G., & Thompson, W. L. (2001). Neural foundations of imagery. Nature Reviews Neuroscience, 2(9), 635–642. https://doi.org/10.1038/35090055
Kosslyn, S. M., Thompson, W. L., & Ganis, G. (2006). The case for mental imagery. Oxford University Press. https://doi.org/10.1093/acprof:oso/9780195179088.001.0001
McClelland, J. L., & Rogers, T. T. (2003). The parallel distributed processing approach to semantic cognition. Nature Reviews Neuroscience, 4(4), 310–322. https://doi.org/10.1038/nrn1076
McClelland, J. L., Rumelhart, D. E., & PDP Research Group. (1986). Parallel distributed processing: Explorations in the microstructure of cognition. MIT Press.
McRae, K., Cree, G. S., Seidenberg, M. S., & Mcnorgan, C. (2005). Semantic feature production norms for a large set of living and nonliving things. Behavior Research Methods, 37(4), 547–559. https://doi.org/10.3758/BF03192726
Mesquita, B., & Boiger, M. (2014). Emotions in context: A Sociodynamic model of emotions. Emotion Review, 6(4), 298–302. https://doi.org/10.1177/1754073914534480
Mesquita, B., Boiger, M., & De Leersnyder, J. (2016). The cultural construction of emotions. Current Opinion in Psychology, 8, 31–36. https://doi.org/10.1016/j.copsyc.2015.09.015
Moseley, R., Carota, F., Hauk, O., Mohr, B., & Pulvermüller, F. (2012). A role for the motor system in binding abstract emotional meaning. Cerebral Cortex, 22(7), 1634–1647. https://doi.org/10.1093/cercor/bhr238
Nesse, R. M. (1990). Evolutionary explanations of emotions. Human Nature, 1(3), 261–289. https://doi.org/10.1007/BF02733986
Newman, M. E. J., & Girvan, M. (2004). Finding and evaluating community structure in networks. Physical Review E, 69(2), 026113. https://doi.org/10.1103/PhysRevE.69.026113
Ono, M. (2007). Giongo/Gitaigo 4500 nihongo onomatope jiten [Japanese onomatopoeia dictionary 4500 mimetics]. Shogakukan.
Patterson, K., Nestor, P. J., & Rogers, T. T. (2007). Where do you know what you know? The representation of semantic knowledge in the human brain. Nature Reviews Neuroscience, 8(12), 976–987. https://doi.org/10.1038/nrn2277
Pearson, J., & Kosslyn, S. M. (2015). The heterogeneity of mental representation: Ending the imagery debate. Proceedings of the National Academy of Sciences, 112(33), 10089–10092. https://doi.org/10.1073/pnas.1504933112
Poyo Solanas, M., Vaessen, M., & de Gelder, B. (2020). Computation-based feature representation of body expressions in the human brain. Cerebral Cortex, 30(12), 6376–6390. https://doi.org/10.1093/cercor/bhaa196
Pylyshyn, Z. W. (1973). What the mind’s eye tells the mind’s brain: A critique of mental imagery. Psychological Bulletin, 80(1), 1–24.
Lambon Ralph, M. A., Jefferies, E., Patterson, K., & Rogers, T. T. (2017). The neural and computational bases of semantic cognition. Nature Reviews Neuroscience, 18(1), 42–55. https://doi.org/10.1038/nrn.2016.150
Rogers, T. T., & McClelland, J. L. (2004). Semantic cognition: A parallel distributed processing approach. MIT press.
Rosch, E. (1975). Cognitive representations of semantic categories. Journal of Experimental Psychology: General, 104(3), 192–233. https://doi.org/10.1037/0096-3445.104.3.192
Rydblom, O. (2016). Universals in usage of and attitudes to onomatopoeia. (Bach. thesis). Lunds Universitet, Sweden. Retrieved from http://lup.lub.lu.se/student-papers/record/8888824. Accessed 20 Apr 2022.
Shiota, M. N., & Kalat, J. W. (2018). Emotion (3rd edition). Oxford University Press. https://www.kriso.ee/emotion-3rd-edition-db-9780190635510.html
Shirai, M., & Soshi, T. (2019). Why is heartache associated with sadness? Sadness is represented by specific physical pain through verbal knowledge. PLoS One, 14(5), e0216331. https://doi.org/10.1371/journal.pone.0216331
Shirai, M., & Soshi, T. (2021). The role of bodily expression in memory representations of sadness. Journal of Nonverbal Behavior. https://doi.org/10.1007/s10919-021-00360-8
Shirai, M., & Suzuki, N. (2017). Is sadness only one emotion? Psychological and physiological responses to sadness induced by two different situations: “Loss of someone” and “failure to achieve a goal.” Frontiers in Psychology, 8. https://doi.org/10.3389/fpsyg.2017.00288
Shirai, M., Soshi, T., & Suzuki, N. (2021). Knowledge of sadness: Emotion-related behavioral words differently encode loss and failure sadness. Current Psychology, 40(2), 895–909. https://doi.org/10.1007/s12144-018-0010-9
Simmons, W. K., Ramjee, V., Beauchamp, M. S., McRae, K., Martin, A., & Barsalou, L. W. (2007). A common neural substrate for perceiving and knowing about color. Neuropsychologia, 45(12), 2802–2810. https://doi.org/10.1016/j.neuropsychologia.2007.05.002
Tamori, I. (2002). Onomatope: Giongo/Gitaigo wo tanoshimu [onomatopoeia: Enjoy giongo/gitaigo]. Iwanami.
Tyler, L., & Moss, H. (2001). Towards a distributed account of conceptual knowledge. Trends in Cognitive Sciences, 5(6), 244–252. https://doi.org/10.1016/S1364-6613(00)01651-X
Vingerhoets, A. J. J. M. (2013). Why only humans weep: Unravelling the mysteries of tears. In Why Only Humans Weep. Oxford University Press.
Wallbott, H. G. (1998). Bodily expression of emotion. European Journal of Social Psychology, 28(6), 879–896. https://doi.org/10.1002/(SICI)1099-0992(1998110)28:6<879::AID-EJSP901>3.0.CO;2-W
Witkower, Z., & Tracy, J. L. (2019). Bodily communication of emotion: Evidence for extrafacial behavioral expressions and available coding systems. Emotion Review, 11(2), 184–193. https://doi.org/10.1177/1754073917749880
Witkower, Z., Mercadante, E. J., & Tracy, J. L. (2020). How affect shapes status: Distinct emotional experiences and expressions facilitate social hierarchy navigation. Current Opinion in Psychology, 33, 18–22. https://doi.org/10.1016/j.copsyc.2019.06.006
Acknowledgements
We gratefully acknowledge the work of participants. We are also grateful to the reviewers for their valuable comments. We would like to thank Rika Yanase for her support and Editage for English language editing. This work was financially supported by JSPS KAKENHI Grant Number JP 20 K14143.
Funding
This work was financially supported by JSPS KAKENHI (Grant Number: JP 20 K14143).
Author information
Authors and Affiliations
Contributions
MS and TS designed the experiments. MS performed the survey. MS and TS analyzed the data. MS wrote the manuscript in consultation with TS. All authors have discussed the results and contributed to the final manuscript.
Corresponding author
Ethics declarations
Research Involving Human Participants
The experimental protocol was approved by the University Institutional Ethics Board of the first author’s university and was conducted in accordance with the 1975 Helsinki Declaration and its later amendments.
Informed Consent
All participants provided informed consent prior to completing the survey.
Competing Interests
All authors declare that they have no conflicts of interest.
Additional information
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Shirai, M., Soshi, T. Hierarchical memory representation of verbal and nonverbal features for emotion. Curr Psychol 42, 21298–21309 (2023). https://doi.org/10.1007/s12144-022-03200-0
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s12144-022-03200-0