Abstract
Over recent years, although multi-sensory studies have increasingly revealed modality-specific mechanisms underlying lexical processing, validated lexical databases with reliable affective norms in both visual and auditory modalities have been scantly established, especially in Chinese. Therefore, this study aims to establish a cross-modal affective database consisting of 350 two-character Chinese emotion-label words, and investigate how neutral speech prosody changes semantic emotion perception in Chinese. Affective ratings of six variables were collected, including familiarity, valence, arousal, dominance, intensity and emotion type, from 364 participants in both visual and auditory modalities. Reliability and validity of the ratings were strictly examined. Statistical analyses manifested the U-shape relationships for valence-arousal and valence-dominance pairwise correlations in the within-modality setting, and identified the existence of a neutral prosody influence on semantic emotion access, thus showing no direct juxtaposition in lexical emotion perception across the two modalities. Specifically, the auditory modality imposed a neutrality convergence on valence perception, decreased the familiarity and dominance feelings, but did not change the intensity parameter. This study is among the first to introduce the multi-modal perspective into Chinese lexical database construction, which not only supplements extant research tools for selecting grammatically homogeneous Chinese emotion-label words as experiment stimuli, but also warrants further investigations on how speech prosody influences lexical semantic perception.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Data availability
All data generated or analysed during this study are included in this published article and its supplementary information files.
References
Anderson, C., & Platten, C. R. (2011). Sleep deprivation lowers inhibition and enhances impulsivity to negative stimuli. Behavioural Brain Research, 217(2), 463–466. https://doi.org/10.1016/j.bbr.2010.09.020
Barr, D. J. (2013). Random effects structure for testing interactions in linear mixed-effects models. Frontiers in Psychology, 4,. https://doi.org/10.3389/fpsyg.2013.00328
Bartoń, K. (2020). MuMIn: Multi-Model Inference. R package, version 1.43.17. Retrieved from: https://cran.r-project.org/package=MuMIn. Accessed 3 April 2022
Bates, D., Mächler, M., Bolker, B. M., & Walker, S. C. (2015). Fitting linear mixed-effects models using lme4. Journal of Statistical Software, 67(1), 1–48. https://doi.org/10.18637/jss.v067.i01
Bergman, P., Västfjäll, D., Tajadura-Jiménez, A., & Asutay, E. (2016). Auditory-induced emotion mediates perceptual categorization of everyday sounds. Frontiers in Psychology, 7,. https://doi.org/10.3389/fpsyg.2016.01565
Bhullar, N. (2013). Rating of intensity of emotions across auditory, visual, and auditory-visual sensory domains. Current Psychology, 32(2), 186–196. https://doi.org/10.1007/s12144-013-9173-6
Bradley, M. M., & Lang, P. J. (1999). Affective norms for English words (ANEW): Instruction manual and affective ratings (Technical Report C-1). University of Florida, NIMH Center for Research in Psychophysiology.
Cai, Q., & Brysbaert, M. (2010). SUBTLEX-CH: Chinese word and character frequencies based on film subtitles. PLoS ONE, 5(6). https://doi.org/10.1371/journal.pone.0010729
Chang, Y. N., Hsu, C. H., Tsai, J. L., Chen, C. L., & Lee, C. Y. (2016). A psycholinguistic database for traditional Chinese character naming. Behavior Research Methods, 48, 112–122. https://doi.org/10.3758/s13428-014-0559-7
Citron, F. M. M., Weekes, B. S., & Ferstl, E. C. (2014). How are affective word ratings related to lexico-semantic properties? Evidence from the Sussex Affective Word List (SAWL). Applied Psycholinguistics, 35(2), 313–331. https://doi.org/10.1017/S0142716412000409
Ding, H., & Zhang, Y. (2023). Speech prosody in mental disorders. Annual Review of Linguistics, 9, 335–355. https://doi.org/10.1146/annurev-linguistics-030421-065139
Ekman, P. (1992). An argument for basic emotions. Cognition and Emotion, 6(3–4), 169–200. https://doi.org/10.1080/02699939208411068
Fairfield, B., Ambrosini, E., Mammarella, N., & Montefinese, M. (2017). Affective norms for Italian words in older adults: Age differences in ratings of valence, arousal and dominance. PLoS ONE, 12(1). https://doi.org/10.1371/journal.pone.0169472
Fontaine, J. R. J., Scherer, K. R., Roesch, E. B., & Ellsworth, P. C. (2007). The world of emotions is not two-dimensional. Psychological Science, 18(12), 1050–1057. https://doi.org/10.1111/j.1467-9280.2007.02024.x
Ge, Y., Zhao, G., Zhang, Y., Houston, R. J., & Song, J. (2019). A standardised database of Chinese emotional film clips. Cognition and Emotion, 33(5), 976–990. https://doi.org/10.1080/02699931.2018.1530197
Hinojosa, J. A., Moreno, E. M., & Ferré, P. (2019). Affective neurolinguistics: Towards a framework for reconciling language and emotion. Language Cognition and Neuroscience, 35(7), 813–839. https://doi.org/10.1080/23273798.2019.1620957
Inceoglu, S. (2022). Language experience and subjective word familiarity on the multimodal perception of non-native vowels. Language and Speech, 65(1), 173–192. https://doi.org/10.1177/0023830921998723
Kuznetsova, A., Brockhoff, P. B., & Christensen, R. H. B. (2017). lmerTest package: Tests in linear mixed effects models. Journal of Statistical Software, 82(13), 1–26. https://doi.org/10.18637/jss.v082.i13
Lenth, R. V. (2016). Least-squares means: The R package lsmeans. Journal of Statistical Software, 69(1), 1–33. https://doi.org/10.18637/jss.v069.i01
Liebenthal, E., Silbersweig, D. A., & Stern, E. (2016). The language, tone and prosody of emotions: Neural substrates and dynamics of spoken-word emotion perception. Frontiers in Neuroscience, 10,. https://doi.org/10.3389/fnins.2016.00506
Lin, J., & Yao, Y. (2016). Encoding emotion in Chinese: A database of Chinese emotion words with information of emotion type, intensity, and valence. Lingua Sinica, 2(1). https://doi.org/10.1186/s40655-016-0015-y
Lin, Y., Ding, H., & Zhang, Y. (2020). Prosody dominates over semantics in emotion word processing: Evidence from cross-channel and cross-modal Stroop effects. Journal of Speech Language and Hearing Research, 63(3), 896–912. https://doi.org/10.1044/2020_JSLHR-19-00258
Liu, P., Lu, Q., Zhang, Z., Tang, J., & Han, B. (2021). Age-related differences in affective norms for Chinese words (AANC). Frontiers in Psychology, 12,. https://doi.org/10.3389/fpsyg.2021.585666
Liu, P., & Pell, M. D. (2012). Recognizing vocal emotions in Mandarin Chinese: A validated database of Chinese vocal emotional stimuli. Behavior Research Methods, 44(4), 1042–1051. https://doi.org/10.3758/s13428-012-0203-3
López Zunini, R. A., Baart, M., Samuel, A. G., & Armstrong, B. C. (2020). Lexical access versus lexical decision processes for auditory, visual, and audiovisual items: Insights from behavioral and neural measures. Neuropsychologia, 137,. https://doi.org/10.1016/j.neuropsychologia.2019.107305
Meteyard, L., & Davies, R. A. (2020). Best practice guidance for linear mixed-effects models in psychological science. Journal of Memory and Language, 112,. https://doi.org/10.1016/j.jml.2020.104092
Monnier, C., & Syssau, A. (2014). Affective norms for French words (FAN). Behavior Research Methods, 46(4), 1128–1137. https://doi.org/10.3758/s13428-013-0431-1
Montefinese, M., Ambrosini, E., Fairfield, B., & Mammarella, N. (2014). The adaptation of the affective norms for English words (ANEW) for Italian. Behavior Research Methods, 46(3), 887–903. https://doi.org/10.3758/s13428-013-0405-3
Nakagawa, S., Johnson, P. C., & Schielzeth, H. (2017). The coefficient of determination R2 and intra-class correlation coefficient from generalized linear mixed-effects models revisited and expanded. Journal of the Royal Society Interface, 14(134), 20170213. https://doi.org/10.1098/rsif.2017.0213
National Language Commission, Ministry of Education, P. R. C. (2011–2018). Language Situation in China. The Commercial Press.
Paulmann, S., & Kotz, S. A. (2008). An ERP investigation on the temporal dynamics of emotional prosody and emotional semantics in pseudo- and lexical-sentence context. Brain and Language, 105(1), 59–69. https://doi.org/10.1016/j.bandl.2007.11.005
Pavlenko, A. (2008). Emotion and emotion-laden words in the bilingual lexicon. Bilingualism: Language and Cognition, 11(2), 147–164. https://doi.org/10.1017/S1366728908003283
Peelle, J. E. (2018). Listening effort: How the cognitive consequences of acoustic challenge are reflected in brain and behavior. Ear and Hearing, 39(2), 204–214. https://doi.org/10.1097/AUD.0000000000000494
Pinheiro, A. P., Dias, M., Pedrosa, J., & Soares, A. P. (2017). Minho affective sentences (MAS): Probing the roles of sex, mood, and empathy in affective ratings of verbal stimuli. Behavior Research Methods, 49(2), 698–716. https://doi.org/10.3758/s13428-016-0726-0
R Core Team (2020). R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, Vienna. Retrieved from: https://www.R-project.org/ (Version 4.0.3). Accessed 3 April 2022
Russell, J. A. (2003). Core affect and the psychological construction of emotion. Psychological Review, 110(1), 145–172. https://doi.org/10.1037/0033-295x.110.1.145
Schmidtke, D. S., Schroder, T., Jacobs, A. M., & Conrad, M. (2014). ANGST: Affective norms for German sentiment terms, derived from the affective norms for English words. Behavior Research Methods, 46(4), 1108–1118. https://doi.org/10.3758/s13428-013-0426-y
Schreuder, E., van Erp, J., Toet, A., & Kallen, V. L. (2016). Emotional responses to multisensory environmental stimuli: A conceptual framework and literature review. SAGE Open, 6(1), 1–19. https://doi.org/10.1177/2158244016630591
Soares, A. P., Pinheiro, A. P., Costa, A., Frade, C. S., Comesaña, M., & Pureza, R. (2013). Affective auditory stimuli: Adaptation of the International Affective Digitized sounds (IADS-2) for European Portuguese. Behavior Research Methods, 45, 1168–1181. https://doi.org/10.3758/s13428-012-0310-1
Song, Y., & Zhang, R. (2012). Revisiting the distinctive standards of Chinese adjectives. Journal of Hebei University of Science and Technology (Social Sciences), 12(2), 72–77. https://doi.org/10.3969/j.issn.1671-1653.2012.02.012
Stadthagen-Gonzalez, H., Imbault, C., Pérez Sánchez, M. A., & Brysbaert, M. (2017). Norms of valence and arousal for 14,031 Spanish words. Behavior Research Methods, 49, 111–123. https://doi.org/10.3758/s13428-015-0700-2
Steiger, J. H. (1980). Tests for comparing elements of a correlation matrix. Psychological Bulletin, 87(2), 245–251. https://doi.org/10.1037/0033-2909.87.2.245
Tanaka, A., Koizumi, A., Imai, H., Hiramatsu, S., Hiramoto, E., & de Gelder, B. (2010). I feel your voice: Cultural differences in the multisensory perception of emotion. Psychological Science, 21(9), 1259–1262. https://doi.org/10.1177/0956797610380698
Tang, E., Gong, J., Guan, J., & Ding, H. (2022). Neutral speech prosody imposes perceptive alterations on semantic emotion: Evidence from a large-scale affective rating experiment. Chinese Journal of Phonetics, (02), 53–68. [汤恩泽, 龚婕, 管晶晶, 丁红卫. (2022). 中性语音韵律影响语义情绪感知: 来自大规模情感评价实验的证据. 中国语音学报, (02), 53–68.].
Tucker, B. V., Brenner, D., Danielson, D. K., Kelley, M. C., Nenadić, F., & Sims, M. (2019). The massive auditory lexical decision (MALD) database. Behavior Research Methods, 51, 1187–1204. https://doi.org/10.3758/s13428-018-1056-1
Vaiouli, P., Panteli, M., & Panayiotou, G. (2021). Affective and psycholinguistic norms of Greek words: Manipulating their affective or psycho-linguistic dimensions. Current Psychology, 42, 10299–10309. https://doi.org/10.1007/s12144-021-02329-8
Vélez-Uribe, I., & Rosselli, M. (2017). The auditory and visual appraisal of emotion-related words in Spanish–English bilinguals. Bilingualism: Language and Cognition, 22(1), 30–46. https://doi.org/10.1017/s1366728917000517
Verheyen, S., De Deyne, S., Linsen, S., & Storms, G. (2020). Lexicosemantic, affective, and distributional norms for 1,000 Dutch adjectives. Behavior Research Methods, 52(3), 1108–1121. https://doi.org/10.3758/s13428-019-01303-4
Wang, Y., Zhou, L., & Luo, Y. (2008). The pilot establishment and evaluation of Chinese affective words system. Chinese Mental Health Journal, 22(8), 608–612. https://doi.org/10.3321/j.issn:1000-6729.2008.08.014
Warriner, A. B., Kuperman, V., & Brysbaert, M. (2013). Norms of valence, arousal, and dominance for 13,915 English lemmas. Behavior Research Methods, 45(4), 1191–1207. https://doi.org/10.3758/s13428-012-0314-x
Williams, C., & Bever, T. (2010). Chinese character decoding: A semantic bias? Reading and Writing, 23(5), 589–605. https://doi.org/10.1007/s11145-010-9228-0
Wu, C., Zhang, J., & Yuan, Z. (2021). Exploring affective priming effect of emotion-label words and emotion-laden words: An event-related potential study. Brain Sciences, 11(5). https://doi.org/10.3390/brainsci11050553
Xu, X., & Tao, J. (2003). The study of affective word categorization in Chinese. Paper presented at the 1st Chinese Conference on Affective Computing and Intelligent Interaction. [许小颖, 陶建华. (2003). 汉语情感系统中情感划分的研究. 第一届中国情感计算及智能交互学术会议论文集.]
Xu, X., Li, J., & Chen, H. (2022). Valence and arousal ratings for 11,310 simplified Chinese words. Behavior Research Methods, 54, 26–41. https://doi.org/10.3758/s13428-021-01607-4
Xu, X., Li, J., & Guo, S. (2021). Age of acquisition ratings for 19,716 simplified Chinese words. Behavior Research Methods, 53, 558–573. https://doi.org/10.3758/s13428-020-01455-8
Yang, W., Makita, K., Nakao, T., Kanayama, N., Machizawa, M. G., Sasaoka, T., & Miyatani, M. (2018). Affective auditory stimulus database: An expanded version of the International Affective digitized sounds (IADS-E). Behavior Research Methods, 50(4), 1415–1429. https://doi.org/10.3758/s13428-018-1027-6
Yao, Z., Wu, J., Zhang, Y., & Wang, Z. (2017). Norms of valence, arousal, concreteness, familiarity, imageability, and context availability for 1,100 Chinese words. Behavior Research Methods, 49(4), 1374–1385. https://doi.org/10.3758/s13428-016-0793-2
Yee, L. T. (2017). Valence, arousal, familiarity, concreteness, and imageability ratings for 292 two-character Chinese nouns in cantonese speakers in Hong Kong. PLoS ONE, 12(3). https://doi.org/10.1371/journal.pone.0174569
Zhong, J., & Qian, M. (2005). A study of development and validation of Chinese mood adjective check list. Chinese Journal of Clinical Psychology, 13(1), 8–13. [钟杰, 钱铭怡. (2005). 中文情绪形容词检测表的编制与信效度研究. 中国临床心理学杂志, 13 (1), 8–13.]
Acknowledgements
We thank the support and participation of research assistants, including Jingyi Wu, Jiaqi Zhang, Luyao Jiang, Zhuorui Gao, Leqi Zhou, Yi Lin, Minyue Zhang, Yu Chen. We are also grateful for all suggestions provided by experts throughout our experiments.
Funding
This work was supported by grants from Major Program of National Social Science Foundation of China (Grant number: 18ZDA293) and a research funding from SONOVA.
Author information
Authors and Affiliations
Contributions
Conceptualization, Data Curation, Formal Analysis, Investigation, Methodology, Visualization, Writing - Original Draft [Enze Tang]. Investigation, Material Preparation [Xinran Fan, Ruomei Fang]. Investigation [Yuhan Zhang, Jie Gong]. Conceptualization [Jingjing Guan]. Conceptualization, Funding Acquisition, Supervision, Writing - Review & Editing [Hongwei Ding].
Corresponding author
Ethics declarations
Ethics approval
This research was approved by the Ethics Committee of School of Foreign Languages, Shanghai Jiao Tong University (Ethics approval number: 2006S12002).
Informed consent
An informed consent was obtained from all participants before the formal experiment.
Competing interests
Dr. Jingjing Guan is employed by the company Sonova China, Shanghai. The remaining authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interests.
Additional information
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Tang, E., Fan, X., Fang, R. et al. Not perceptually equivalent in semantic emotion across visual and auditory modalities: cross-modal affective norms of two-character Chinese emotion-label words. Curr Psychol 43, 15308–15327 (2024). https://doi.org/10.1007/s12144-023-05476-2
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s12144-023-05476-2