Is there an advantage for recognizing multi-modal emotional stimuli?
Rent the article at a discountRent now
* Final gross prices may vary according to local VAT.Get Access
Emotions can be recognized whether conveyed by facial expressions, linguistic cues (semantics), or prosody (voice tone). However, few studies have empirically documented the extent to which multi-modal emotion perception differs from uni-modal emotion perception. Here, we tested whether emotion recognition is more accurate for multi-modal stimuli by presenting stimuli with different combinations of facial, semantic, and prosodic cues. Participants judged the emotion conveyed by short utterances in six channel conditions. Results indicated that emotion recognition is significantly better in response to multi-modal versus uni-modal stimuli. When stimuli contained only one emotional channel, recognition tended to be higher in the visual modality (i.e., facial expressions, semantic information conveyed by text) than in the auditory modality (prosody), although this pattern was not uniform across emotion categories. The advantage for multi-modal recognition may reflect the automatic integration of congruent emotional information across channels which enhances the accessibility of emotion-related knowledge in memory.
- Baenziger, T, Grandjean, D, Scherer, KR (2009) Emotion recognition from expressions in face, voice, and body. The Multimodal Emotion Recognition Test (MERT). Emotion 9: pp. 691-704 CrossRef
- Banse, R, Scherer, KR (1996) Acoustic profiles in vocal emotion expression. Journal of Personality and Social Psychology 3: pp. 614-636 CrossRef
- Borod, JC, Cicero, B, Obler, LK, Welkowitz, J, Erhan, HM, Santschi, C, Grunwald, IS, Agosti, RM, Whalen, JR (1998) Right hemisphere emotional perception. Evidence across multiple channels. Neuropsychology 12: pp. 446-458 CrossRef
- Borod, JC, Pick, LH, Hall, S, Sliwinski, M, Madigan, N, Obler, LK, Welkowitz, J, Canino, E, Erhan, HM, Goral, M, Morrison, C, Tabert, M (2000) Relationships among facial, prosodic, and lexical channels of emotional perceptual processing. Cognition and Emotion 14: pp. 193-211 CrossRef
- Bower, GH (1981) Mood and memory. American Psychologist 36: pp. 129-148 CrossRef
- Busso, C., Deng, Z., Yildirim, S., Bulut, M., Lee, C. M., Kazemzadeh, A., et al. (2004). Analysis of emotion recognition using facial expressions, speech and multimodal information. In Proceedings of ACM 6th International Conference on Multimodal Interfaces (ICMI 2004), State College, PA, 2004.
- Castro, SL, Lima, CF (2010) Recognizing emotions in spoken language: A validated set of Portuguese sentences and pseudosentences for research on emotional prosody. Behavior Research Methods 42: pp. 74-81 CrossRef
- Collignon, O, Girard, S, Gosselin, F, Roy, S, Saint-Amour, D, Lassonde, M, Lepore, F (2008) Audio-visual integration of emotion expression. Brain Research 1242: pp. 126-135 CrossRef
- De Silva, L. C., Miyasato, T., & Natatsu, R. (1997). Facial emotion recognition using multimodal information. In Proceedings of IEEE International Conference on Information, Communications and Signal Processing (ICICS’97), pp. 397–401.
- DeGelder, B, Bertelson, P (2003) Multisensory integration, perception and ecological validity. Trends in Cognitive Sciences 7: pp. 460-467 CrossRef
- DeGelder, B, Böcker, KBE, Tuomainen, J, Hensen, M, Vroomen, J (1999) The combined perception of emotion from voice and face: Early interaction revealed by human electric brain responses. Neuroscience Letters 260: pp. 133-136 CrossRef
- DeGelder, B, Vroomen, J (2000) The perception of emotions by ear and by eye. Cognition and Emotion 14: pp. 289-311 CrossRef
- Ekman, P (1992) An argument for basic emotions. Cognition and Emotion 6: pp. 169-200 CrossRef
- Ekman, P, Friesen, W (1976) Pictures of facial affect. Consulting Psychologist’s Press, Palo Alto, CA
- Elfenbein, HA, Ambady, N (2002) On the universality and cultural specificity of emotion recognition: A meta-analysis. Psychological Bulletin 128: pp. 203-235 CrossRef
- Etcoff, NL, Magee, JJ (1992) Categorical perception of facial expressions. Cognition 44: pp. 227-240 CrossRef
- Hawk, ST, van Kleef, GA, Fischer, AH, van der Schalk, J (2009) Worth a thousand words: Absolute and relative decodability of nonlinguistic affect vocalizations. Emotion 9: pp. 293-305 CrossRef
- Johnstone, T, Scherer, KR Vocal communication of emotion. In: Lewis, M, Haviland, J eds. (2000) Handbook of emotions. Guilford Press, New York, pp. 220-235
- Juslin, PN, Laukka, P (2003) Communication of emotions in vocal expression and music performance: Different channels, same code?. Psychological Bulletin 129: pp. 770-814 CrossRef
- Keppel, G (1991) Design and analysis: A researcher’s handbook. Prentice Hall, Englewood Cliffs, NJ
- Kotz, SA, Paulmann, S (2007) When emotional prosody and semantics dance cheek to cheek: ERP evidence. Brain Research 1151: pp. 107-118 CrossRef
- Kreifelts, B, Ethofer, T, Grodd, W, Erb, M, Wildgruber, D (2007) Audiovisual integration of emotional signals in voice and face: An event-related fMRI study. Neuroimage 37: pp. 1445-1456 CrossRef
- Kreifelts, B, Ethofer, T, Huberle, E, Grodd, W, Wildgruber, D (2010) Association of trait emotional intelligence and individual fMRI-activation patterns during the perception of social signals from voice and face. Human Brain Mapping 31: pp. 979-991 CrossRef
- Levitt, E. A. (1964). The relationship between abilities to express emotional meanings vocally and facially. In J. R. Davitz (Ed.), The communication of emotional meaning (pp. 87–100). New York: McGraw-Hill.
- Massaro, DW, Egan, PB (1996) Perceiving affect from the voice and the face. Psychonomic Bulletin Review 3: pp. 215-221 CrossRef
- Niedenthal, PM (2007) Embodying emotion. Science 316: pp. 1002-1005 CrossRef
- Niedenthal, PM, Halberstadt, JB (1995) The acquisition and structure of emotional response categories. The Psychology of Learning and Motivation 33: pp. 23-63 CrossRef
- Nowicki, S, Duke, M (1994) Individual differences in the nonverbal communication of affect. Journal of Nonverbal Behavior 18: pp. 9-36 CrossRef
- Paulmann, S, Pell, MD, Kotz, SA (2008) How aging affects the recognition of emotional speech. Brain and Language 104: pp. 262-269 CrossRef
- Pell, MD (2002) Evaluation of nonverbal emotion in face and voice: Some preliminary findings on a new battery of tests. Brain and Cognition 48: pp. 499-504
- Pell, MD (2005) Nonverbal emotion priming: evidence from the ‘facial affect decision task’. Journal of Nonverbal Behavior 29: pp. 45-73 CrossRef
- Pell, MD (2006) Cerebral mechanisms for understanding emotional prosody in speech. Brain and Language 96: pp. 221-234 CrossRef
- Pell, M. D., Jaywant, A., Monetta, L., & Kotz, S. A. (in press). Emotional speech processing: disentangling the effects of prosody and semantic cues. Cognition & Emotion. doi:10.1080/02699931.2010.516915.
- Pell, MD, Paulmann, S, Dara, C, Alasseri, A, Kotz, SA (2009) Factors in the recognition of vocally expressed emotions: A comparison of four languages. Journal of Phonetics 37: pp. 417-435 CrossRef
- Rosenthal, R, Hall, JA, DiMatteo, MR, Rogers, PL, Archer, D (1979) Sensitivity to nonverbal communication: The PONS test. John Hopkins University Press, Baltimore
- Russell, J, Lemay, G Emotion concepts. In: Lewis, M, Haviland-Jones, MJ eds. (2000) Handbook of emotion. Guilford Press, New York, pp. 491-503
- Schwartz, J-L, Robert-Ribes, J, Escudier, P Ten years after Summerfield: A taxonomy of models for audio-visual fusion in speech perception. In: Campbell, R eds. (1998) Hearing by eye: The psychology of lipreading. Lawrence Erlbaum Associcates, Hove, UK, pp. 3-51
- Welch, RB, Warren, DH (1980) Immediate perceptual response to intersensory discrepancy. Psychological Bulletin 88: pp. 638-667 CrossRef
- Is there an advantage for recognizing multi-modal emotional stimuli?
Motivation and Emotion
Volume 35, Issue 2 , pp 192-201
- Cover Date
- Print ISSN
- Online ISSN
- Springer US
- Additional Links
- Emotional prosody
- Emotional semantics
- Emotional facial expressions
- Industry Sectors