Skip to main content
SpringerLink
Log in

Emotions by Ear and by Eye

  • Chapter
  • First Online:
Integrating Face and Voice in Person Perception

Abstract

Multisensory integration must stand out among the fields of research that have witnessed one of the most impressive explosions of interest this last decade, at least as measured by published papers and meetings. From a highly specialized niche occupation multisensory research has become a mainstream scientific interest in a very short time span. One of these new areas of multisensory research is emotion. Since our first exploration of this phenomenon [de Gelder Böcker, Tuomainen, Hensen, & Vroomen Neuroscience Letters 260(2):133–136, 1999], a number of studies have appeared and they have used a wide variety of behavioral, neuropsychological, and neuroimaging methods.

The goal of this chapter is fourfold. First, we review the research on audiovisual perception of emotional signals from the face and the voice. In the next section we discuss some outstanding methodological and theoretical issues followed by a report and comment on integrating emotional information provided by the voice and whole body expressions. We also include some recent work on music. Finally, we discuss findings about abnormal affective audiovisual integration in schizophrenia and in autism.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 169.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 219.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 219.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  • Alsius, A., Navarra, J., Campbell, R., & Soto-Faraco, S. (2005). Audiovisual integration of speech falters under high attention demands. Current Biology, 15(9), 839–843.

    Article  PubMed  CAS  Google Scholar 

  • Banse, R., & Scherer, K. R. (1996). Acoustic profiles in vocal emotion expression. Journal of Personality and Social Psychology, 70(3), 614–636.

    Article  PubMed  CAS  Google Scholar 

  • Bertelson, P., & de Gelder, B. (2004). The psychology of multimodal perception. In C. Spence & J. Driver (Eds.), Crossmodal space and crossmodal attention (pp. 151–177). Oxford: Oxford University Press.

    Google Scholar 

  • Borod, J. C., Martin, C. C., Alpert, M., Brozgold, A., & Welkowitz, J. (1993). Perception of facial emotion in schizophrenic and right brain-damaged patients. The Journal of Nervous and Mental Disease, 181(8), 494–502.

    Article  PubMed  CAS  Google Scholar 

  • Bostanov, V., & Kotchoubey, B. (2004). Recognition of affective prosody: Continuous wavelet measures of event-related brain potentials to emotional exclamations. Psychophysiology, 41(2), 259–268.

    Article  PubMed  Google Scholar 

  • Calvert, G. A., & Thesen, T. (2004). Multisensory integration: Methodological approaches and emerging principles in the human brain. Journal of Physiology, Paris, 98(1–3), 191–205.

    Article  PubMed  Google Scholar 

  • Cheesman, J., & Merikle, P. M. (1986). Distinguishing conscious from unconscious perceptual processes. Canadian Journal of Psychology, 40(4), 343–367.

    Article  PubMed  CAS  Google Scholar 

  • de Gelder, B. (2000). More to seeing than meets the eye. Science, 289(5482), 1148–1149.

    Article  PubMed  Google Scholar 

  • de Gelder, B., & Bertelson, P. (2003). Multisensory integration, perception and ecological validity. Trends in Cognitive Sciences, 7(10), 460–467.

    Article  PubMed  Google Scholar 

  • de Gelder, B., Böcker, K. B., Tuomainen, J., Hensen, M., & Vroomen, J. (1999). The combined perception of emotion from voice and face: Early interaction revealed by human electric brain responses. Neuroscience Letters, 260(2), 133–136.

    Article  PubMed  Google Scholar 

  • de Gelder, B., Morris, J. S., & Dolan, R. J. (2005). Unconscious fear influences emotional awareness of faces and voices. Proceedings of the National Academy of Sciences of the United States of America, 102(51), 18682–18687.

    Article  PubMed  Google Scholar 

  • de Gelder, B., Pourtois, G., & Weiskrantz, L. (2002). Fear recognition in the voice is modulated by unconsciously recognized facial expressions but not by unconsciously recognized affective pictures. Proceedings of the National Academy of Sciences of the United States of America, 99(6), 4121–4126.

    Article  PubMed  Google Scholar 

  • de Gelder, B., & Van den Stock, J. (2011). Real faces, real emotions: Perceiving facial expressions in naturalistic contexts of voices, bodies and scenes. In A. J. Calder, G. Rhodes, M. H. Johnson, & J. V. Haxby (Eds.), The Oxford handbook of face perception (pp. 535–550). New York: Oxford University Press.

    Google Scholar 

  • de Gelder, B., Van den Stock, J., Meeren, H. K., Sinke, C. B., Kret, M. E., & Tamietto, M. (2010). Standing up for the body. Recent progress in uncovering the networks involved in processing bodies and bodily expressions. Neuroscience and Biobehavioral Reviews, 34(4), 513–527.

    Article  PubMed  Google Scholar 

  • de Gelder, B., & Vroomen, J. (2000). The perception of emotions by ear and by eye. Cognition and Emotion, 14(3), 289–311.

    Article  Google Scholar 

  • de Gelder, B., Vroomen, J., de Jong, S. J., Masthoff, E. D., Trompenaars, F. J., & Hodiamont, P. (2005). Multisensory integration of emotional faces and voices in schizophrenics. Schizophrenia Research, 72(2–3), 195–203.

    Article  PubMed  Google Scholar 

  • de Gelder, B., Vroomen, J., & Pourtois, G. (2004). Multisensory perception of emotion, its time course and its neural basis. In G. Calvert, C. Spence, & B. E. Stein (Eds.), Handbook of multisensory processes (pp. 581–596). Cambridge, MA: MIT.

    Google Scholar 

  • de Gelder, B., Vroomen, J., & Teunisse, J. P. (1995). Hearing smiles and seeing cries. The bimodal perception of emotion. Bulletin of the Psychonomic Society, 29, 309.

    Google Scholar 

  • de Jong, J. J., Hodiamont, P. P., & de Gelder, B. (2010). Modality-specific attention and multisensory integration of emotions in schizophrenia: Reduced regulatory effects. Schizophrenia Research, 122(1–3), 136–143.

    Article  PubMed  Google Scholar 

  • de Jong, J. J., Hodiamont, P. P., Van den Stock, J., & de Gelder, B. (2009). Audiovisual emotion recognition in schizophrenia: Reduced integration of facial and vocal affect. Schizophrenia Research, 107(2–3), 286–293.

    Article  PubMed  Google Scholar 

  • Dolan, R. J., Morris, J. S., & de Gelder, B. (2001). Crossmodal binding of fear in voice and face. Proceedings of the National Academy of Sciences of the United States of America, 98(17), 10006–10010.

    Article  PubMed  CAS  Google Scholar 

  • Ekman, P., & Friesen, W. V. (1976). Pictures of facial affects. Palo Alto: Consulting Psychologists Press.

    Google Scholar 

  • Esteves, F., & Öhman, A. (1993). Masking the face: Recognition of emotional facial expressions as a function of the parameters of backward masking. Scandinavian Journal of Psychology, 34(1), 1–18.

    Article  PubMed  CAS  Google Scholar 

  • Ethofer, T., Anders, S., Erb, M., Droll, C., Royen, L., Saur, R., et al. (2006). Impact of voice on emotional judgment of faces: An event-related fMRI study. Human Brain Mapping, 27(9), 707–714.

    Article  PubMed  Google Scholar 

  • Feinberg, T. E., Rifkin, A., Schaffer, C., & Walker, E. (1986). Facial discrimination and emotional recognition in schizophrenia and affective disorders. Archives of General Psychiatry, 43(3), 276–279.

    Article  PubMed  CAS  Google Scholar 

  • Fernberger, S. W. (1928). False suggestion and the Piderit model. The American Journal of Psychology, 40, 562–568.

    Article  Google Scholar 

  • George, M. S., Parekh, P. I., Rosinsky, N., Ketter, T. A., Kimbrell, T. A., Heilman, K. M., et al. (1996). Understanding emotional prosody activates right hemisphere regions. Archives of Neurology, 53(7), 665–670.

    Article  PubMed  CAS  Google Scholar 

  • Ghazanfar, A. A., Chandrasekaran, C., & Logothetis, N. K. (2008). Interactions between the superior temporal sulcus and auditory cortex mediate dynamic face/voice integration in rhesus monkeys. Journal of Neuroscience, 28(17), 4457–4469.

    Article  PubMed  CAS  Google Scholar 

  • Ghazanfar, A. A., & Logothetis, N. K. (2003). Neuroperception: Facial expressions linked to monkey calls. Nature, 423(6943), 937–938.

    Article  PubMed  CAS  Google Scholar 

  • Ghazanfar, A. A., Maier, J. X., Hoffman, K. L., & Logothetis, N. K. (2005). Multisensory integration of dynamic faces and voices in rhesus monkey auditory cortex. Journal of Neuroscience, 25(20), 5004–5012.

    Article  PubMed  CAS  Google Scholar 

  • Ghazanfar, A. A., & Santos, L. R. (2004). Primate brains in the wild: The sensory bases for social interactions. Nature Reviews. Neuroscience, 5(8), 603–616.

    Article  PubMed  CAS  Google Scholar 

  • Goydke, K. N., Altenmuller, E., Moller, J., & Munte, T. F. (2004). Changes in emotional tone and instrumental timbre are reflected by the mismatch negativity. Cognitive Brain Research, 21(3), 351–359.

    Article  PubMed  Google Scholar 

  • Grandjean, D., Sander, D., Pourtois, G., Schwartz, S., Seghier, M. L., Scherer, K. R., et al. (2005). The voices of wrath: Brain responses to angry prosody in meaningless speech. Nature Neuroscience, 8(2), 145–146.

    Article  PubMed  CAS  Google Scholar 

  • Gur, R. E., McGrath, C., Chan, R. M., Schroeder, L., Turner, T., Turetsky, B. I., et al. (2002). An fMRI study of facial emotion processing in patients with schizophrenia. The American Journal of Psychiatry, 159(12), 1992–1999.

    Article  PubMed  Google Scholar 

  • Heimberg, C., Gur, R. E., Erwin, R. J., Shtasel, D. L., & Gur, R. C. (1992). Facial emotion discrimination: III. Behavioral findings in schizophrenia. Psychiatry Research, 42(3), 253–265.

    Article  PubMed  CAS  Google Scholar 

  • Kee, K. S., Horan, W. P., Wynn, J. K., Mintz, J., & Green, M. F. (2006). An analysis of categorical perception of facial emotion in schizophrenia. Schizophrenia Research, 87(1–3), 228–237.

    Article  PubMed  Google Scholar 

  • Kohler, C. G., Turner, T. H., Bilker, W. B., Brensinger, C. M., Siegel, S. J., Kanes, S. J., et al. (2003). Facial emotion recognition in schizophrenia: Intensity effects and error pattern. The American Journal of Psychiatry, 160(10), 1768–1774.

    Article  PubMed  Google Scholar 

  • Kraepelin, E. (1919). Dementia praecox and paraphrenia. Chicago: Medical Book Co.

    Google Scholar 

  • Kreifelts, B., Ethofer, T., Grodd, W., Erb, M., & Wildgruber, D. (2007). Audiovisual integration of emotional signals in voice and face: An event-related fMRI study. NeuroImage, 37(4), 1445–1456.

    Article  PubMed  Google Scholar 

  • Kreifelts, B., Ethofer, T., Huberle, E., Grodd, W., & Wildgruber, D. (2010). Association of trait emotional intelligence and individual fMRI-activation patterns during the perception of social signals from voice and face. Human Brain Mapping, 31(7), 979–991.

    Article  PubMed  Google Scholar 

  • Leitman, D. I., Loughead, J., Wolf, D. H., Ruparel, K., Kohler, C. G., Elliott, M. A., et al. (2008). Abnormal superior temporal connectivity during fear perception in schizophrenia. Schizophrenia Bulletin, 34(4), 673–678.

    Article  PubMed  Google Scholar 

  • MacLeod, C. M., & MacDonald, P. A. (2000). Interdimensional interference in the Stroop effect: Uncovering the cognitive and neural anatomy of attention. Trends in Cognitive Science, 4(10), 383–391.

    Article  Google Scholar 

  • Magnée, M. J., de Gelder, B., van Engeland, H., & Kemner, C. (2007). Facial electromyographic responses to emotional information from faces and voices in individuals with pervasive developmental disorder. Journal of Child Psychology and Psychiatry, 48(11), 1122–1130.

    Article  PubMed  Google Scholar 

  • Magnée, M. J., de Gelder, B., van Engeland, H., & Kemner, C. (2008). Atypical processing of fearful face–voice pairs in pervasive developmental disorder: An ERP Study. Clinical Neurophysiology, 119(9), 2004–2010.

    Article  Google Scholar 

  • Magnée, M. J. C. M., de Gelder, B., van Engeland, H., & Kemner, C. (2011). Multisensory integration in autism spectrum disorder: Critical influence of attention and noise. PLoS One, 6, e24196.

    Article  PubMed  Google Scholar 

  • Mandal, M. K., Pandey, R., & Prasad, A. B. (1998). Facial expressions of emotions and schizophrenia: A review. Schizophrenia Bulletin, 24(3), 399–412.

    Article  PubMed  CAS  Google Scholar 

  • Massaro, D. W., & Egan, P. B. (1996). Perceiving affect from the voice and the face. Psychonomic Bulletin and Review, 3, 215–221.

    Article  Google Scholar 

  • McGurk, H., & MacDonald, J. (1976). Hearing lips and seeing voices. Nature, 264(5588), 746–748.

    Article  PubMed  CAS  Google Scholar 

  • Meeren, H. K., van Heijnsbergen, C. C., & de Gelder, B. (2005). Rapid perceptual integration of facial expression and emotional body language. Proceedings of the National Academy of Sciences of the United States of America, 102(45), 16518–16523.

    Article  PubMed  CAS  Google Scholar 

  • Michalopoulou, P. G., Surguladze, S., Morley, L. A., Giampietro, V. P., Murray, R. M., & Shergill, S. S. (2008). Facial fear processing and psychotic symptoms in schizophrenia: Functional magnetic resonance imaging study. The British Journal of Psychiatry, 192(3), 191–196.

    Article  PubMed  Google Scholar 

  • Panksepp, J. (1998). Affective neuroscience: The foundation of human and animal emotions. New York: Oxford University Press.

    Google Scholar 

  • Panksepp, J. (2005). Psychology. Beyond a joke: From animal laughter to human joy? Science, 308(5718), 62–63.

    Article  PubMed  CAS  Google Scholar 

  • Parr, L. A. (2004). Perceptual biases for multimodal cues in chimpanzee (Pan troglodytes) affect recognition. Animal Cognition, 7(3), 171–178.

    Article  PubMed  Google Scholar 

  • Phillips, M. L., Williams, L., Senior, C., Bullmore, E. T., Brammer, M. J., Andrew, C., et al. (1999). A differential neural response to threatening and non-threatening negative facial expressions in paranoid and non-paranoid schizophrenics. Psychiatry Research, 92(1), 11–31.

    Article  PubMed  CAS  Google Scholar 

  • Pinkham, A. E., Hopfinger, J. B., Ruparel, K., & Penn, D. L. (2008). An investigation of the relationship between activation of a social cognitive neural network and social functioning. Schizophrenia Bulletin, 34(4), 688–697.

    Article  PubMed  Google Scholar 

  • Pourtois, G., de Gelder, B., Bol, A., & Crommelinck, M. (2005). Perception of facial expressions and voices and of their combination in the human brain. Cortex, 41(1), 49–59.

    Article  PubMed  Google Scholar 

  • Pourtois, G., de Gelder, B., Vroomen, J., Rossion, B., & Crommelinck, M. (2000). The time-course of intermodal binding between seeing and hearing affective information. NeuroReport, 11(6), 1329–1333.

    Article  PubMed  CAS  Google Scholar 

  • Pourtois, G., Debatisse, D., Despland, P. A., & de Gelder, B. (2002). Facial expressions modulate the time course of long latency auditory brain potentials. Cognitive Brain Research, 14(1), 99–105.

    Article  PubMed  Google Scholar 

  • Remedios, R., Logothetis, N. K., & Kayser, C. (2009). Monkey drumming reveals common networks for perceiving vocal and nonvocal communication sounds. Proceedings of the National Academy of Sciences of the United States of America, 106(42), 18010–18015.

    Article  PubMed  CAS  Google Scholar 

  • Ross, E. D. (2000). Affective prosody and the aprosodias. In M. M. Mesulam (Ed.), Principles of behavioral and cognitive neurology (2nd ed.). London: Oxford University Press.

    Google Scholar 

  • Ross, L. A., Saint-Amour, D., Leavitt, V. M., Molholm, S., Javitt, D. C., & Foxe, J. J. (2007). Impaired multisensory processing in schizophrenia: Deficits in the visual enhancement of speech comprehension under noisy environmental conditions. Schizophrenia Research, 97(1–3), 173–183.

    Article  PubMed  Google Scholar 

  • Scherer, K. R., Banse, R., Wallbott, H. G., & Goldbeck, T. (1991). Vocal cues in emotion encoding and decoding. Motivation and Emotion, 15(2), 123–148.

    Article  Google Scholar 

  • Stein, B. E., & Stanford, T. R. (2008). Multisensory integration: Current issues from the perspective of the single neuron. Nature Reviews. Neuroscience, 9(4), 255–266.

    Article  PubMed  CAS  Google Scholar 

  • Stienen, B. M. C., & de Gelder, B. (2011). Fear detection and visual awareness in perceiving bodily expressions. Emotion, 11(5), 1182–1189.

    Article  PubMed  Google Scholar 

  • Stienen, B.M.C., Tanaka, A., de Gelder, B. (2011). Emotional voice and emotional body postures influence each other independently of visual awareness. PLoS ONE, 6(10): e25517.

    Article  PubMed  Google Scholar 

  • Sugihara, T., Diltz, M. D., Averbeck, B. B., & Romanski, L. M. (2006). Integration of auditory and visual communication information in the primate ventrolateral prefrontal cortex. Journal of Neuroscience, 26(43), 11138–11147.

    Article  PubMed  CAS  Google Scholar 

  • Tamietto, M., & de Gelder, B. (2010). Neural bases of the non-conscious perception of emotional signals. Nature Reviews. Neuroscience, 11(10), 697–709.

    Article  PubMed  CAS  Google Scholar 

  • Van den Stock, J., de Jong, J. J., Hodiamont, P. P. G., & de Gelder, B. (2011). Perceiving emotions from bodily expressions and multisensory integration of emotion cues in schizophrenia. Social Neuroscience, 6(5–6), 537–547.

    Article  PubMed  Google Scholar 

  • Van den Stock, J., Grèzes, J., & de Gelder, B. (2008). Human and animal sounds influence recognition of body language. Brain Research, 1242, 185–190.

    Article  PubMed  Google Scholar 

  • Van den Stock, J., Peretz, I., Grèzes, J., & de Gelder, B. (2009). Instrumental music influences recognition of emotional body language. Brain Topography, 21(3–4), 216–220.

    Article  PubMed  Google Scholar 

  • Van den Stock, J., Righart, R., & de Gelder, B. (2007). Body expressions influence recognition of emotions in the face and voice. Emotion, 7(3), 487–494.

    Article  PubMed  Google Scholar 

  • Vroomen, J., Driver, J., & de Gelder, B. (2001). Is cross-modal integration of emotional expressions independent of attentional resources? Cognitive, Affective, & Behavioral Neuroscience, 1(4), 382–387.

    Article  CAS  Google Scholar 

  • Wolwer, W., Streit, M., Polzer, U., & Gaebel, W. (1996). Facial affect recognition in the course of schizophrenia. European Archives of Psychiatry and Clinical Neuroscience, 246(3), 165–170.

    Article  PubMed  CAS  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Cognitive and Affective Neuroscience Laboratory, Tilburg University, Room P 511, 90153, 5000 LE, Tilburg, The Netherlands

    Beatrice de Gelder, Bernard M. C. Stienen & Jan Van den Stock

Authors
  1. Beatrice de Gelder
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Bernard M. C. Stienen
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jan Van den Stock
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Beatrice de Gelder .

Editor information

Editors and Affiliations

  1. Dept. Psychology, Voice Neurocognition Lab., University of Glasgow, Glasgow, Glasgow, G12 8QB, United Kingdom

    Pascal Belin

  2. Brugmann University Hospital, Brussels, B-1020, Belgium

    Salvatore Campanella

  3. University of Tubingen, Tubingen, 72074, Germany

    Thomas Ethofer

Rights and permissions

Reprints and permissions

Copyright information

© 2013 Springer Science+Business Media New York

About this chapter

Cite this chapter

de Gelder, B., Stienen, B.M.C., Van den Stock, J. (2013). Emotions by Ear and by Eye. In: Belin, P., Campanella, S., Ethofer, T. (eds) Integrating Face and Voice in Person Perception. Springer, New York, NY. https://doi.org/10.1007/978-1-4614-3585-3_13

Download citation

Publish with us

Policies and ethics

Search

Navigation