Skip to main content
SpringerLink
Log in

Audiovisual Expression of Emotions in Communication

  • Chapter
  • First Online:
Book cover Sensing Emotions

Part of the book series: Philips Research Book Series ((PRBS,volume 12))

  • 1369 Accesses

Abstract

Non-verbal cues may reveal a lot about the emotional state of a user. However, the way these expressions of emotion are often studied in the scientific literature may be rather different from the actual expressions in reality, which are dynamic, spontaneous and potentially multimodal. In this chapter, we systematically compare posed and spontaneous emotional expressions, which were collected using an experimental language-based induction method in which participants were asked to produce sentences with an increasingly emotional content. It was found that spontaneous positive and negative expressions lead to more positive and negative self-reported emotion scores than the posed ones. Interestingly, however, perception studies revealed that judges rate the posed dynamic facial expressions as significantly stronger than the spontaneous ones. Finally, it was studied whether better acting skills lead to more realistic expressions, which turned out not to be the case.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 109.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    English translation: “On the other hand, if gesture and the expression of the face are out of harmony with the speech, if we look cheerful when our words are sad, or shake our heads when making a positive assertion, our words will not only lack weight, but will fail to carry conviction” (Quintilianus, 1958, 11.3.67).

  2. 2.

    Here and elsewhere, we report on the normal degrees of freedom and error values after such a correction, for the sake of readability.

References

  • Adolphs, R.: Neural systems for recognizing emotion. Curr. Opin. Neurobiol. 12, 169–177 (2002)

    Article  Google Scholar 

  • Bachorowski, J.: Vocal expression and perception of emotion. Curr. Dir. Psychol. Sci. 8(2), 53–57 (1999)

    Article  Google Scholar 

  • Barkhuysen, P., Krahmer, E., Swerts, M.: Cross-modal and incremental perception of audiovisual cues to emotional speech. Lang. Speech, 53, 3–30 (2010, in press)

    Google Scholar 

  • Beukenboom, C., Semin, G.: How mood turns on language. J. Exp. Soc. Psychol. 42, 553–566 (2006)

    Article  Google Scholar 

  • Buchwald, A., Strack, J., Coyne, J.: Demand characteristics and the Velten mood induction procedure. J. Consult. Clin. Psychol. 49, 478–479 (1981)

    Google Scholar 

  • Cohn, J., Schmidt, K.: The timing of facial motion in posed and spontaneous smiles. Int. J. Wavelets Multiresolut. Inf. Proc. 2, 1–12 (2004)

    Article  Google Scholar 

  • Darwin, C.: The Expression of the Emotions in Man and Animals, 3rd edn. Oxford University Press, New York, NY [1998] (1872)

    Book  Google Scholar 

  • Del Giudice, M. and Colle, L. (2007). Differences between children and adults in the recognition of enjoyment smiles. Developmental Psychology, 43, 796–803

    Article  Google Scholar 

  • Dimberg, U., Thunberg, M., Elmehed, K.: Unconscious facial reactions to emotional facial expressions. Psychol. Sci. 11, 86–89 (2000)

    Article  Google Scholar 

  • Drahota, A., Costalla, A., Reddya, V.: The vocal communication of different kinds of smile. Speech Commun. 50, 278–287 (2008)

    Google Scholar 

  • Ekman, P.: Emotion in the Human Face. Pergamon Press, New York, NY (1972)

    Google Scholar 

  • Ekman, P., Friesen, W., O‘Sullivan, M.: Smiles when lying. J. Personal. Soc. Psychol. 54, 414–420 (1988)

    Article  Google Scholar 

  • Elfenbein, H., Ambday, N.: On the universality and cultural specificity of emotion recognition: a meta-analysis. Psychol. Bull. 128, 203–235 (2002)

    Article  Google Scholar 

  • Enos, F., Hirschberg, J.: A framework for eliciting emotional speech: capitalizing on the actor’s process. In: L. Devillers, J.-C. Martin, R. Cowie, E. Douglas-Cowie and A. Batliner (Eds.) LREC 2006 Workshop on Corpora for Research on Emotion and Affect. Genova, Italy (2006)

    Google Scholar 

  • Feldman Barett, L., Lindquist, K., Gendron, M.: Language as context for the perception of emotion. Trends Cogn. Sci. 11(8), 327–332 (2007)

    Article  Google Scholar 

  • Finegan, J., Seligman, C.: In defense of the Velten mood induction procedure. Can. J. Behav. Sci. 27, 405–418 (1995)

    Google Scholar 

  • Frank, M., Ekman, P., Friesen, W.: Behavioral markers and recognizability of the smile of enjoyment. J. Personal. Soc. Psychol. 64, 83–93 (1993)

    Article  Google Scholar 

  • Fridlund, A.: The sociality of solitary smiles: effects of an implicit audience. J. Personal. Soc. Psychol. 60, 229–240 (1991)

    Article  Google Scholar 

  • Fridlund, A.: Human Facial Expression. An Evolutionary View. Academic, San Diego (1994)

    Google Scholar 

  • Galati, D., Scherer, K., Ricci-Bitti, P.: Voluntary facial expression of emotion: comparing congenitally blind with normal sighted encoders. J. Personal. Soc. Psychol. 73, 1363–1379 (1997)

    Article  Google Scholar 

  • de Gelder, B., Vroomen, J.: The perception of emotions by ear and by eye. Cogn. Emot. 14, 289–311 (2000)

    Article  Google Scholar 

  • Gosselin, P., Perron, M., Legault, M., Campanella, P.: Children’s and adults’ knowledge of the facial features distinguishing genuine from simulated smiles. J. Nonverbal Behav. 26, 83–108 (2002)

    Article  Google Scholar 

  • Hess, U., Kleck, R.: Differentiating emotion elicited and deliberate emotional facial expressions. Eur. J. Soc. Psychol. 24, 367–381 (1990)

    Article  Google Scholar 

  • Hess, U., Kleck, R.: The cues decoders use in attempting to differentiate emotion-elicited and posed facial expressions. Eur. J. Soc. Psychol. 20, 369–385 (1994)

    Article  Google Scholar 

  • Hess, U., Kappas, A., McHugo, G.J., Lanzetta, J., Kleck, R.: The facilitative effect of facial expression on the self-generation of emotion. Int. J. Psychophysiol. 12, 251–265 (1992)

    Google Scholar 

  • Horstmann, G.: Facial expressions of emotion: does the prototype represent central tendency, frequency of instantiation, or an ideal? Emotion 2, 297–305 (2002)

    Article  Google Scholar 

  • James, W.: What is an emotion? Mind 9, 190–204 (1884)

    Google Scholar 

  • Konijn, E.: Acting Emotions: Shaping Emotions on Stage. Amsterdam University Press, Amsterdam (2000)

    Google Scholar 

  • Kotchemidova, C.: From good cheer to "drive-by smiling": a social history of cheerfulness. J. Soc. Hist. 39(1), 5–37 (2005)

    Article  Google Scholar 

  • Krahmer, E., van Dorst, J., Ummelen, N.: Mood, persuasion and information presentation. Inf. Des. J. 12, 40–52 (2004)

    Google Scholar 

  • Krahmer, E., Schaafsma, J., Swerts, M., Vingerhoets, A. Nonverbal responses to social inclusion and exclusion. In: Proceedings of the International Conference on Spoken Language Processing (Interspeech 2008), ISCA, Brisbane, pp. 809–812 (2008)

    Google Scholar 

  • Mackie, D., Worth, L.: Processing deficits and the mediation of positive affect in persuasion. J. Personal. Soc. Psychol. 57, 27–40 (1989)

    Article  Google Scholar 

  • Marsella, S., Carnicke, S., Gratch, J., Okhmatovskaia, A., Rizzo, A.: An exploration of Delsarte’s structural acting system. In: Proceedings of the 6th International Conference on Intelligent Virtual Agents (IVA), Marina del Rey, CA (2006)

    Google Scholar 

  • Massaro, D., Egan, P.: Perceiving affect from the voice and the face. Psychon. Bull. Rev. 3, 215–221 (1996)

    Google Scholar 

  • Motley, M., Camden, C.: Facial expression of emotion: a comparison of posed expressions versus spontaneous expressions in an interpersonal communication setting. West. J. Speech Commun. 52, 1–22 (1988)

    Google Scholar 

  • Pantie, M., Patras, I.: Dynamics of facial expression: recognition of facial actions and their temporal segments from face profile image sequences. IEEE Trans. Syst. Man Cybern. B 36, 433–449 (2006)

    Article  Google Scholar 

  • Picard, R., Klein, J.: Computers that recognise and respond to user emotion: theoretical and practical implications. Interact. Comput. 14(2), 141–169 (2002)

    Article  Google Scholar 

  • Picard, R., Vyzas, E., Healy, J.: Toward machine emotional intelligence: analysis of affective physiological state. IEEE Trans. Pattern Anal. Mach. Intell. 23(10), 1175–1191 (2001)

    Article  Google Scholar 

  • Polivy, J., Doyle, C.: Laboratory induction of mood states through the reading of self-referent mood statements: affective changes or demand characteristics. J. Abnorm. Psychol. 89, 286–290 (1980)

    Article  Google Scholar 

  • Quintilianus, M.F.: Instituţia Oratoria (trans. in English by Butler, H.E.). Harvard University Press, Cambridge (1958)

    Google Scholar 

  • Reisenzein, R., Bördgen, S., Holtbernd, T., Matz, D.: Evidence for strong dissociation between emotion and facial displays: the case of surprise. J. Personal. Soc. Psychol. 91, 295–315 (2006)

    Google Scholar 

  • Rottenberg, J., Ray, R., Gross, J.: Emotion elicitation using films. In: Coan, J.A., Allen, J.J.B. (eds.) The Handbook of Emotion Elicitation and Assessment. Oxford University Press, London, 9–28. (2007)

    Google Scholar 

  • Russell, J., Bachorowski, J., Fernandez-Dols, J.: Facial and vocal expressions of emotion. Ann. Rev. Psychol. 54, 329–349 (2003)

    Article  Google Scholar 

  • Ruys, K., Stapel, D.: The secret life of emotions. Psychol. Sci. 19, 385–391 (2008)

    Article  Google Scholar 

  • Scherer, K.: Vocal communication of emotion: a review of research paradigms. Speech Commun. 40, 227–256 (2003)

    Article  MATH  Google Scholar 

  • Scherer, K., Ellgring, H.: Multimodal expression of emotion: affect programs or componential appraisal patterns? Emotion 7, 158–171 (2007)

    Article  Google Scholar 

  • Schmidt, K., Cohn, J.: Human facial expressions as adaptations: evolutionary questions in facial expressions. Yearb. Phys. Anthropol. 44, 3–24 (2002)

    Google Scholar 

  • Schmidt, K., Ambadar, Z., Cohn, J., Reed, L.: Movement differences between deliberate and spontaneous facial expressions: zygomaticus major action in smiling. J. Nonverbal Behav. 3, 37–52 (2006)

    Article  Google Scholar 

  • Schmidt, K., Cohn, J., Tian, Y.: Signal characteristics of spontaneous facial expressions: automatic movement in solitary and social smiles. Biol. Psychol. 65, 49–66 (2003)

    Article  Google Scholar 

  • Shahid, S., Krahmer, E., Swerts, M.: Emotional speech: comparing South-Asian and Caucasian speakers and observers. In: Proceedings of the Fourth Speech Prosody Conferеnсе, Campinas, Brazil (2008a)

    Google Scholar 

  • Shahid, S., Krahmer, E., Swerts, M.: Alone or together: exploring the effect of physical co-presence on the emotional expressions of game playing children across cultures. In: Markopoulus, P. et al. (eds.) Fun and Games. Lecture Notes in Computer Science LNCS, vol. 5294, pp. 94–105. Springer, Berlin/Heidelberg (2008b)

    Chapter  Google Scholar 

  • Shahid, S., Krahmer, E., Swerts, M., Melder, W.A., Neerinex, M.A. Exploring social and temporal dimensions of emotion induction using an adaptive affective mirror. In: Proceedings CHI 2009: Extended Abstracts, Boston, MA, pp. 3727–3732 (2009)

    Google Scholar 

  • Stapel, D., Koomеn, W.: When less is more. The consequences of affective primacy for subliminal priming effects. Personal. Soc. Psychol. Bull. 31, 1286–1295 (2005)

    Article  Google Scholar 

  • Stepper, S., Strack, F.: Proprioceptive determinants of emotional and non-emotional feelings. J. Personal. Soc. Psychol. 64, 210–22 (1993)

    Google Scholar 

  • Swerts, M., Krahmer, E.: Gender-related differences in the production and perception of emotion. In: Proceedings of the International Conference on Spoken Language Processing (Interspeech 2008), ISCA, Brisbane, pp. 334–337 (2008)

    Google Scholar 

  • Velten, E.: A laboratory task for induction of mood states. Behav. Res. Therару 6, 473–482 (1968)

    Article  Google Scholar 

  • Vogt, T., André, E.: Comparing feature sets for acted and spontaneous speech in view of automatic emotion recognition. In: Proceedings of IEEE International Conference on Multimedia & Expo (ICME 2005), Trento, Italy (2005)

    Google Scholar 

  • Walbott, H., Scherer, L.: Cues and channels in emotion recognition. J. Personal. Soc. Psychol. 51, 690–699 (1986)

    Article  Google Scholar 

  • Wagner, H., Lee, V.: Facial behavior alone and in the presence of others. In: Philippott, P., Feldman, R., Coats, E. (eds.) The social context of nonverbal behavior, pp. 262–286. Cambridge University Press, New York, NY (1999)

    Google Scholar 

  • Westermann, R., Spies, K., Stahl, G., Hesse, F.: Relative effectiveness and validity of mood induction procedures. Eur. J. Soc. Psychol. 26, 557–580 (1996)

    Article  Google Scholar 

Download references

Acknowledgements

We are grateful to Emmett Velten for supplying us with the sentences used in his original experiment, and for some useful discussion on acting and demand effects. We would like to thank Lennard van de Laar, Marieke Rikken, Suleman Shahid, Pashiera Barkhuysen and, especially, Janneke Wilting for their help with and discussions about the experiments. Thanks are due to Harold Miesen and Carel van Wijk for statistical advice, and to Diederik Stapel, Kirsten Ruys and Maarten Brouwers for useful comments on a previous version of this chapter.

Author information

Authors and Affiliations

  1. Faculty of Humanities, Tilburg centre for Cognition and Communication (TiCC), Tilburg University, Tilburg, The Netherlands

    Emiel Krahmer & Marc Swerts

Authors
  1. Emiel Krahmer
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Marc Swerts
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Emiel Krahmer .

Editor information

Editors and Affiliations

  1. Philips Research, High Tech Campus 34, Eindhoven, 5656 AE, Netherlands

    Joyce Westerink

  2. Philips Research, High Tech Campus 34, Eindhoven, 5656 AE, Netherlands

    Martijn Krans

  3. Philips Research, High Tech Campus 34, Eindhoven, 5656 AE, Netherlands

    Martin Ouwerkerk

Rights and permissions

Reprints and permissions

Copyright information

© 2010 Springer Science+Business Media B.V.

About this chapter

Cite this chapter

Krahmer, E., Swerts, M. (2010). Audiovisual Expression of Emotions in Communication. In: Westerink, J., Krans, M., Ouwerkerk, M. (eds) Sensing Emotions. Philips Research Book Series, vol 12. Springer, Dordrecht. https://doi.org/10.1007/978-90-481-3258-4_6

Download citation

  • DOI: https://doi.org/10.1007/978-90-481-3258-4_6

  • Published:

  • Publisher Name: Springer, Dordrecht

  • Print ISBN: 978-90-481-3257-7

  • Online ISBN: 978-90-481-3258-4

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation