Combining Facial and Postural Expressions of Emotions in a Virtual Character

  • Céline Clavel
  • Justine Plessier
  • Jean-Claude Martin
  • Laurent Ach
  • Benoit Morel
Part of the Lecture Notes in Computer Science book series (LNCS, volume 5773)


Psychology suggests highly synchronized expressions of emotion across different modalities. Few experiments jointly studied the relative contribution of facial expression and body posture to the overall perception of emotion. Computational models for expressive virtual characters have to consider how such combinations will be perceived by users. This paper reports on two studies exploring how subjects perceived a virtual agent. The first study evaluates the contribution of the facial and postural expressions to the overall perception of basic emotion categories, as well as the valence and activation dimensions. The second study explores the impact of incongruent expressions on the perception of superposed emotions which are known to be frequent in everyday life. Our results suggest that the congruence of facial and bodily expression facilitates the recognition of emotion categories. Yet, judgments were mainly based on the emotion expressed in the face but were nevertheless affected by postures for the perception of the activation dimension.


evaluation of virtual agents affective interaction conversational and non-verbal behavior multimodal interaction with intelligent virtual agent 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Albrecht, I., Schröder, M., Haber, J., Seidel, H.-P.: Mixed feelings: Expression of non-basic emotions in a muscle-based talking head. Special issue of Journal of Virtual Reality on Language, Speech & Gesture 8(4) (2005)Google Scholar
  2. 2.
    Ballin, D., Gillies, M., Crabtree, B.: A Framework For Interpersonal Attitude And Non-Verbal Communication In Improvisational Visual Media Production. In: The First European Conference on Visual Media Production IEE, London (2004)Google Scholar
  3. 3.
    Bänziger, T., Scherer, K.: Using Actor Portrayals to Systematically Study Multimodal Emotion Expression: The GEMEP Corpus. In: Paiva, A.C.R., Prada, R., Picard, R.W. (eds.) ACII 2007. LNCS, vol. 4738, pp. 476–487. Springer, Heidelberg (2007)CrossRefGoogle Scholar
  4. 4.
    Bianchi-Berthouze, N., Kleinsmith, A.: A categorical approach to affective gesture recognition. Connection Science 15(4), 259–269 (2003)CrossRefGoogle Scholar
  5. 5.
    Buisine, S., Abrilian, S., Niewiadomski, R., Martin, J.C., Devillers, L., Pelachaud, C.: Perception of blended emotions: From video corpus to expressive agent. In: Gratch, J., Young, M., Aylett, R.S., Ballin, D., Olivier, P. (eds.) IVA 2006. LNCS (LNAI), vol. 4133, pp. 93–106. Springer, Heidelberg (2006)CrossRefGoogle Scholar
  6. 6.
    De Silva, R., Bianchi-Berthouze, N.: Modelling human affective postures: an information theoretic characterization of posture features. Computer Animation and Virtual Worlds 15(3-4), 269–276 (2004)CrossRefGoogle Scholar
  7. 7.
    Dittman, A.T.: The role of body movement in communication. In: Nonverbal behavior and communication. Lawrence Erlbaum, Mahwah (1987)Google Scholar
  8. 8.
    Ekman, P.: An argument for basic emotions. Cognition and Emotion 6, 169–200 (1992)CrossRefGoogle Scholar
  9. 9.
    Ekman, P., Friesen, W.: Manual for the facial action coding system. Consulting Psychology Press, Palo Alto (1978)Google Scholar
  10. 10.
    Ekman, P., Friesen, W.: A new pan-cultural facial expression of emotion. Motivation and Emotion 10, 159–168 (1986)CrossRefGoogle Scholar
  11. 11.
    Ekman, P., Friesen, W.V.: Nonverbal behavior and psychopathology. In: Friedman, R.J., Datz, M.M. (eds.) The psychology of depression: contemporary theory and research, pp. 203–232. Winston & Sons, Washington (1974)Google Scholar
  12. 12.
    Ekman, P., Friesen, W.V.: Unmasking the face. A guide to recognizing emotions from facial clues. Prentice-Hall Inc., Englewood Cliffs (1975)Google Scholar
  13. 13.
    Ekman, P., Friesen., W.: Unmasking the face: a guide to recognizing emotions from facial clues. Prentice-Hall, Englewood Cliffs (1975)Google Scholar
  14. 14.
    Fontaine, J.R., Scherer, K.R., Roesch, E.B., Ellsworth, P.: The world of emotion is not two-dimensional. Psychological Science 18(2), 1050–1057 (2007)CrossRefGoogle Scholar
  15. 15.
    Gosselin, P., Kirouac, G.: Le décodage de prototypes emotionels faciaux. Canadian journal of experimental psychology 49(3), 313-329 (1995)Google Scholar
  16. 16.
    Gunes, H., Piccardi, M.: Bi-modal emotion recognition from expressive face and body gestures. Journal of Network & Computer Applications 30(4), 1334–1345 (2007)CrossRefGoogle Scholar
  17. 17.
    Hall, J.A., Matsumoto, D.: Gender differences in judgments of multiple emotions from facial expressions. Emotion 4(2), 201–206 (2004)CrossRefGoogle Scholar
  18. 18.
    Harrigan, J.A.: Proxemics, kinesics, and gaze. In: Harrigan, J.A., Rosenthal, R., Scherer, K. (eds.) The new handbook of methods in nonverbal behavior research. Series in Affective Science, pp. 137–198. Oxford University Press, Oxford (2005)Google Scholar
  19. 19.
    Hietanen, J.K., Leppänen, J.M.: Judgment of other people’s facial expressions of emotions is influenced by their concurrent affective hand movements. Scandinavian Journal of Psychology 49, 221–230 (2008)CrossRefGoogle Scholar
  20. 20.
    Hietanen, J.K., Leppänen, J.M.: Judgment of other people’s facial expressions of emotions is influenced by their concurrent affective hand movements. Scandinavian Journal of Psychology 49, 221–230 (2008)CrossRefGoogle Scholar
  21. 21.
    Izard, C.: Human Emotions. Plenum, New York (1977)Google Scholar
  22. 22.
    Kleinsmith, A., Bianchi-Berthouze, N.: Recognizing affective dimensions from body posture. In: Paiva, A.C.R., Prada, R., Picard, R.W. (eds.) ACII 2007. LNCS, vol. 4738, pp. 48–58. Springer, Heidelberg (2007)CrossRefGoogle Scholar
  23. 23.
    Kleinsmith, A., De Silva, R., Bianchi-Berthouze, N.: Cross-Cultural Differences in Recognizing Affect from Body Posture. Interacting with Computers 18(6), 1371–1389 (2006)CrossRefGoogle Scholar
  24. 24.
    Meeren, H.K.M., van Heijnsbergen, C., de Gelder, B.: Rapid perceptual integration of facial expression and emotional body language. Proceedings of the National Academy of Sciences of the USA, 16518–16523 (2005)Google Scholar
  25. 25.
    Mota, S., Picard, R.W.: Automated posture analysis for detecting learner’s interest level. In: Workshop on Computer Vision and Pattern Recognition for Human-Computer Interaction (2003)Google Scholar
  26. 26.
    Niewiadomski, R.: A model of complex facial expressions in interpersonal relations for animated agents. PhD Thesis, University of Perugia (2007)Google Scholar
  27. 27.
    Pelachaud, C.: Multimodal expressive embodied conversational agents. In: Proceedings of the 13th annual ACM international conference on Multimedia. ACM, Hilton (2005)Google Scholar
  28. 28.
    Plutchik, R.: A general psychoevolutionary theory of emotion. In: Plutchik, R. (ed.) Emotion: Theory, research, and experience, Theories of emotion, Academic, New York. vol. 1, pp. 3–33 (1980)Google Scholar
  29. 29.
    Russell, J.A.: A circumplex model of affect. Journal of Personality and Social Psychology 39(6), 1161–1178 (1980)CrossRefGoogle Scholar
  30. 30.
    Russell, J.A., Mehrabian, A.: Evidence for a three-factor theory of emotions. Journal of Research in Personality 11, 273–294 (1977)CrossRefGoogle Scholar
  31. 31.
    Scherer, K.R.: The nonverbal dimension: A fad, a field, or a behavioral modality? The social dimension: European developments in social psychology, 160–183 (1984)Google Scholar
  32. 32.
    Scherer, K.R.: Analyzing Emotion Blends. In: Fischer, A. (ed.) Proceedings of the Xth Conference of the International Society for Research on Emotions, pp. 142–148 (1998)Google Scholar
  33. 33.
    Scherer, K.R.: Emotion. In: Stroebe, M.H.W. (ed.) Introduction to Social Psychology: A European perspective, pp. 151–191. Blackwell, Oxford (2000)Google Scholar
  34. 34.
    Scherer, K.R., Ellgring, H.: Are facial expressions of emotion produced by categorical affect programs or dynamically driven by appraisal? Emotion 7(1), 113–130 (2007)CrossRefGoogle Scholar
  35. 35.
    Scherer, K.R., Ellgring, H.: Multimodal Expression of Emotion: Affect Programs or Componential Appraisal Patterns? Emotion 7(1) (2007)Google Scholar
  36. 36.
    Wallbott, H.G.: Bodily expression of emotion. European Journal of Social Psychology 28, 879–896 (1998)CrossRefGoogle Scholar
  37. 37.
    Woo, W., Park, J., Iwadate, Y.: Emotion Analysis from Dance Performance Using Time-Delay Neural Networks. In: JCIS-CVPRIP 2000, pp. 374–377 (2000)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2009

Authors and Affiliations

  • Céline Clavel
    • 1
  • Justine Plessier
    • 2
  • Jean-Claude Martin
    • 1
  • Laurent Ach
    • 2
  • Benoit Morel
    • 2
  1. 1.LIMSI CNRSOrsay cedexFrance
  2. 2.CantocheParisFrance

Personalised recommendations