Features in the Recognition of Emotions from Dynamic Bodily Expression

  • Claire L. Roether
  • Lars Omlor
  • Martin A. Giese


Body movements can reveal important information about a person’s emotional state. The visual system efficiently extracts subtle information about the emotional style of a movement, even from point-light stimuli. While much existing work has addressed the problem of style perception from a holistic perspective, we try to investigate which features are critical for the recognition of emotions from full-body movements. This work is inspired by the motor-control concept of “synergies,” which define spatial components of movements that encompass only a limited set of degrees of freedom that are jointly controlled. We present an algorithm that learns a highly compact generative model for the joint-angle trajectories of emotional body movements. The model approximates movements by nonlinear superpositions of a small number of basis components. Applying sparse feature learning, we extracted from this representation the spatial components that are characteristic for happy, sad, fearful and angry movements. The extracted features for walking were highly consistent with emotion-specific features of gait, as described in the literature. We further show that this type of result is not restricted to locomotor movements. Compared to other techniques, the proposed algorithm requires significantly fewer basic components to accomplish the same level of accuracy. In addition, we show that feature learning based on such less compact representations does not result in easily interpretable local features. Based on the features extracted from the trajectory data, we studied how spatio-temporal components that convey information about emotional styles of body movements are integrated in visual perception. Using motion morphing to vary the information content of different components, we show that the integration of spatial features is slightly suboptimal compared to a Bayesian ideal observer. Besides, integration was worse for components that matched the components extracted from the movement trajectories. This result is inconsistent with the hypothesis that emotional body movements are recognized by a parallel internal simulation of the underlying motor behavior. Instead, it seems that the recognition of emotion from body movements is based on a purely visual process that is influenced by the distribution of attention.


Independent Component Analysis Spatial Component Emotional Expressiveness Blind Source Separation Emotional Style 



We thank T. Flash for many interesting discussions, and for pointing our interest to synergies as classical concept of spatio-temporal components in motor control, and B. de Gelder and A. Berthoz for interesting comments. We are grateful to W. Ilg for help with the motion capturing. This research was supported by HFSP, EC FP6 project COBOL, and the Volkswagenstiftung. Further support by the Max Planck Institute for Biological Cybernetics and the Hermann und Lilly Schilling-Stiftung is gratefully acknowledged.


  1. Alais D, Burr D (2004) The ventriloquist effect results from near-optimal bimodal integration. Curr Biol 14(3):257–262PubMedGoogle Scholar
  2. Amaya K, Bruderlin A, Calvert T (1996) Emotion from motion. In: Proceedings of the conference on graphics interface ’96, Canadian Information Processing Society, Toronto, Ontario, CanadaGoogle Scholar
  3. Andrew YN (2004) Feature selection, L1 vs. L2 regularization, and rotational invariance. In: Proceedings of the twenty-first international conference on machine learning, ACM Press, Banff, Alberta, CanadaGoogle Scholar
  4. Atkinson AP, Dittrich WH, Gemmell AJ, Young AW (2004) Emotion perception from dynamic and static body expressions in point-light and full-light displays. Perception 33(6):717–746PubMedCrossRefGoogle Scholar
  5. Atkinson AP, Tunstall ML, Dittrich WH (2007) Evidence for distinct contributions of form and motion information to the recognition of emotions from body gestures. Cognition 104(1):59–72PubMedCrossRefGoogle Scholar
  6. Bell AJ, Sejnowski TJ (1995) An information-maximization approach to blind separation and blind deconvolution. Neural Comput 7(6):1129–1159PubMedCrossRefGoogle Scholar
  7. Bernstein NA (1967) The coordination and regulation of movements. Pergamon Press, Oxford, New YorkGoogle Scholar
  8. Boone RT, Cunningham JG (1998) Children’s decoding of emotion in expressive body movement: the development of cue attunement. Dev Psychol 34(5):1007–1016PubMedCrossRefGoogle Scholar
  9. Brainard DH (1997) The psychophysics toolbox. Spat Vis 10(4):433–436PubMedCrossRefGoogle Scholar
  10. Brand M, Hertzmann A (2000) Style machines. In: Proceedings of the 27th annual conference on computer graphics and interactive techniques, ACM Press/Addison-Wesley Publishing CoGoogle Scholar
  11. Bruderlin A, Williams L (1995) Motion signal processing. In: Proceedings of the 22nd annual conference on computer graphics and interactive techniques, ACM Press, pp 97–104Google Scholar
  12. Cacioppo JT, Berntson GG, Larsen JT, Poehlmann KM, Ito TA (2000) The psychophysiology of emotion. In: Lewis R, Haviland-Jones JM (eds) The handbook of emotion. Guilford Press, New York, pp 173–191Google Scholar
  13. Cavanagh P, Labianca AT, Thornton IM (2001) Attention-based visual routines: sprites. Cognition 80(1–2):47–60PubMedCrossRefGoogle Scholar
  14. Clarke TJ, Bradshaw MF, Field DT, Hampson SE, Rose D (2005) The perception of emotion from body movement in point-light displays of interpersonal dialogue. Perception 34(10):1171–1180PubMedCrossRefGoogle Scholar
  15. d’Avella A, Bizzi E (2005) Shared and specific muscle synergies in natural motor behaviors. Proc Natl Acad Sci USA 102(8):3076–3081PubMedCrossRefGoogle Scholar
  16. Darwin, Charles (1872) The expression of the emotions in man and animals, London: John MurrayGoogle Scholar
  17. de Gelder B (2006) Towards the neurobiology of emotional body language. Nat Rev Neurosci 7(3):242–249PubMedCrossRefGoogle Scholar
  18. de Gelder B, Hadjikhani N (2006) Non-conscious recognition of emotional body language. Neuroreport 17(6):583–586PubMedCrossRefGoogle Scholar
  19. de Meijer M (1989) The contribution of general features of body movement to the attribution of emotions. J Nonverbal Behav 13(4):247–268CrossRefGoogle Scholar
  20. de Meijer M (1991) The attribution of aggression and grief to body movements: The effect of sex-stereotypes. Eur J Soc Psychol 21(3):249–259CrossRefGoogle Scholar
  21. Ekman P (1965) Differential communication of affect by head and body cues. J Pers Soc Psychol 2(5):726–735PubMedCrossRefGoogle Scholar
  22. Ekman P (1992) Are there basic emotions? Psychol Rev 99(3):550–553PubMedCrossRefGoogle Scholar
  23. Ekman P, Friesen WV (1967) Head and body cues in the judgment of emotion: a reformulation. Percept Mot Skills 24(3):711–724PubMedCrossRefGoogle Scholar
  24. Ekman P, Friesen WV (1971) Constants across cultures in the face and emotion. J Pers Soc Psychol 17(2):124–129PubMedCrossRefGoogle Scholar
  25. Ekman P, Friesen WV (1972) Hand movements. J Commun 22(4):353–374CrossRefGoogle Scholar
  26. Elfenbein HA, Foo MD, White JB, Tan HH, Aik VC (2007) Reading your counterpart: the benefit of emotion recognition accuracy for effectiveness in negotiation. J Nonverbal Behav 31(4):205–223CrossRefGoogle Scholar
  27. Ernst MO, Banks MS (2002) Humans integrate visual and haptic information in a statistically optimal fashion. Nature 415(6870):429–433PubMedCrossRefGoogle Scholar
  28. Flash T, Hochner B (2005) Motor primitives in vertebrates and invertebrates. Curr Opin Neurobiol 15(6):660–666PubMedCrossRefGoogle Scholar
  29. Friesen WV, Ekman P, Wallbott H (1979) Measuring hand movements. J Nonverbal Behav 4(2):97–112CrossRefGoogle Scholar
  30. Gallese V (2006) Intentional attunement: a neurophysiological perspective on social cognition and its disruption in autism. Brain Res 1079(1):15–24PubMedCrossRefGoogle Scholar
  31. Giese MA, Lappe M (2002) Measurement of generalization fields for the recognition of biological motion. Vision Res 42(15):1847–1858PubMedCrossRefGoogle Scholar
  32. Giese MA, Poggio T (2000) Morphable models for the analysis and synthesis of complex motion patterns. Int J Comput Vis 38(1):59–73CrossRefGoogle Scholar
  33. Giese MA, Poggio T (2003) Neural mechanisms for the recognition of biological movements. Nat Rev Neurosci 4(3):179–192PubMedCrossRefGoogle Scholar
  34. Grezes J, Pichon S, de Gelder B (2007) Perceiving fear in dynamic body expressions. Neuroimage 35(2):959–967PubMedCrossRefGoogle Scholar
  35. Harel A, Ullman S, Epshtein B, Bentin S (2007) Mutual information of image fragments predicts categorization in humans: electrophysiological and behavioral evidence. Vision Res 47(15):2010–2020PubMedCrossRefGoogle Scholar
  36. Hietanen JK, Leppanen JM, Lehtonen U (2004) Perception of emotions in the hand movement quality of finnish sign language. J Nonverbal Behav 28(1):53–64CrossRefGoogle Scholar
  37. Hillis JM, Watt SJ, Landy MS, Banks MS (2004) Slant from texture and disparity cues: optimal cue combination. J Vis 4(12):967–992PubMedCrossRefGoogle Scholar
  38. Hojen-Sorensen PAdFR, Winther O, Hansen LK (2002) Mean-field approaches to independent component analysis. Neural Comput 14(4):889–918PubMedCrossRefGoogle Scholar
  39. Ivanenko YP, Poppele RE, Lacquaniti F (2004) Five basic muscle activation patterns account for muscle activity during human locomotion. J Physiol 556(Pt 1):267–282PubMedGoogle Scholar
  40. Izard CE (1977) Human emotions. Plenum Press, New YorkGoogle Scholar
  41. Jordan H, Fallah M, Stoner GR (2006) Adaptation of gender derived from biological motion. Nat Neurosci 9(6):738–739PubMedCrossRefGoogle Scholar
  42. Knill DC (2003) Mixture models and the probabilistic structure of depth cues. Vision Res 43(7):831–854PubMedCrossRefGoogle Scholar
  43. Knill DC (2007) Robust cue integration: a Bayesian model and evidence from cue-conflict studies with stereoscopic and figure cues to slant. J Vis 7(7):1–24CrossRefGoogle Scholar
  44. Landy MS, Kojima H (2001) Ideal cue combination for localizing texture-defined edges. J Opt Soc Am A Opt Image Sci Vis 18(9):2307–2320PubMedCrossRefGoogle Scholar
  45. Landy MS, Maloney LT, Johnston EB, Young M (1995) Measurement and modeling of depth cue combination: in defense of weak fusion. Vision Res 35(3):389–412PubMedCrossRefGoogle Scholar
  46. Lee DD, Seung HS (1999) Learning the parts of objects by non-negative matrix factorization. Nature 401(6755):788–791PubMedCrossRefGoogle Scholar
  47. Logothetis NK, Pauls J, Poggio T (1995) Shape representation in the inferior temporal cortex of monkeys. Curr Biol 5(5):552–563PubMedCrossRefGoogle Scholar
  48. Mezger J, Ilg W, Giese MA (2005) Trajectory synthesis by hierarchical spatio-temporal correspondence: comparison of different methods. In: ACM SIGGRAPH symposium on applied perception in graphics and visualization, A Coruna, Spain, pp 25–32Google Scholar
  49. Montepare J, Koff E, Zaitchik D, Albert M (1999) The use of body movements and gestures as cues to emotions in younger and older adults. J Nonverbal Behav 23(2):133–152CrossRefGoogle Scholar
  50. Montepare JM, Goldstein SB, Clausen A (1987) The identification of emotions from gait information. J Nonverbal Behav 11(1):33–42CrossRefGoogle Scholar
  51. Omlor L, Giese MA (2007) Blind source separation for over-determined delayed mixtures. In: Schoülkopf, Bernhard; Platt, John; Hofmann, Thomas (ed) Advances in neural information processing systems, vol 19. MIT Press, Cambridge, MA, pp 1049–1056Google Scholar
  52. Poggio T, Bizzi E (2004) Generalization in vision and motor control. Nature 431(7010):768–774PubMedCrossRefGoogle Scholar
  53. Pollick FE, Lestou V, Ryu J, Cho SB (2002) Estimating the efficiency of recognizing gender and affect from biological motion. Vision Res 42(20):2345–2355PubMedCrossRefGoogle Scholar
  54. Pollick FE, Paterson HM, Bruderlin A, Sanford AJ (2001) Perceiving affect from arm movement. Cognition 82(2):B51–B61PubMedCrossRefGoogle Scholar
  55. Roether CL, Omlor L, Christensen A, Giese MA (2009) Critical features for the perception of emotion from gait. Journal of Vision, 9(6):15, 1–32, http://Journalofvision.org/9/6/15/, doi:10.1167/9.6.15Google Scholar
  56. Safonova A, Hodgins JK, Pollard, NS (2004) Synthesizing physically realistic human motion in low-dimensional, behavior-specific spaces. In: ACM SIGGRAPH 2004 Papers, ACM Press, Los Angeles, CaliforniaGoogle Scholar
  57. Santello M, Soechting JF (1997) Matching object size by controlling finger span and hand shape. Somatosens Mot Res 14(3):203–212PubMedCrossRefGoogle Scholar
  58. Sawada M, Suda K, Ishii M (2003) Expression of emotions in dance: relation between arm movement characteristics and emotion. Percept Mot Skills 97(3 Pt 1):697–708PubMedGoogle Scholar
  59. Sogon S, Masutani M (1989) Identification of emotion from body movements: a cross-cultural study of Americans and Japanese. Psychol Rep 65(1):35–46CrossRefGoogle Scholar
  60. Thornton IM, Rensink RA, Shiffrar M (2002) Active versus passive processing of biological motion. Perception 31(7):837–853PubMedCrossRefGoogle Scholar
  61. Troje NF (2002) Decomposing biological motion: a framework for analysis and synthesis of human gait patterns. J Vis 2(5):371–387PubMedCrossRefGoogle Scholar
  62. Unuma M, Anjyo K, Takeuchi R (1995) Fourier principles for emotion-based human figure animation. In: Proceedings of the 22nd annual conference on Computer graphics and interactive techniques, ACM Press, New York, N.Y., USA, pp 91–96Google Scholar
  63. Walk RD, Homan CP (1984) Emotion and dance in dynamic light displays. Bull Psychon Soc 22(5):437–440Google Scholar
  64. Wallbott HG (1998) Bodily expression of emotion. Eur J Soc Psychol 28(6):879–896CrossRefGoogle Scholar
  65. Wallbott HG, Scherer KR (1986) Cues and channels in emotion recognition. J Pers Soc Psychol 51(4):690–699CrossRefGoogle Scholar
  66. Westermann R, Spies K, Stahl G, Hesse FW (1996) Relative effectiveness and validity of mood induction procedures: A meta-analysis. Eur J Soc Psychol 26(4):557–580CrossRefGoogle Scholar
  67. Wiley DJ, Hahn JK (1997) Interpolation synthesis of articulated figure motion. IEEE Comput Graph Appl 17(6):39–45CrossRefGoogle Scholar
  68. Wolpert DM, Doya K, Kawato M (2003) A unifying computational framework for motor control and social interaction. Philos Trans R Soc Lond B Biol Sci 358(1431):593–602PubMedCrossRefGoogle Scholar
  69. Yacoob Y, Black MJ (1999) Parameterized modeling and recognition of activities. Comput Vis Image Underst 73(2):232–247(216)Google Scholar

Copyright information

© Springer Science+Business Media, LLC 2009

Authors and Affiliations

  • Claire L. Roether
    • 1
  • Lars Omlor
  • Martin A. Giese
  1. 1.Section for Computational Sensomotorics, Department of Cognitive NeurologyHertie Institute for Clinical Brain Research & Center for Integrative NeuroscienceTuebingenGermany

Personalised recommendations