Gesture-Based Affective Computing on Motion Capture Data

  • Asha Kapur
  • Ajay Kapur
  • Naznin Virji-Babul
  • George Tzanetakis
  • Peter F. Driessen
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 3784)


This paper presents research using full body skeletal movements captured using video-based sensor technology developed by Vicon Motion Systems, to train a machine to identify different human emotions. The Vicon system uses a series of 6 cameras to capture lightweight markers placed on various points of the body in 3D space, and digitizes movement into x, y, and z displacement data. Gestural data from five subjects was collected depicting four emotions: sadness, joy, anger, and fear. Experimental results with different machine learning techniques show that automatic classification of this data ranges from 84% to 92% depending on how it is calculated. In order to put these automatic classification results into perspective a user study on the human perception of the same data was conducted with average classification accuracy of 93%.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Kapoor, S.M., Picard, R.W.: Towards a Learning Companion that Recognizes Affect. In: Proc. Emotional and Intelligent II: The Tangled Knot of Social Cognition, AAAI Fall Symposium, North Falmouth, MA (November 2001)Google Scholar
  2. 2.
    Fernandez, R., Picard, R.W.: Modeling Driver’s Speech under Stress. In: Proc. ISCA Workshop on Speech and Emotions, Belfast (2000)Google Scholar
  3. 3.
    Pantic, M.: Toward an Affect-Sensitve Multimodal Human-Computer Interaction. In: Proc of the IEEE 91(9) (September2003)Google Scholar
  4. 4.
    Kang, B.S., Han, C.H., Lee, S.T., Youn, D.H., Lee, C.: Speaker dependent emotion recognition using speech signals. In: Proc ICSLP, pp. 383–386 (2000)Google Scholar
  5. 5.
    Cowie, R., Douglas-Cowie, E., Tsapatsoulis, N., Votsis, G., Kollias, S., Fellenz, W., Taylor, J.G.: Emotion recognition in human-computer interaction. IEEE Signal Processing Mag. 18, 32–80 (2001)CrossRefGoogle Scholar
  6. 6.
    Ververidis, D., Kotropoulos, C., Pitas, I.: Automatic Emotional Speech Classification. In: Proc ICASSP, pp. 593–596 (2004)Google Scholar
  7. 7.
    Schiano, D.J., Ehrlich, S.M., Rahardja, K., Sheridan, K.: Face to interface: Facial affect in human and machine. In: Proc. CHI, pp. 193–200 (2000)Google Scholar
  8. 8.
    Essa, I., Pentland, A.: Coding analysis interpretation recognition of facial expressions. IEEE Trans. Pattern Anal. Machine Intell. 19, 757–763 (1997)CrossRefGoogle Scholar
  9. 9.
    Blackand, M.J., Yacoob, Y.: Recognizing facial expressions in image sequences using local parameterized models of image motion. Int. J. Conmput. Vis. 25(1), 23–48 (1997)CrossRefGoogle Scholar
  10. 10.
    Chen, L.S., Huang, T.S., Miyasato, T., Nakatsu, R.: Multimodal Human Emotion/Expression Recognition. In: Proc Third International Conference on Automatic Face and Gesture Recognition, Nara, Japan (1998)Google Scholar
  11. 11.
    De Silva, L.C., Miyasato, T., Nakatsu, R.: Facial emotion recognition using multimodal information. In: Proc FG, pp. 332–335 (2000)Google Scholar
  12. 12.
    Yoshitomi, Y., Kim, S., Kawano, T., Kitazoe, T.: Effect of sensor fusion for recognition of emotional states using voice, face image and thermal image of face. In: Proc. ROMAN, pp. 178–183 (2000)Google Scholar
  13. 13.
    Picard, R.W.: Towards Computers that Recognize and Respond to User Emotions. IBM System Journal 39, 705–719 (2001)CrossRefGoogle Scholar
  14. 14.
    Picard, R.W., Healey, J.: Affective Wearables. Personal Technologies 1(4), 231–240 (1997)CrossRefGoogle Scholar
  15. 15.
    Pollick, F.E., Paterson, H., Bruderlin, A., Sanford, A.J.: Perceiving affect from arm movement. Cognition 82, B51–B61 (2001)CrossRefGoogle Scholar
  16. 16.
    Vines, M., Wanderley, M., Krumhansl, C., Nuzzo, R., Levirin, D.: Performance Gestures of Musicians: What Structural and Emotional Information do they Convey? In: Gesture-Based Communication in Human-Computer Interaction - 5th International Gesture Workshop, Genova, Italy (2003)Google Scholar
  17. 17.
    Wallbott, H.G.: Bodily expression of emotion. European Journal of Social Psychology 28, 879–896 (1998)CrossRefGoogle Scholar
  18. 18.
    DeMeijer, M.: The contribution of general features of body movement to the attribution of emotions. Journal of Nonverbal Behavior 13, 247–268 (1989)CrossRefGoogle Scholar
  19. 19.
    Woolard, A.: Vicon 512 User Manual, Vicon Motion Systems, Tustin CA (January 1999)Google Scholar
  20. 20.
    Ian, H., Frank, E., Kaufmann, M.: Data Mining: Practical machine learning tools with Java implementations, San Francisco (2000)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2005

Authors and Affiliations

  • Asha Kapur
    • 2
  • Ajay Kapur
    • 1
  • Naznin Virji-Babul
    • 1
  • George Tzanetakis
    • 1
  • Peter F. Driessen
    • 1
  1. 1.University of VictoriaVictoriaCanada
  2. 2.School of MedicineWake Forest UniversityNorth CarolinaUnited States

Personalised recommendations