Body Motion Analysis for Emotion Recognition in Serious Games

  • Kyriaki Kaza
  • Athanasios Psaltis
  • Kiriakos Stefanidis
  • Konstantinos C. Apostolakis
  • Spyridon Thermos
  • Kosmas DimitropoulosEmail author
  • Petros Daras
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9738)


In this paper, we present an emotion recognition methodology that utilizes information extracted from body motion analysis to assess affective state during gameplay scenarios. A set of kinematic and geometrical features are extracted from joint-oriented skeleton tracking and are fed to a deep learning network classifier. In order to evaluate the performance of our methodology, we created a dataset with Microsoft Kinect recordings of body motions expressing the five basic emotions (anger, happiness, fear, sadness and surprise) which are likely to appear in a gameplay scenario. In this five emotions recognition problem, our methodology outperformed all other classifiers, achieving an overall recognition rate of 93 %. Furthermore, we conducted a second series of experiments to perform a qualitative analysis of the features and assess the descriptive power of different groups of features.


Body motion analysis 3D body movement features Emotion recognition RBM Serious games 



The research leading to this work has received funding from the EU Horizon 2020 Framework Programme under grant agreement no. 644204 (ProsocialLearn project).


  1. 1.
    Ekman, P.: Differential communication of affect by head and body cues. J. Pers. Soc. Psychol. 2(5), 726 (1965)CrossRefGoogle Scholar
  2. 2.
    Wallbott, H.G.: Bodily expression of emotion. Eur. J. Soc. Psychol. 28, 879–896 (1998)CrossRefGoogle Scholar
  3. 3.
    Boone, R.T., Cunningham, J.G.: Children’s decoding of emotion in expressive body movement: The development of cue attunement. Dev. Psychol. 34, 1007–1016 (1998)CrossRefGoogle Scholar
  4. 4.
    de Meijer, M.: The contribution of general features of body movement to the attribution of emotions. J. Nonverbal Behav. 13(4), 247–268 (1989)CrossRefGoogle Scholar
  5. 5.
    de Gelder, B.: Why bodies? twelve reasons for including bodily expressions in affective neuroscience. Philos. Trans. R. Soc. B Biol. Sci. 364(1535), 3475–3484 (2009)CrossRefGoogle Scholar
  6. 6.
    Gunes, H., Shan, C., Chen, S., Tian, Y.: Bodily expression for automatic affect recognition. In: Emotion Recognition A Pattern Analysis Approach, pp. 343–377 (2015)Google Scholar
  7. 7.
    Castellano, G., Villalba, S.D., Camurri, A.: Recognising human emotions from body movement and gesture dynamics. In: Paiva, A.C., Prada, R., Picard, R.W. (eds.) ACII 2007. LNCS, vol. 4738, pp. 71–82. Springer, Heidelberg (2007)CrossRefGoogle Scholar
  8. 8.
    Coulson, M.: Attributing emotion to static body postures: recognition accuracy, confusions, and viewpoint dependence. J. Nonverbal Behav. 28, 117–139 (2004)CrossRefGoogle Scholar
  9. 9.
    Kleinsmith, A., De Silva, R., Bianchi-Berthouze, N.: Cross-cultural differences in recognizing affect from body posture. Interact. Comput. 18(6), 1371–1389 (2006)CrossRefGoogle Scholar
  10. 10.
    Camurri, A., Mazzarino, B., Ricchetti, M., Timmers, R., Volpe, G.: Multimodal analysis of expressive gesture in music and dance performances. In: Camurri, A., Volpe, G. (eds.) GW 2003. LNCS (LNAI), vol. 2915, pp. 20–39. Springer, Heidelberg (2004)CrossRefGoogle Scholar
  11. 11.
    Kleinsmith, A., Fushimi, T., Bianchi-Berthouze, N.: An incremental and interactive affective posture recognition system. In: Carberry, S., De Rosis, F. (eds.) International Workshop on Adapting the Interaction Style to Affective Factors, in Conjunction with the International Conference on User Modeling (2005)Google Scholar
  12. 12.
    Piana, S., Stagliano, A., Odone, F., Verri, A., Camurri, A.: Real-time automatic emotion recognition from body gestures. In: Proceedings of IDGEI (2014)Google Scholar
  13. 13.
    Savva, N., Bianchi-Berthouze, N.: Automatic recognition of affective body movement in a video game scenario. In: Camurri, A., Costa, C. (eds.) INTETAIN 2011. LNICST, vol. 78, pp. 149–159. Springer, Heidelberg (2012)CrossRefGoogle Scholar
  14. 14.
    Piana, S., Stagliano, A., Camurri, A., Odone, F.: A set of full-body movement features for emotion recognition to help children affected by autism spectrum condition. In: IDGEI International Workshop (2013)Google Scholar
  15. 15.
  16. 16.
    Shan, J., Akella, S.: 3D human action segmentation and recognition using pose kinetic energy. In: 2014 IEEE Workshop on Advanced Robotics and its Social Impacts (ARSO), pp. 69–75 (2014)Google Scholar
  17. 17.
    Camurri, A., Lagerlöf, I., Volpe, G.: Recognizing emotion from dance movement: comparison of spectator recognition and automated techniques. Int. J. Hum Comput Stud. 59(1), 213–225 (2003)CrossRefGoogle Scholar
  18. 18.
    Roether, C.L., Omlor, L., Giese, M.A.: Lateral asymmetry of bodily emotion expression. Curr. Biol. 18(8), R329–R330 (2008)CrossRefGoogle Scholar
  19. 19.
    Smolensky, P.: Information processing in dynamical systems: foundations of harmony theory. In: Rumelhart, D.E., McClelland, J.L. (eds.) Parallel Distributed Processing: Explorations in the Microstructure of Cognition, vol. 1, pp. 194–281. MIT Press, Cambridge (1986)Google Scholar
  20. 20.
    Erhan, D., Bengio, Y., Courville, A., Manzagol, P.-A., Vincent, P., Bengio, S.: Why does unsupervised pre-training help deep learning? J. Mach. Learn. Res. 11, 625–660 (2010)MathSciNetzbMATHGoogle Scholar
  21. 21.
    Apostolakis, K., Kaza, K., Psaltis, A., Stefanidis, K., Thermos, S., Dimitropoulos, K., Dimaraki, E., Daras, P.: Path of trust: a prosocial co-op game for building up trustworthiness and teamwork. In: Β Games and Learning Alliance: Fourth International Conference, GALA (2015)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2016

Authors and Affiliations

  • Kyriaki Kaza
    • 1
  • Athanasios Psaltis
    • 1
  • Kiriakos Stefanidis
    • 1
  • Konstantinos C. Apostolakis
    • 1
  • Spyridon Thermos
    • 1
  • Kosmas Dimitropoulos
    • 1
    Email author
  • Petros Daras
    • 1
  1. 1.Information Technologies InstituteCentre for Research and Technology HellasThessalonikiGreece

Personalised recommendations