Facial Virtual Tracking: A System to Mirror Emotions

  • Pedro Santos
  • Vinícius Silva
  • Filomena SoaresEmail author
  • Alberto Simões
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11805)


The capacity for social interaction, communication and recognition of emotions is a characteristic that the human being possesses and that allows him/her to be socially included. However, some have difficulties in expressing and interpreting emotional states, which can contribute to their marginalisation in society. A particular case is children with Autism Spectrum Disorder. These children have difficulties in social interaction and manifest repetitive patterns. In this work, a serious game is being developed in which an avatar (ZECA avatar) is able to interact with the child, through challenges and training of certain facial movements that will be validated by the system, facilitating imitation and recognition of emotions (happiness, sadness, anger, surprise and fear). The tests performed in the laboratory environment allowed to conclude the adequacy of the game for promoting emotional states in a friendly way.


Serious game Facial expressions Emotions Human Computer Interaction 



This work has been supported by COMPETE: POCI-01-0145-FEDER-007043 and FCT – Fundação para a Ciência e Tecnologia within the Project Scope: UID/CEC/00319/2019. Vinicius Silva also thanks FCT for the PhD scholarship SFRH/BD/SFRH/BD/133314/2017.


  1. 1.
    Martins, C.: Face a face com o Autismo-será a Inclusão um mito ou uma realidade?, Escola Superior de Educação João de Deus, Lisboa (2012). (in Portuguese)Google Scholar
  2. 2.
    Costa, S., Soares, F., Pereira, A.P., Santos, C., Hiolle, A.: A pilot study using imitation and storytelling scenarios as activities for labelling emotions by children with autism using a humanoid robot. In: The Fourth Joint IEEE International Conference on Development and Learning and on Epigenetic Robotics, IEEE ICDL-EPIROB 2014, Genova, Italy, 13–16 October (2014)Google Scholar
  3. 3.
    Silva, V., Soares, F., Esteves, J.S.: Mirroring and recognizing emotions through facial expressions for a RoboKind platform. In: IEEE ENBENG 2017 - 5th Portuguese Meeting in Bioengineering, Coimbra, Portugal, 16–18 February 2017Google Scholar
  4. 4.
    Costa, S., Soares, F., Santos, C., Pereira, A.P., Hiolle, A., Silva, V.: Socio-emotional development in children with autism spectrum disorders using a humanoid robot, interaction studies (in press)Google Scholar
  5. 5.
    Azevedo, J., Silva, V., Soares, F., Pereira, A.P., Esteves, J.S.: An application to promote emotional skills in children with autism spectrum disorders. In: Göbel, S., et al. (eds.) Serious Games JCSG 2018. LNCS, vol. 11243, pp. 282–287. Springer, Cham (2018). Scholar
  6. 6.
    Dautenhahn, K., Billard, A.: Games children with autism can play with robota, a humanoid robotic doll. In: Keates, S., Langdon, P., Clarkson, P.J., Robinson, P. (eds.) Universal Access and Assistive Technology, pp. 179–190. Springer, London (2002). Scholar
  7. 7.
    Dautenhahn, K., et al.: KASPAR - a minimally expressive humanoid robot for human-robot interaction research. Appl. Bionics Biomech. 6(3–4), 369–397 (2009)CrossRefGoogle Scholar
  8. 8.
    Kozima, H., Michalowski, M.P., Nakagawa, C.: Keepon. Int. J. Soc. Robot. 1(1), 3–18 (2008)CrossRefGoogle Scholar
  9. 9.
    Miskam, M.A., Shamsuddin, S., Samat, M., Yussof, H., Ainudin, H., Omar, A.: Humanoid robot NAO as a teaching tool of emotion recognition for children with autism using the Android app. In: 2014 International Symposium on Micro-NanoMechatronics and Human Science, MHS 2014, pp. 2–6 (2015)Google Scholar
  10. 10.
    Freitas, H., Costa, P., Silva, V., Pereira, A.P., Soares, F., Sena Esteves, J.: Using a humanoid robot as the promoter of the interaction with children in the context of educational games. Int. J. Mechatron. Appl. Mech. 1(1), 282–288 (2017)Google Scholar
  11. 11.
    Silva, V., Soares, F., Sena Esteves, J.: Mirroring emotion system – on-line synthesizing facial expressions on a robot face. In: 8th International Congress on Ultra Modern Telecommunications and Control Systems and Workshops (ICUMT), Lisboa, Portugal, 18–20 October 2016Google Scholar
  12. 12.
    LuxAI: “QTrobot -,”. Accessed 6 Dec 2018
  13. 13.
    Livox: Livox: para pessoas com deficiências e transtornos de aprendizagem. Accessed 6 Dec 2018. (in Portuguese)
  14. 14.
    Tobii Dynavox logo: “Sono Flex for Communicator 5 - Tobii Dynavox.” Accessed 7 Dec 2018
  15. 15.
    TippyTalk - Comunicação Instantânea para Pessoas Não Verbais. (n.d.). Accessed 16 Jan 2019. (in Portuguese)
  16. 16.
    Ferreira, V.: Utilização das tecnologias em crianças com perturbações do espectro autista em contexto da Prática de Ensino Supervisionada, Universidade Católica Portuguesa, Centro Regional de Braga, Faculdade de Ciências Sociais, Braga (2013). (in Portuguese)Google Scholar
  17. 17.
    Ekman, P., Rosenberg, E.L.: What the Face Reveals: Basic and Applied Studies of Spontaneous Expression Using the Facial Action Coding System (FACS). Oxford University Press, Oxford (2005)Google Scholar
  18. 18.
    Darwin, C.: The Expression of Emotions in Man and Animals. D. Appleton and Company, New York (1897)Google Scholar
  19. 19.
    Baltrušaitis, T., Mahmoud, M., Robinson, P.: Cross-dataset learning and person-specific normalisation for automatic action unit detection. In: Facial Expression Recognition and Analysis Challenge. In: IEEE International Conference on Automatic Face and Gesture Recognition (2015)Google Scholar
  20. 20.
    Fuse -> Adobe. Fuse (Beta). Accessed 30 April 2019.
  21. 21.
    Mixamo -> Adobe. Mixamo. Accessed 30 April 2019.

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  • Pedro Santos
    • 1
  • Vinícius Silva
    • 2
  • Filomena Soares
    • 1
    • 2
    Email author
  • Alberto Simões
    • 3
  1. 1.Industrial Electronics Department, School of EngineeringUniversity of MinhoGuimarãesPortugal
  2. 2.R&D Centre Algoritmi, School of EngineeringUniversity of MinhoGuimarãesPortugal
  3. 3.IPCABarcelosPortugal

Personalised recommendations