Automated Analysis of Head Pose, Facial Expression and Affect

  • Robert NieseEmail author
  • Ayoub Al-Hamadi
  • Heiko Neumann
Part of the Cognitive Technologies book series (COGTECH)


Automated analysis of facial expressions is a well-investigated research area in the field of computer vision, with impending applications such as human-computer interaction (HCI). The conducted work proposes new methods for the automated evaluation of facial expression in image sequences of color and depth data. In particular, we present the main components of our system, i.e. accurate estimation of the observed person’s head pose, followed by facial feature extraction and, third, by classification. Through the application of dimensional affect models, we overcome the use of strict categories, i.e. basic emotions, which are focused on by most state-of-the-art facial expression recognition techniques. This is of importance as in most HCI applications classical basic emotions are only occurring sparsely, and hence are often inadequate to guide the dialog with the user. To resolve this issue we suggest the mapping to the so-called “Circumplex model of affect”, which enables us to determine the current affective state of the user, which can then be used in the interaction. Especially, the output of the proposed machine vision-based recognition method gives insight to the observed person’s arousal and valence states. In this chapter, we give comprehensive information on the approach and experimental evaluation.



This work was done within the Transregional Collaborative Research Centre SFB/TRR 62 “Companion-Technology for Cognitive Technical Systems” funded by the German Research Foundation (DFG).


  1. 1.
    Bentley, J.L.: Multidimensional binary search trees used for associative searching. Commun. ACM 18(9), 509–517 (1975). Scholar
  2. 2.
    Besl, P.J., McKay, N.D.: A method for registration of 3-d shapes. IEEE Trans. Pattern Anal. Mach. Intell. 14(2), 239–256 (1992). doi:10.1109/34.121791. Scholar
  3. 3.
    Bradski, G.: The OpenCV library. Dr. Dobb’s J. Softw. Tools 25(11), 120–126 (2000)Google Scholar
  4. 4.
    Calder, A.J., Lawrence, A.D., Young, A.W.: Neuropsychology of fear and loathing. Nat. Rev. Neurosci. 2(5), 352–363 (2001). doi:10.1038/35072584. Scholar
  5. 5.
    Cootes, T.F., Taylor, C.J., Cooper, D.H., Graham, J.: Active shape models - their training and application. Comput. Vis. Image Underst. 61, 38–59 (1995)CrossRefGoogle Scholar
  6. 6.
    Ekman, P.: Strong evidence for universals in facial expressions: a reply to Russell’s mistaken critique. Psychol. Bull. 115, 268–287 (1994)CrossRefGoogle Scholar
  7. 7.
    Facegen modeller. (June 2017)Google Scholar
  8. 8.
    Fanelli, G., Gall, J., Gool, L.J.V.: Real time head pose estimation with random regression forests. In: The 24th IEEE Conference on Computer Vision and Pattern Recognition, pp. 617–624 (2011)Google Scholar
  9. 9.
    Foley, J.D., van Dam, A., Feiner, S., Hughes, J.: Computer Graphics: Principles and Practice, 3rd edn. Addison-Wesley, Boston (2013)zbMATHGoogle Scholar
  10. 10.
    GFMesstechnik GmbH Germany, T.: Facescan 3d. (2015)Google Scholar
  11. 11.
    Haykin, S.: Neural Networks: A Comprehensive Foundation, 3rd edn. Prentice-Hall, Upper Saddle River, NJ (2008)zbMATHGoogle Scholar
  12. 12.
    McGlone, C., Mikhail, E., Bethe, J.: Manual of Photogrammetry, 5th edn. ASPRS, ISBN: 1-57083-071-1 (2004)Google Scholar
  13. 13.
    Niese, R., Al-Hamadi, A., Panning, A., Brammen, D.G., Ebmeyer, U., Michaelis, B.: Towards pain recognition in post-operative phases using 3d-based features from video and support vector machines. Int. J. Digit. Content Technol. Appl. 3(4), 21–33 (2009)Google Scholar
  14. 14.
    Niese, R., Al-Hamadi, A., Heuer, M., Michaelis, B., Matuszewski, B.: Machine vision based recognition of emotions using the circumplex model of affect. In: International Conference on Multimedia Technology (ICMT), pp. 6424–6427 (2011)Google Scholar
  15. 15.
    Niese, R., Werner, P., Al-Hamadi, A.: Accurate, fast and robust realtime face pose estimation using kinect camera. In: IEEE SMC International Conference, pp. 487–490 (2013)Google Scholar
  16. 16.
    Pandzic, I.S., Forchheimer, R.: MPEG-4 Facial Animation: The Standard, Implementation and Applications, 1st edn. Wiley, New York (2002). ISBN: 0-470-84465-5CrossRefGoogle Scholar
  17. 17.
    Pantic, M., Pentland, A., Nijholt, A., Huang, T.S.: Human computing and machine understanding of human behavior: a survey. In: Artificial Intelligence for Human Computing, ICMI, pp. 47–71 (2007)Google Scholar
  18. 18.
    Ren, S., Cao, X., Wei, Y., Sun, J.: Face alignment at 3000 FPS via regressing local binary features. In: 2014 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2014, Columbus, OH, 23–28 June 2014, pp. 1685–1692 (2014)Google Scholar
  19. 19.
    Russell, J.A.: A circumplex model of affect. J. Pers. Soc. Psychol. 39, 1161–1178 (1980)CrossRefGoogle Scholar
  20. 20.
    Saeed, A., Al-Hamadi, A., Niese, R., Elzobi, M.: Frame-based facial expression recognition using geometrical features. Hindawi Adv. Hum.-Comput. Interaction (2014). doi:10.1155/2014/408953Google Scholar
  21. 21.
    Viola, P.A., Jones, M.J.: Robust real-time face detection. Int. J. Comput. Vis. 57(2), 137–154 (2004)CrossRefGoogle Scholar
  22. 22.
    Wang, J., Yin, L., Wei, X., Sun, Y.: 3d facial expression recognition based on primitive surface feature distribution. In: IEEE International Conference on Computer Vision and Pattern Recognition, CVPR06, pp. 1399–1406 (2006)Google Scholar
  23. 23.
    Xiong, X., De la Torre, F.: Supervised descent method and its applications to face alignment. In: 2013 IEEE Conference on Computer Vision and Pattern Recognition, pp. 532–539 (2013)Google Scholar
  24. 24.
    Yin, L., Chen, X., Sun, Y., Worm, T., Reale, M.: A high-resolution 3d dynamic facial expression database. In: IEEE International Conference on Automatic Face and Gesture Recognition (FG’08), pp. 1–6 (2008)Google Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  1. 1.Institute for Information Technology and Communications (IIKT)University of MagdeburgMagdeburgGermany
  2. 2.Institute for Neural Information ProcessingUniversity of UlmUlmGermany

Personalised recommendations