Real-Time Emotion Recognition: An Improved Hybrid Approach for Classification Performance

  • Claudio Loconsole
  • Domenico Chiaradia
  • Vitoantonio Bevilacqua
  • Antonio Frisoli
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8588)


Emotions, and more in detail facial emotions, play a crucial role in human communication. While for humans the recognition of facial states and their changes is automatic and performed in real-time, for machines the modeling and the emulation of this natural process through computer vision-based approaches are still a challenge, since real-time and automation system requirements negatively affect the accuracy in emotion detection processes.

In this work, we propose an approach which improves the classification performance of our previous computer vision-based algorithm for facial feature extraction and automatic emotion recognition. The proposed approach integrates the previous one adding six geometrical and two appearance-based features, still meeting the real-time requirement. As result, we obtain an improved processing pipeline classifier (classification accuracy incremented up to 6-7%) which allows the recognition of eight facial emotions (six basic Ekman’s emotions plus Contemptuous and Neutral).


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Bartlett, M., Littlewort, G., Fasel, I., Movellan, J.: Real time face detection and facial expression recognition: Development and applications to human computer interaction. In: Computer Vision and Pattern Recognition Workshop, CVPRW 2003, vol. 5, p. 53. IEEE (2003)Google Scholar
  2. 2.
    Bettadapura, V.: Face expression recognition and analysis: The state of the art. Emotion, 1–27 (2009)Google Scholar
  3. 3.
    Bevilacqua, V., Cariello, L., Carro, G., Daleno, D., Mastronardi, G.: A face recognition system based on pseudo 2d hmm applied to neural network coefficients. Soft Computing 12(7), 615–621 (2008)CrossRefGoogle Scholar
  4. 4.
    Bevilacqua, V., Casorio, P., Mastronardi, G.: Extending hough transform to a points cloud for 3d-face nose-tip detection, pp. 1200–1209 (2008)Google Scholar
  5. 5.
    Breiman, L.: Random forests. Machine Learning 45(1), 5–32 (2001)CrossRefzbMATHGoogle Scholar
  6. 6.
    Cheon, Y., Kim, D.: Natural facial expression recognition using differential-aam and manifold learning. Pattern Recognition 42(7), 1340–1350 (2009)CrossRefzbMATHGoogle Scholar
  7. 7.
    Cohen, I., Sebe, N., Garg, A., Chen, L., Huang, T.: Facial expression recognition from video sequences: temporal and static modeling. Computer Vision and Image Understanding 91(1), 160–187 (2003)CrossRefGoogle Scholar
  8. 8.
    Ekman, P., Friesen, W.: Facial Action Coding System: A Technique for the Meas-urement of Facial Movement. Consulting Psychologists Press, Palo Alto (1978)Google Scholar
  9. 9.
    Fernandes, T., Miranda, J., Alvarez, X., Orvalho, V.: LIFE is- GAME - An Interactive Serious Game for Teaching Facial Expression Recognition. Interfaces, 1–2 (2011)Google Scholar
  10. 10.
    Fischer, R.: Automatic Facial Expression Analysis and Emotional Classification (2004)Google Scholar
  11. 11.
    Gang, L., Xiao-hua, L., Ji-liu, Z., Xiao-gang, G.: Geometric feature based facial expression recognition using multiclass support vector machines. In: IEEE International Conference on Granular Computing, GRC 2009, pp. 318–321 (2009)Google Scholar
  12. 12.
    Hong, J., Han, M., Song, K., Chang, F.: A fast learning algorithm for robotic emotion recognition. In: International Symposium on Computational Intelligence in Robotics and Automatic, CIRA 2007, pp. 25–30. IEEE (2007)Google Scholar
  13. 13.
    Jamshidnezhad, A., Nordin, M.: Challenging of facial expressions classification systems: Survey, critical considerations and direction of future work. Research Journal of Applied Sciences 4 (2012)Google Scholar
  14. 14.
    Kapoor, A., Qi, Y., Picard, R.W.: Fully automatic upper facial action recognition. In: Proceedings of the IEEE International Workshop on Analysis and Modeling of Faces and Gestures, AMFG 2003, p. 195 (2003)Google Scholar
  15. 15.
    Ko, K., Sim, K.: Development of a facial emotion recognition method based on combining aam with dbn. In: International Conference on Cyberworlds (CW), pp. 87–91. IEEE (2010)Google Scholar
  16. 16.
    Kotsia, I., Buciu, I., Pitas, I.: An analysis of facial expression recognition under partial facial image occlusion. Image and Vision Computing 26(7), 1052–1067 (2008)CrossRefGoogle Scholar
  17. 17.
    Kotsia, I., Pitas, I.: Facial expression recognition in image sequences using geometric deformation features and support vector machines. IEEE Transactions on Image Processing 16(1), 172–187 (2007)CrossRefMathSciNetGoogle Scholar
  18. 18.
    Langner, O., Dotsch, R., Bijlstra, G., Wigboldus, D.H., Hawk, S.T., van Knippenberg, A.: Presentation and validation of the radboud faces database. Cognition and Emotion 24(8), 1377–1388 (2010)CrossRefGoogle Scholar
  19. 19.
    Loconsole, C., Runa Miranda, C., Augusto, G., Frisoli, A., Orvalho, V.: Real-time emotion recognition: a novel method for geometrical facial features extraction. In: 9th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISAPP 2014). SCITEPRESS (2014)Google Scholar
  20. 20.
    Luximon, Y., Ball, R., Justice, L.: The 3d Chinese head and face modeling. Computer-Aided Design (2011)Google Scholar
  21. 21.
    Michel, P., El Kaliouby, R.: Real time facial expression recognition in video using support vector machines. In: Proceedings of the 5th International Conference on Multimodal Interfaces, pp. 258–264. ACM (2003)Google Scholar
  22. 22.
    Niese, R., Al-Hamadi, A., Farag, A., Neumann, H., Michaelis, B.: Facial expression recognition based on geometric and optical flow features in colour image sequences. IET Computer Vision 6(2), 79–89 (2012)CrossRefMathSciNetGoogle Scholar
  23. 23.
    Pardàs, M., Bonafonte, A.: Facial animation parameters extraction and expression recognition using hidden markov models. Signal Processing: Image Communication 17(9), 675–688 (2002)Google Scholar
  24. 24.
    Rodriguez, J., Perez, A., Lozano, J.: Sensitivity analysis of k-fold cross validation in prediction error estimation. IEEE Transactions on Pattern Analysis and Machine Intelligence 32(3), 569–575 (2010)CrossRefGoogle Scholar
  25. 25.
    Saragih, J., Lucey, S., Cohn, J.: Deformable model fitting by regularized landmark mean-shift. International Journal of Computer Vision, 1–16 (2011)Google Scholar
  26. 26.
    Saragih, J., Lucey, S.: Cohn, J.: Real-time avatar animation from a single image. In: IEEE International Conference on Automatic Face & Gesture Recognition and Workshops (FG 2011), pp. 117–124 (2011)Google Scholar
  27. 27.
    Wang, J., Yin, L.: Static topographic modeling for facial expression recognition and analysis. Computer Vision and Image Understanding 108(1-2), 19–34 (2007)CrossRefGoogle Scholar
  28. 28.
    Youssif, A.A.A., Asker, W.A.A.: Automatic facial expression recognition system based on geometric and appearance features. Computer and Information Science, 115–124 (2011)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2014

Authors and Affiliations

  • Claudio Loconsole
    • 1
  • Domenico Chiaradia
    • 2
  • Vitoantonio Bevilacqua
    • 2
  • Antonio Frisoli
    • 1
  1. 1.PERCROTeCIP Scuola Superiore Sant’AnnaPisaItaly
  2. 2.Dipartimento di Ingegneria Elettrica e dell’Informazione (DEI)Politecnico di BariBariItaly

Personalised recommendations