Pattern Analysis and Applications

, Volume 16, Issue 1, pp 41–54 | Cite as

Facial emotional classification: from a discrete perspective to a continuous emotional space

  • Isabelle HupontEmail author
  • Sandra  Baldassarri
  • Eva Cerezo
Theoretical Advances


User emotion detection is a very useful input to develop affective computing strategies in modern human computer interaction. In this paper, an effective system for facial emotional classification is described. The main distinguishing feature of our work is that the system does not simply provide a classification in terms of a set of discrete emotional labels, but that it operates in a continuous 2D emotional space enabling a wide range of intermediary emotional states to be obtained. As output, an expressional face is represented as a point in a 2D space characterized by evaluation and activation factors. The classification method is based on a novel combination of five classifiers and takes into consideration human assessment for the evaluation of the results. The system has been tested with an extensive universal database so that it is capable of analyzing any subject, male or female of any age and ethnicity. The results are very encouraging and show that our classification strategy is consistent with human brain emotional classification mechanisms.


Affective computing Algorithms Facial expression analysis Intelligent user interfaces 



This work has been partly financed by the Spanish Government through the DGICYT contract TIN2011-24660, by the project FEDER ATIC, and the SISTRONIC Group of the Aragon Institute of Technology (Ref. T84).


  1. 1.
    Anderson K, McOwan P (2006) A real-time automated system for the recognition of human facial expressions. IEEE Trans Syst Man Cybern Part B 36:96–105CrossRefGoogle Scholar
  2. 2.
    Bassili J (1979) Emotion recognition: the role of facial movement and the relative importance of upper and lower areas of the face. J Pers Soc Psychol 37:2049–2058CrossRefGoogle Scholar
  3. 3.
    Boukricha H, Becker C, Wachsmuth I (2007) Simulating empathy for the virtual human max. In: Proceedings 2nd international workshop on emotion and computing in conjunction with the German conference on artificial intelligence (KI2007), pp 22–27Google Scholar
  4. 4.
    Caridakis G, Malatesta L, Kessous L, Amir N, Paouzaiou A, Karpouzis K (2006) Modeling naturalistic affective states via facial and vocal expression recognition. In: International conference on multimodal interfaces, pp 146–154Google Scholar
  5. 5.
    Chang C, Tsai J, Wang C, Chung P (2009) Emotion recognition with consideration of facial expression and physiological signals. In: Proceedings of the 6th annual IEEE conference on computational intelligence in bioinformatics and computational biology, pp 278–283Google Scholar
  6. 6.
    Cohen I, Sebe N, Garg A, Chen L, Huang T (2003) Facial expression recognition from video sequences: temporal and static modeling. Comput Vis Image Underst 11(1–2):160–187CrossRefGoogle Scholar
  7. 7.
    Datcu D, Rothkrantz L (2007) Facial expression recognition in still pictures and videos using active appearance models: a comparison approach. In: Proceedings of the international conference on computer systems and technologies, pp 1–6Google Scholar
  8. 8.
    Du Y, Bi W, Wang T, Zhang Y, Ai H (2007) Distributing expressional faces in 2-d emotional space. In: Proceedings of the 6th ACM international conference on image and video retrieval, pp 395–400Google Scholar
  9. 9.
    Ekman P, Dalgleish T, Power M (1999) Handbook of cognition and emotion. Wiley, ChihesterGoogle Scholar
  10. 10.
    Ekman P, Friesen W, Hager J (2002) Facial action coding system. Research Nexus eBookGoogle Scholar
  11. 11.
    Fragopanagos N, Taylor J (2005) Emotion recognition in human–computer interaction. Neural Netw 18:389–405CrossRefGoogle Scholar
  12. 12.
    Fredrickson B (2003) The value of positive emotions. Am Sci 91:330–335Google Scholar
  13. 13.
    Gosselin F, Schyns P (2001) Bubbles: A technique to reveal the use of information in recognition tasks. Vis Res 41:2261–2271CrossRefGoogle Scholar
  14. 14.
    Gunes H, Schuller B, Pantic M, Cowie R (2011) Emotion representation, analysis and synthesis in continuous space: a survey. In: 2011 IEEE international conference on automatic face gesture recognition and workshops (FG 2011), pp 827–834Google Scholar
  15. 15.
    Hall M (1998) Correlation-based feature selection for machine learning. PhD thesis, Hamilton, New ZealandGoogle Scholar
  16. 16.
    Hammal Z, Couvreur L, Caplier A, Rombaut M (2007) Facial expression classification: An approach based on the fusion of facial deformations using the transferable belief model. Int J Approx Reason 46:542–567CrossRefGoogle Scholar
  17. 17.
    Ji Q, Lan P, Looney C (2006) A probabilistic framework for modeling and real-time monitoring human fatigue. IEEE Trans Syst Man Cybern Part A 36:862–875CrossRefGoogle Scholar
  18. 18.
    Kapoor A, Burleson W, Picard R (2007) Automatic prediction of frustration. Int J Hum Comput Stud 65:724–736CrossRefGoogle Scholar
  19. 19.
    Keltner D, Ekman P (2000) Facial expression of emotion. Handb Emot 2:236–249Google Scholar
  20. 20.
    Kotsia I, Pitas I (2007) Facial expression recognition in image sequences using geometric deformation features and support vector machines. IEEE Trans Image Process 16:172–187MathSciNetCrossRefGoogle Scholar
  21. 21.
    Kumar P, Yildirim E (2005) Minimum-volume enclosing ellipsoids and core sets. J Optim Theory Appl 126:1–21MathSciNetzbMATHCrossRefGoogle Scholar
  22. 22.
    Littlewort G, Bartlett M, Fasel I, Chenu J, Movellan J (2004) Analysis of machine learning methods for real-time recognition of facial expressions from video. In: Computer vision and pattern recognition, face processing workshopGoogle Scholar
  23. 23.
    Littlewort G, Bartlett M, Fasel I, Susskind J, Movellan J (2006) Dynamics of facial expression extracted automatically from video. Image Vis Comput 24:615–625CrossRefGoogle Scholar
  24. 24.
    Liu W, Lu J, Wang Z, Song H (2008) An expression space model for facial expression analysis. In: Congress on image and signal processing (CISP 08), vol 2, pp 680–684Google Scholar
  25. 25.
    Lucey S, Ashraf A, Cohn J (2007) Investigating spontaneous facial action recognition through aam representations of the face. In: Handbook on Face Recognition, pp 275–286Google Scholar
  26. 26.
    Maalej A, Amor BB, Daoudi M, Srivastava A, Berretti S (2011) Shape analysis of local facial patches for 3d facial expression recognition. Pattern Recogn 44(8):1581–1589CrossRefGoogle Scholar
  27. 27.
    Pantic M, Bartlett M (2007) Machine analysis of facial expressions. Face recognition pp 377–416Google Scholar
  28. 28.
    Pantic M, Valstar M, Rademaker R, Maat L (2005) Web-based database for facial expression analysis. In: IEEE international conference on multimedia and expo, pp 317–321Google Scholar
  29. 29.
    Plutchik R (1980) Emotion: a psychoevolutionary synthesis. Harper & Row, New YorkGoogle Scholar
  30. 30.
    SeeingMachines (2008) Face api technical specifications brochure. Accessed 1 Mar 2010
  31. 31.
    Shin Y (2007) Facial expression recognition based on emotion dimensions on manifold learning. In: International conference on computational science. Springer, Berlin, pp 81–88Google Scholar
  32. 32.
    Soyel H, Demirel H (2007) Facial expression recognition using 3d facial feature distances. Lect Notes Comput Sci 4633:831–838CrossRefGoogle Scholar
  33. 33.
    Stoiber N, Seguier R, Breton G (2009) Automatic design of a control interface for a synthetic face. In: Proceedings of the 13th international conference on intelligent user interfaces, pp 207–216Google Scholar
  34. 34.
    Tang H, Huang T (2008) 3D facial expression recognition based on automatically selected features. In: IEEE international conference on computer vision and pattern recognition, pp 1–8Google Scholar
  35. 35.
    Wallhoff F (2006) Facial expressions and emotion database., Technische Universität München
  36. 36.
    Whissell C (1989) The dictionary of affect in language. In: Emotion: theory, research and experience. The measurement of emotions, vol 4. Academic, New YorkGoogle Scholar
  37. 37.
    Whissell C (2000) Whissell’s dictionary of affect in language technical manual and user’s guide. Accessed 18 Dec 2010
  38. 38.
    Whitehill J, Bartlett M, Movellan J (2008) Automated teacher feedback using facial expression recognition. In: Workshop on CVPR for human communicative behavior analysis, IEEE conference on computer vision and pattern recognition Google Scholar
  39. 39.
    Witten I, Frank E (2005) Data Mining: practical machine learning tools and techniques, 2nd edn. Morgan Kaufmann, San FranciscozbMATHGoogle Scholar
  40. 40.
    Wu Y, Liu H, Zha H (2005) Modeling facial expression space for recognition. In: 2005 IEEE/RSJ International conference on intelligent robots and systems (IROS 2005), pp 1968–1973Google Scholar
  41. 41.
    Yeasin M, Bullot B, Sharma R (2006) Recognition of facial expressions and measurement of levels of interest from video. IEEE Trans Multimedia 8:500–508CrossRefGoogle Scholar
  42. 42.
    Zeng Z, Pantic M, Roisman G, Huang T (2009) A survey of affect recognition methods: audio, visual, and spontaneous expressions. IEEE Trans Pattern Anal Mach Intell 39–58Google Scholar
  43. 43.
    Zhang Y, Ji Q (2005) Active and dynamic information fusion for facial expression understanding from image sequences. IEEE Trans Pattern Anal Mach Intell 699–714Google Scholar
  44. 44.
    Zhang Z, Lyons M, Schuster M, Akamatsu S (1998) Comparison between geometry-based and gabor-wavelets-based facial expression recognition using multi-layer perceptron. In: Proceedings of the 3rd international conference on face and gesture recognition, pp 454–459Google Scholar

Copyright information

© Springer-Verlag London Limited 2012

Authors and Affiliations

  • Isabelle Hupont
    • 1
    Email author
  • Sandra  Baldassarri
    • 2
  • Eva Cerezo
    • 2
  1. 1.Multimedia Technologies DivisionAragon Institute of TechnologyZaragozaSpain
  2. 2.Computer Science and Systems Engineering Department, Engineering Research Institute of Aragon (I3A)University of ZaragozaZaragozaSpain

Personalised recommendations