, 43:48 | Cite as

Emotion recognition based on facial components



Machine analysis of facial emotion recognition is a challenging and an innovative research topic in human–computer interaction. Though a face displays different facial expressions, which can be immediately recognized by human eyes, it is very hard for a computer to extract and use the information content from these expressions. This paper proposes an approach for emotion recognition based on facial components. The local features are extracted in each frame using Gabor wavelets with selected scales and orientations. These features are passed on to an ensemble classifier for detecting the location of face region. From the signature of each pixel on the face, the eye and the mouth regions are detected using the ensemble classifier. The eye and the mouth features are extracted using normalized semi-local binary patterns. The multiclass Adaboost algorithm is used to select and classify these discriminative features for recognizing the emotion of the face. The developed methods are deployed on the RML, CK and CMU-MIT databases, and they exhibit significant performance improvement owing to their novel features when compared with the existing techniques.


Semi-local binary patterns Gabor wavelet multiclass Adaboost normalization face detection 


  1. 1.
    Fakhreddine K, Milad A, Jamil A S and Mo N A 2008 Human–computer interaction: overview on state of the art. Int. J. Smart Sens. Intell. Syst. 1(1): 23Google Scholar
  2. 2.
    Cohn J F 2010 Advances in behavioral science using automated facial image analysis and synthesis. IEEE Signal Process. 27(6): 128–133Google Scholar
  3. 3.
    Ekman P and Friesen A 1978 The facial action coding system. San Francisco, CA: Consulting Psychologist PressGoogle Scholar
  4. 4.
    Yun T and Ling G 2013 A deformable 3-D facial expression model for dynamic human emotional state recognition. IEEE Trans. Circuits Syst. Video Technol. 23(1): 142–157CrossRefGoogle Scholar
  5. 5.
    Mingli S, Dacheng T, Zicheng L and Xuelong L 2010 Image ratio features for facial expression recognition application. IEEE Trans. Syst. Man Cybernet. B: Cybernet. 40(3): 779–788Google Scholar
  6. 6.
    Khandait S P, Thool R C and Khandaitm P D 2011 Automatic facial feature extraction and expression recognition based on neural network. Int. J. Adv. Comput. Sci. Appl. 2(1): 113–118Google Scholar
  7. 7.
    Ji Y and Idrissi K 2010 Learning from essential facial parts and local features for automatic facial expression recognition. In: Proceedings of CBMI, 8th International Workshop on Content-Based Multimedia Indexing Google Scholar
  8. 8.
    Shan C, Gong S and McOwan P 2009 Facial expression recognition based on local binary patterns: a comprehensive study. Image Vis. Comput. 27: 803–816CrossRefGoogle Scholar
  9. 9.
    Ahmed Khana R, Alexandre B, Konika H and Bouakaza S 2013 Framework for reliable, real-time facial expression recognition for low resolution images. Pattern Recogn. Lett. 34: 1159–1168CrossRefGoogle Scholar
  10. 10.
    Liu P, Han S and Meng Z 2014 Facial expression recognition via a boosted deep belief network. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1805–1812Google Scholar
  11. 11.
    Zhong L, Liu Q, Yang P, Huang J and Metaxas N 2015 Learning multiscale active facial patches for expression analysis. IEEE Trans. Cybernet. 45(8): 1499–1510CrossRefGoogle Scholar
  12. 12.
    Littlewort G C, Bartlett M S and Lee K 2009 Automatic coding of facial expressions displayed during posed and genuine pain. Image Vis. Comput. 27: 1797–1803CrossRefGoogle Scholar
  13. 13.
    Praseeda Lekshmi V, Sasikumar M and Naveen S 2008 Analysis of facial expressions from video images using PCA. In: Proceedings of WCE, London, UKGoogle Scholar
  14. 14.
    Yi J and Khalid I 2012 Automatic facial expression recognition based on spatiotemporal descriptors. Pattern Recogn. Lett. 33: 1373–1380CrossRefGoogle Scholar
  15. 15.
    Lee T S 1996 Image representation using 2d Gabor wavelets. IEEE Trans. PAMI 18(10): 959–971CrossRefGoogle Scholar
  16. 16.
    Muneeswaran K, Ganesan L, Arumugam S and Ruba Soundar K 2006 Texture image segmentation using combined features from spatial and spectral distribution. Pattern Recogn. Lett. 27: 755–764CrossRefGoogle Scholar
  17. 17.
    Viola P and Jones M 2001 Rapid object detection using a boosted cascade of simple features. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, vol. 1, pp. 511–518Google Scholar
  18. 18.
    Bo W, Haizhou A, Chang H and Shihong L 2004 Fast rotation invariant multi-view face detection based on RealAdaboost. In: Proceedings of the IEEE International Conference on Automatic Face and Gesture Recognition, pp. 79–84Google Scholar
  19. 19.
    Jerome F, Trevor H and Robert T 2000 Additive logistic regression: a statistical view of boosting. Ann. Stat. 28(2): 337–407MathSciNetzbMATHGoogle Scholar
  20. 20.
    Vezhnevets A and Vezhnevets V 2005 Modest AdaBoost—teaching AdaBoost to generalize better. In: Proceedings of Graphicon, Akademgorodok, NovosibirskGoogle Scholar
  21. 21.
    Kyungjoong J, Choi J and Jang G 2015 Semi-local structure patterns for robust face detection. In: IEEE Signal Process. Lett. 22(9): 1400–1403CrossRefGoogle Scholar
  22. 22.
    Ojala T, Pietikainen M and Harwood D 1996 A comparative study of texture measures with classification based on features distributions. Pattern Recogn. 29(1): 51–59CrossRefGoogle Scholar
  23. 23.
    Freund Y and Schapire R 1997 A decision-theoretic generalization of on-line learning and an application to boosting. J. Comput. Syst. Sci. 55: 119–139MathSciNetCrossRefzbMATHGoogle Scholar
  24. 24.
    Saberian M and Vasoncelos J 2011 Multiclass boosting: theory and algorithms. In: Proceedings of Neural Information Processing Systems (NIPS), Granada, Spain, pp. 2124–2132Google Scholar
  25. 25.
    Yongjin W and Ling G 2008 Recognizing human emotional state from audiovisual signals. IEEE Trans. Multimedia 10(5): 659–668Google Scholar
  26. 26.
    Kanade T, Cohn J F and Yingli T 2000 Comprehensive database for facial expression analysis. In: Proceedings of the IEEE International Conference on Automatic Face and Gesture Recognition (FG) Google Scholar
  27. 27.
    Rowley H, Baluja S and Kanade T 1998 Rotation invariant neural network-based face detection. In: Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition Google Scholar

Copyright information

© Indian Academy of Sciences 2018

Authors and Affiliations

  1. 1.Department of Computer Science and EngineeringMepco Schlenk Engineering CollegeSivakasiIndia

Personalised recommendations