Smile detection using hybrid face representation


Smile detection has attracted considerable amount of research interests in the domain of computer vision. It possesses several potential applications in gaming, human-to-computer and human-to-human interaction. This paper investigates the challenging problem of smile detection from face images acquired under unconstrained conditions. First, a locally weighted multiblock shape-texture descriptor is proposed to extract detailed local and global information from faces with diverse variations such as orientation, illumination, pose, and occlusion. The proposed technique combines the robustness of pyramid histogram of oriented gradient and local binary pattern for image feature representation using an adaptive local weight assignment. By locally weighting the descriptors from very dense patches of the image, we induce discriminating local spatial context to the distribution of the descriptions from the face image. Second, in order to minimize redundancy and extract the most relevant facial information from the feature vectors, a correlation based filter feature selection approach is adopted. Finally, kernel based classifiers such as support vector machine and kernel extreme learning machine are utilized for performing classification. Based on our findings, the proposed framework provides very competitive detection rate to related approaches that have exploited image alignment as an important stage for improving performance of smile detection.

This is a preview of subscription content, access via your institution.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9


  1. Ahonen T, Hadid A, Pietikainen M (2006) Face description with local binary patterns: application to face recognition. IEEE Trans Pattern Anal Mach Intell 28(12):2037–2041

    Article  MATH  Google Scholar 

  2. An L, Yang S, Bhanu B (2015) Efficient smile detection by extreme learning machine. Neurocomputing 149:354–363

    Article  Google Scholar 

  3. Arigbabu OA, Ahmad SMS, Adnan WAW, Yussof S, Iranmanesh V, Malallah FL (2014) Gender recognition on real world faces based on shape representation and neural network. In: Proceedings of 2014 IEEE international conference on computer and information sciences (ICCOINS). pp 1–5

  4. Bosch A, Zisserman A, Munoz X (2007) Representing shape with a spatial pyramid kernel. In: Proceedings of the 6th ACM international conference on image and video retrieval. pp 401–408

  5. Cohen I, Sebe N, Garg A, Chen LS, Huang TS (2003) Facial expression recognition from video sequences: temporal and static modeling. Comput Vis Image Underst 91(1–2):160–187

    Article  Google Scholar 

  6. Cortes C, Vapnik V (1995) Support-vector networks. Mach Learn 297:273–297

    MATH  Google Scholar 

  7. Deniz O, Castrillon M, Lorenzo J, Anton L, Bueno G (2008) Smile detection for user interfaces. In: Proceedings of international symposium on advances in visual computing. pp 602–611

  8. Ding S, Zhao H, Zhang Y, Xu X, Nie R (2013) Extreme learning machine: algorithm, theory and applications. Artif Intell Rev 44(1):103–115

    Article  Google Scholar 

  9. Ekman P, Friesen WV (1978) Action coding system: a technique for the measurement of facial movement. Consulting Psychologists Press, San Francisco 14

    Google Scholar 

  10. Ekman P, Davidson RJ, Friesen WV (1990) The Duchenne smile: emotional expression and brain physiology. II. J Pers Soc Psychol 58(2):342–353

    Article  Google Scholar 

  11. Freire D, Castrillón SM, Déniz-Suárez O (2009) A novel approach for smile detection combining Ulbp and Pca. In: Proceedings of the EUROCAST

  12. Girard JM, Cohn JF, De la Torre F (2014) Estimating smile intensity: a better way. Pattern Recogn Lett 000:1–9

    Google Scholar 

  13. Huang GB (2014) An insight into extreme learning machines: random neurons, random features and kernels. Cogn Comput 6(3):376–390

    Article  Google Scholar 

  14. Huang G-B, Zhu Q-Y, Siew C-K (2006) Extreme learning machine: theory and applications. Neurocomputing 70(1–3):489–501

    Article  Google Scholar 

  15. Huang GB, Mattar M, Berg T, Learned-Miller E (2007) Labeled faces in the wild: a database for studying face recognition in unconstrained environments. University of Massachusetts, Amherst, Technical Report

  16. Liu C, Yuen J, Torralba A (2011) SIFT flow: dense correspondence across scenes and its applications. IEEE Trans Pattern Anal Mach Intell 33(5):978–994

    Article  Google Scholar 

  17. Ojala T, Pietikäinen M, Mäenpää T (2002) Multiresolution gray-scale and rotation invariant texture classification with local binary patterns. IEEE Trans Pattern Anal Mach Intell 24(7):971–987. doi:10.1109/TPAMI.2002.1017623

    Article  MATH  Google Scholar 

  18. Pantic M, Rothkrantz LJM (2000) Automatic analysis of facial expressions: the state of the art. IEEE Trans Pattern Anal Mach Intell 22(12):1424–1445

    Article  Google Scholar 

  19. Peng H, Long F, Ding C (2005) Feature selection based on mutual information criteria of max-dependency, max-relevance, and min-redundancy. IEEE Trans Pattern Anal Mach Intell 27(8):1226–1238

    Article  Google Scholar 

  20. Sanderson C, Lovell BC (2009) Multi-region probabilistic histograms for robust and scalable identity inference. In: Proceedings of 2009 international conference on biometrics. Lecture notes in computer science vol 5558. pp 199–208

  21. Shan C (2012) Smile detection by boosting pixel differences. IEEE Trans Image Process 21(1):431–436

    MathSciNet  Article  Google Scholar 

  22. Tsanas A, Little MA, Mcsharry PE (2010) A simple filter benchmark for feature selection. J Mach Learn Res 1–24

  23. Viola P, Jones MJ (2004) Robust real-time face detection. Int J Comput Vis 57(2):137–154

    Article  Google Scholar 

  24. Whitehill J, Littlewort G, Fasel I, Bartlett M, Movellan J (2009) Towards practical smile detection. IEEE Trans Pattern Anal Mach Intell 31(11):2106–2111

    Article  Google Scholar 

  25. Yang H, Lin W-Y, Lu J (2014) DAISY filter flow: a generalized discrete approach to dense correspondences. In: 2014 IEEE conference on computer vision and pattern recognition. pp 3406–3413

Download references


We would like to acknowledge Malaysian Ministry of Higher Education for the provision of Exploratory Research Grant Schemes, through which this research was made possible.

Author information



Corresponding author

Correspondence to Olasimbo Ayodeji Arigbabu.

Ethics declarations

This work complies with the ethical standards.

Conflict of interest

The authors declare that they have no conflict of interest.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Arigbabu, O.A., Mahmood, S., Ahmad, S.M.S. et al. Smile detection using hybrid face representation. J Ambient Intell Human Comput 7, 415–426 (2016).

Download citation


  • Smile detection
  • Unconstrained face images
  • Local feature descriptors
  • Feature selection
  • Kernel extreme learning machine
  • Support vector machine