Skip to main content

Smile detection using hybrid face representation


Smile detection has attracted considerable amount of research interests in the domain of computer vision. It possesses several potential applications in gaming, human-to-computer and human-to-human interaction. This paper investigates the challenging problem of smile detection from face images acquired under unconstrained conditions. First, a locally weighted multiblock shape-texture descriptor is proposed to extract detailed local and global information from faces with diverse variations such as orientation, illumination, pose, and occlusion. The proposed technique combines the robustness of pyramid histogram of oriented gradient and local binary pattern for image feature representation using an adaptive local weight assignment. By locally weighting the descriptors from very dense patches of the image, we induce discriminating local spatial context to the distribution of the descriptions from the face image. Second, in order to minimize redundancy and extract the most relevant facial information from the feature vectors, a correlation based filter feature selection approach is adopted. Finally, kernel based classifiers such as support vector machine and kernel extreme learning machine are utilized for performing classification. Based on our findings, the proposed framework provides very competitive detection rate to related approaches that have exploited image alignment as an important stage for improving performance of smile detection.

This is a preview of subscription content, access via your institution.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9


  • Ahonen T, Hadid A, Pietikainen M (2006) Face description with local binary patterns: application to face recognition. IEEE Trans Pattern Anal Mach Intell 28(12):2037–2041

    Article  MATH  Google Scholar 

  • An L, Yang S, Bhanu B (2015) Efficient smile detection by extreme learning machine. Neurocomputing 149:354–363

    Article  Google Scholar 

  • Arigbabu OA, Ahmad SMS, Adnan WAW, Yussof S, Iranmanesh V, Malallah FL (2014) Gender recognition on real world faces based on shape representation and neural network. In: Proceedings of 2014 IEEE international conference on computer and information sciences (ICCOINS). pp 1–5

  • Bosch A, Zisserman A, Munoz X (2007) Representing shape with a spatial pyramid kernel. In: Proceedings of the 6th ACM international conference on image and video retrieval. pp 401–408

  • Cohen I, Sebe N, Garg A, Chen LS, Huang TS (2003) Facial expression recognition from video sequences: temporal and static modeling. Comput Vis Image Underst 91(1–2):160–187

    Article  Google Scholar 

  • Cortes C, Vapnik V (1995) Support-vector networks. Mach Learn 297:273–297

    MATH  Google Scholar 

  • Deniz O, Castrillon M, Lorenzo J, Anton L, Bueno G (2008) Smile detection for user interfaces. In: Proceedings of international symposium on advances in visual computing. pp 602–611

  • Ding S, Zhao H, Zhang Y, Xu X, Nie R (2013) Extreme learning machine: algorithm, theory and applications. Artif Intell Rev 44(1):103–115

    Article  Google Scholar 

  • Ekman P, Friesen WV (1978) Action coding system: a technique for the measurement of facial movement. Consulting Psychologists Press, San Francisco 14

    Google Scholar 

  • Ekman P, Davidson RJ, Friesen WV (1990) The Duchenne smile: emotional expression and brain physiology. II. J Pers Soc Psychol 58(2):342–353

    Article  Google Scholar 

  • Freire D, Castrillón SM, Déniz-Suárez O (2009) A novel approach for smile detection combining Ulbp and Pca. In: Proceedings of the EUROCAST

  • Girard JM, Cohn JF, De la Torre F (2014) Estimating smile intensity: a better way. Pattern Recogn Lett 000:1–9

    Google Scholar 

  • Huang GB (2014) An insight into extreme learning machines: random neurons, random features and kernels. Cogn Comput 6(3):376–390

    Article  Google Scholar 

  • Huang G-B, Zhu Q-Y, Siew C-K (2006) Extreme learning machine: theory and applications. Neurocomputing 70(1–3):489–501

    Article  Google Scholar 

  • Huang GB, Mattar M, Berg T, Learned-Miller E (2007) Labeled faces in the wild: a database for studying face recognition in unconstrained environments. University of Massachusetts, Amherst, Technical Report

  • Liu C, Yuen J, Torralba A (2011) SIFT flow: dense correspondence across scenes and its applications. IEEE Trans Pattern Anal Mach Intell 33(5):978–994

    Article  Google Scholar 

  • Ojala T, Pietikäinen M, Mäenpää T (2002) Multiresolution gray-scale and rotation invariant texture classification with local binary patterns. IEEE Trans Pattern Anal Mach Intell 24(7):971–987. doi:10.1109/TPAMI.2002.1017623

    Article  MATH  Google Scholar 

  • Pantic M, Rothkrantz LJM (2000) Automatic analysis of facial expressions: the state of the art. IEEE Trans Pattern Anal Mach Intell 22(12):1424–1445

    Article  Google Scholar 

  • Peng H, Long F, Ding C (2005) Feature selection based on mutual information criteria of max-dependency, max-relevance, and min-redundancy. IEEE Trans Pattern Anal Mach Intell 27(8):1226–1238

    Article  Google Scholar 

  • Sanderson C, Lovell BC (2009) Multi-region probabilistic histograms for robust and scalable identity inference. In: Proceedings of 2009 international conference on biometrics. Lecture notes in computer science vol 5558. pp 199–208

  • Shan C (2012) Smile detection by boosting pixel differences. IEEE Trans Image Process 21(1):431–436

    Article  MathSciNet  Google Scholar 

  • Tsanas A, Little MA, Mcsharry PE (2010) A simple filter benchmark for feature selection. J Mach Learn Res 1–24

  • Viola P, Jones MJ (2004) Robust real-time face detection. Int J Comput Vis 57(2):137–154

    Article  Google Scholar 

  • Whitehill J, Littlewort G, Fasel I, Bartlett M, Movellan J (2009) Towards practical smile detection. IEEE Trans Pattern Anal Mach Intell 31(11):2106–2111

    Article  Google Scholar 

  • Yang H, Lin W-Y, Lu J (2014) DAISY filter flow: a generalized discrete approach to dense correspondences. In: 2014 IEEE conference on computer vision and pattern recognition. pp 3406–3413

Download references


We would like to acknowledge Malaysian Ministry of Higher Education for the provision of Exploratory Research Grant Schemes, through which this research was made possible.

Author information

Authors and Affiliations


Corresponding author

Correspondence to Olasimbo Ayodeji Arigbabu.

Ethics declarations

This work complies with the ethical standards.

Conflict of interest

The authors declare that they have no conflict of interest.

Rights and permissions

Reprints and Permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Arigbabu, O.A., Mahmood, S., Ahmad, S.M.S. et al. Smile detection using hybrid face representation. J Ambient Intell Human Comput 7, 415–426 (2016).

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: