Skip to main content

Advertisement

SpringerLink
Log in

A hybrid framework for smile detection in class imbalance scenarios

  • Original Article
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

In this study, we consider the problem of smile detection in both an imbalanced data scenario, in which the number of smile images is in the minority compared with the number of neutral images, and a balanced data scenario. We first propose a smile detection model using a convolutional neural network (SD-CNN) to improve the performance in the balanced data scenario, and then a hybrid deep learning framework (HF-SD) that uses a modification of the SD-CNN model to learn and then extracts the features from dataset. These extracted features are then used to train an extreme gradient boosting approach to handle the imbalanced problem. An experiment shows that the proposed model has impressive discriminative ability for smile detection, in both balanced and imbalanced data scenarios, compared with existing approaches. HF-SD yields an accuracy of 93.6% and outperforms the state-of-the-art approaches for the original GENK14K database in the balanced data scenario. The results of the second experiment show that HF-SD also achieves better AUCs (area under the receiver operating characteristic curve) compared with the state-of-the-art methods for smile detection in an imbalanced data scenario with different balancing ratios.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

References

  1. Ali M, Le HS, Khan M, Nguyen TT (2018) Segmentation of dental X-ray images in medical imaging using neutrosophic orthogonal matrices. Expert Syst Appl 91:434–441

    Article  Google Scholar 

  2. An L, Yang S, Bhanu B (2015) Efficient smile detection by extreme learning machine. Neurocomputing 149:354–363

    Article  Google Scholar 

  3. Bianco S, Celona L, Schettini R (2016) Robust smile detection using convolutional neural networks. J Electron Imaging 25(6):063002

    Article  Google Scholar 

  4. Chen J, Ou Q, Chi Z, Fu H (2017) Smile detection in the wild with deep convolutional neural networks. Mach Vis Appl 28(1–2):173–183

    Article  Google Scholar 

  5. Chen T, Guestrin T (2016) XGBoost: a scalable tree boosting system. In: KDD, pp 785–794

  6. Chen Y, Zou W, Tang Y, Li X, Xu C, Komodakis N (2018) SCOM: spatiotemporal constrained optimization for salient object detection. IEEE Trans Image Process 27(7):3345–3357

    Article  MathSciNet  Google Scholar 

  7. Dinh VS, Le TBC, Do PT (2017) Facial smile detection using convolutional neural networks. In: KSE’17, pp 136–141

  8. Galar M, Fernández A, Tartas EB, Sola HB, Herrera F (2012) A review on ensembles for the class imbalance problem: bagging-, boosting-, and hybrid-based approaches. IEEE Trans Syst Man Cybern Part C 42(4):463–484

    Article  Google Scholar 

  9. Gao Y, Liu H, Wu P, Wang C (2016) A new descriptor of gradients self-similarity for smile detection in unconstrained scenarios. Neurocomputing 174:1077–1086

    Article  Google Scholar 

  10. Han D, Liu Q, Fan W (2018) A new image classification method using CNN transfer learning and web data augmentation. Expert Syst Appl 95:43–56

    Article  Google Scholar 

  11. He H, Garcia EA (2009) Learning from imbalanced data. IEEE Trans Knowl Data Eng 21(9):1263–1284

    Article  Google Scholar 

  12. Hu L, Ni Q (2018) IoT-driven automated object detection algorithm for urban surveillance systems in smart cities. IEEE Internet Things J 5(2):747–754

    Article  Google Scholar 

  13. Ioffe S, Szegedy C (2015) Batch normalization: Accelerating deep network training by reducing internal covariate shift. In: ICML’15, pp 448–456

  14. King DE (2009) Dlib-ml: a machine learning toolkit. J Mach Learn Res 10:1755–1758

    Google Scholar 

  15. Kingma DP, Ba J (2015) Adam: a method for stochastic optimization. In: Proceedings of the 3rd international conference on learning representations (ICLR)

  16. Le A, Yang S, Bhanu B (2015) Efficient smile detection by extreme learning machine. Neurocomputing 149:354–363

    Article  Google Scholar 

  17. Le HS, Pham HT (2017) Some novel hybrid forecast methods based on picture fuzzy clustering for weather nowcasting from satellite image sequences. Appl Intell 46(1):1–15

    Article  MathSciNet  Google Scholar 

  18. Le HS, Tran MT, Fujita H, Dey N, Ashour AS, Vo TNN, Le QA, Chu DT (2018) Dental diagnosis from x-ray images: an expert system based on fuzzy computing. Biomed Signal Process Control 39:64–73

    Article  Google Scholar 

  19. LeCun Y, Bengio Y, Hinton G (2015) Deep learning. Nature 521:436–444

    Article  Google Scholar 

  20. Li H, Lin Z, Shen X, Brandt J, Hua G (2015) A convolutional neural network cascade for face detection. In: CVPR, pp 5325–5334

  21. Liu M, Li S, Shan S, Chen X (2012) Enhancing expression recognition in the wild with unlabeled reference data. In: Asian conference on computer vision (ACCV), pp 577–588

    Chapter  Google Scholar 

  22. Lopes AT, Aguiar E, Souza AFD, Oliveira-Santos T (2017) Facial expression recognition with convolutional neural networks: coping with few data and the training sample order. Pattern Recognit 61:610–628

    Article  Google Scholar 

  23. Parka BH, Oha SY, Kim IJ (2017) Face alignment using a deep neural network with local feature learning and recurrent regression. Expert Syst Appl 89:66–80

    Article  Google Scholar 

  24. Peng H, Li B, Ling H, Hu W, Xiong W, Maybank SJ (2017) Salient object detection via structured matrix decomposition. IEEE Trans Pattern Anal Mach Intell 39(4):818–832

    Article  Google Scholar 

  25. Raza M, Chen Z, Rehman SU, Wang P, Bao P (2018) Appearance based pedestrians’ head pose and body orientation estimation using deep learning. Neurocomputing 272:647–659

    Article  Google Scholar 

  26. Ren S, He K, Girshick RB, Sun J (2017) Faster R-CNN: towards real-time object detection with region proposal networks. IEEE Trans Pattern Anal Mach Intell 39(6):1137–1149

    Article  Google Scholar 

  27. Ryu B, Rivera AR, Kim J, Chae O (2017) Local directional ternary pattern for facial expression recognition. IEEE Trans Image Process 26(12):6006–6018

    Article  MathSciNet  Google Scholar 

  28. Shan C (2012) Smile detection by boosting pixel differences. IEEE Trans Image Process 21(1):431–436

    Article  MathSciNet  Google Scholar 

  29. Simonyan K, Zisserman A (2015) Very deep convolutional networks for large-scale image recognition. In: ICLR

  30. Vo T, Nguyen T, Le CT (2018) Race recognition using deep convolutional neural networks. Symmetry 10(11):564

    Article  Google Scholar 

  31. Wei Y, Xia W, Lin M, Huang J, Ni B, Dong J, Zhao Y, Yan S (2016) HCP: a flexible CNN framework for multi-label image classification. IEEE Trans Pattern Anal Mach Intell 38(9):1901–1907

    Article  Google Scholar 

  32. Whitehill J, Littlewort G, Fasel I, Bartlett M, Movellan J (2009) Toward practical smile detection. IEEE Trans Pattern Anal Mach Intell 31:2106–2111

    Article  Google Scholar 

  33. Zhang Y, Zhang E, Chen W (2016) Deep neural network for halftone image classification based on sparse auto-encoder. Eng Appl Artif Intell 50:245–255

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Institute of Research and Development, Duy Tan University, Da Nang, 550000, Vietnam

    Thanh Vo

  2. Faculty of Information Technology, Ho Chi Minh City Open University, Ho Chi Minh City, Vietnam

    Trang Nguyen

  3. Faculty of Information Technology, Ho Chi Minh City University of Technology (HUTECH), Ho Chi Minh City, Vietnam

    C. T. Le

Authors
  1. Thanh Vo
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Trang Nguyen
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. C. T. Le
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to C. T. Le.

Ethics declarations

Conflict of interest

The authors declare no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Vo, T., Nguyen, T. & Le, C.T. A hybrid framework for smile detection in class imbalance scenarios. Neural Comput & Applic 31, 8583–8592 (2019). https://doi.org/10.1007/s00521-019-04089-w

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-019-04089-w

Keywords

Search

Navigation