Skip to main content
SpringerLink
Log in

Smile detection in the wild with deep convolutional neural networks

  • Original Paper
  • Published:
Machine Vision and Applications Aims and scope Submit manuscript

Abstract

Smile or happiness is one of the most universal facial expressions in our daily life. Smile detection in the wild is an important and challenging problem, which has attracted a growing attention from affective computing community. In this paper, we present an efficient approach for smile detection in the wild with deep learning. Different from some previous work which extracted hand-crafted features from face images and trained a classifier to perform smile recognition in a two-step approach, deep learning can effectively combine feature learning and classification into a single model. In this study, we apply the deep convolutional network, a popular deep learning model, to handle this problem. We construct a deep convolutional network called Smile-CNN to perform feature learning and smile detection simultaneously. Experimental results demonstrate that although a deep learning model is generally developed for tackling “big data,” the model can also effectively deal with “small data.” We further investigate into the discriminative power of the learned features, which are taken from the neuron activations of the last hidden layer of our Smile-CNN. By using the learned features to train an SVM or AdaBoost classifier, we show that the learned features have impressive discriminative ability. Experiments conducted on the GENKI4K database demonstrate that our approach can achieve a promising performance in smile detection.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

References

  1. Lyons, M., Akamatsu, S., Kamachi, M., Gyoba, J.: Coding facial expressions with gabor wavelets. In: Third IEEE International Conference on Automatic Face and Gesture Recognition, Proceedings, pp. 200–205 (1998)

  2. Lucey, P., Cohn, J.F., Kanade, T., Saragih, J., Ambadar, Z., Matthews, I.: The Extended Cohn-Kanade Dataset (CK+): a complete dataset for action unit and emotion-specified expression. IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) 2010, 94–101 (2010)

    Google Scholar 

  3. Hinton, G.E., Osindero, S., Teh, Y.W.: A fast learning algorithm for deep belief nets. Neural Computation 18, 1527–1554 (2006)

    Article  MathSciNet  MATH  Google Scholar 

  4. LeCun, Y., Boser, B., Denker, J.S., Henderson, D., Howard, R.E., Hubbard, W., et al.: Backpropagation applied to handwritten zip code recognition. Neural Computation 1, 541–551 (1989)

    Article  Google Scholar 

  5. Bengio, Y., Lamblin, P., Popovici, D., Larochelle, H.: Greedy layer-wise training of deep networks. In: Advances in Neural Information Processing Systems, pp. 153–160 (2007)

  6. Hubel, D.H., Wiesel, T.N.: Receptive fields, binocular interaction and functional architecture in the cat’s visual cortex. The Journal of Physiology 160, 106–154 (1962)

    Article  Google Scholar 

  7. Serre, T., Kreiman, G., Kouh, M., Cadieu, C., Knoblich, U., Poggio, T.: A quantitative theory of immediate visual recognition. Progress in Brain Research 165, 33–56 (2007)

    Article  Google Scholar 

  8. Liu, M., Li, S., Shan, S., Chen, X.: AU-aware Deep Networks for facial expression recognition. In: 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG), pp. 1–6 (2013)

  9. Tang, Y.: Deep Learning using Linear Support Vector Machines. arXiv preprint arXiv:1306.0239 (2013)

  10. Whitehill, J., Littlewort, G., Fasel, I., Bartlett, M., Movellan, J.: Toward practical smile detection. IEEE Transactions on Pattern Analysis and Machine Intelligence 31, 2106–2111 (2009)

    Article  Google Scholar 

  11. Shan, C., Gritti, T.: Learning discriminative LBP-histogram bins for Facial Expression Recognition. In: BMVC, pp. 1–10 (2008)

  12. Moore, S., Bowden, R.: Local binary patterns for multi-view facial expression recognition. Computer Vision and Image Understanding 115, 541–558 (2011)

    Article  Google Scholar 

  13. Orrite, C., Gañán, A., Rogez, G.: HOG-based decision tree for facial expression classification. In: Iberian Conference on Pattern Recognition and Image Analysis. Springer, Berlin, pp. 176–183 (2009)

  14. Zhang, Z., Lyons, M., Schuster, M., Akamatsu, S.: Comparison between geometry-based and Gabor-wavelets-based facial expression recognition using multi-layer perceptron. In: Third IEEE International Conference on Automatic Face and Gesture Recognition Proceedings, pp. 454–459 (1998)

  15. Rose, N.: Facial expression classification using gabor and log-gabor filters. In: 7th International Conference on Automatic Face and Gesture Recognition, 2006. pp. 346–350 (2006)

  16. Shan, C.: Smile detection by boosting pixel differences. IEEE Transactions on Image Processing 21, 431–436 (2012)

    Article  MathSciNet  Google Scholar 

  17. Liu, M., Li, S., Shan, S., Chen, X.: Enhancing Expression Recognition in the Wild with Unlabeled Reference Data. In: Computer Vision–ACCV 2012, pp. 577–588 (2012)

  18. Jain, V., Crowley, J.L., Lux, A.: Local binary patterns calculated over Gaussian derivative images. In: Presented at the Pattern Recognition (ICPR), 2014 22nd International Conference on, 2014

  19. An, L., Yang, S., Bhanu, B.: Efficient smile detection by extreme learning machine. Neurocomputing 149, 354–363 (2015)

    Article  Google Scholar 

  20. Matsugu, M., Mori, K., Mitari, Y., Kaneda, Y.: Subject independent facial expression recognition with robust face detection using a convolutional neural network. Neural Networks 16, 555–559 (2003)

    Article  Google Scholar 

  21. Devries, T., Biswaranjan, K., Taylor, G.W.: Multi-task learning of facial landmarks and expression. Canadian Conference on Computer and Robot Vision (CRV) 2014, 98–103 (2014)

    Google Scholar 

  22. Ijjina, E.P., Mohan, C.K.: Facial expression recognition using kinect depth sensor and convolutional neural networks. In: 2014 13th International Conference on Machine Learning and Applications (ICMLA), pp. 392–396 (2014)

  23. Glauner, P.O.: Deep convolutional neural networks for smile recognition, arXiv preprint arXiv:1508.06535 (2015)

  24. Mavadati, S.M., Mahoor, M.H., Bartlett, K., Trinh, P., Cohn, J.F.: Disfa: a spontaneous facial action intensity database. IEEE Transactions on Affective Computing 4, 151–160 (2013)

    Article  Google Scholar 

  25. Zhang, K., Huang, Y., Wu, H., Wang, L.: Facial smile detection based on deep learning features, Chinese Academy of Sciences Institute of Automation (2015)

  26. Rifai, S., Bengio, Y., Courville, A., Vincent, P., Mirza, M.: Disentangling factors of variation for facial expression recognition. In: Computer Vision—ECCV 2012, pp. 808–822 (2012)

  27. Kim, Y., Lee, H., Provost, E.M.: Deep learning for robust feature generation in audiovisual emotion recognition. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) 2013, 3687–3691 (2013)

    Google Scholar 

  28. Glorot, X., Bordes, A., Bengio, Y.: Deep sparse rectifier neural networks. In: International Conference on Artificial Intelligence and Statistics, pp. 315–323 (2011)

  29. Krizhevsky, A., Sutskever, I., Hinton, G.: Imagenet classification with deep convolutional neural networks. Advances in Neural Information Processing Systems 25, 1106–1114 (2012)

    Google Scholar 

  30. Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., Salakhutdinov, R.: Dropout: a simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research 15, 1929–1958 (2014)

    MathSciNet  MATH  Google Scholar 

  31. Vedaldi, A., Lenc, K.: Matconvnet: Convolutional neural networks for matlab. In: Proceedings of the 23rd ACM International Conference on Multimedia, pp. 689–692 (2015)

  32. Huang, F.J., LeCun, Y.: Large-scale learning with SVM and convolutional for generic object categorization. IEEE Computer Society Conference on Computer Vision and Pattern Recognition 2006, 284–291 (2006)

    Google Scholar 

  33. Lee, H., Grosse, R., Ranganath, R., Ng, A.Y.: Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations. In: Proceedings of the 26th Annual International Conference on Machine Learning, pp. 609–616 (2009)

  34. Chang, C.-C., Lin, C.-J.: LIBSVM: a library for support vector machines. In: ACM Transactions on Intelligent Systems and Technology (TIST), vol. 2 (2011)

  35. Freund, Y., Schapire, R.E.: A decision-theoretic generalization of on-line learning and an application to boosting. Journal of Computer and System Sciences 55, 119–139 (1997)

  36. Liu, Z., Luo, P., Wang, X., Tang, X.: Deep learning face attributes in the wild. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 3730–3738 (2015)

  37. LeCun, Y., Bengio, Y., Hinton, G.: Deep learning. Nature 521, 436–444 (2015)

    Article  Google Scholar 

Download references

Acknowledgements

The work reported in this paper was supported by a research grant from National Natural Science Foundation of China (project code: 61473243) and a research grant from the Hong Kong Polytechnic University (project code: 4-BCCJ). Junkai Chen would like to acknowledge a postgraduate scholarship from The Hong Kong Polytechnic University.

Author information

Authors and Affiliations

  1. Department of Electronic and Information Engineering, The Hong Kong Polytechnic University, Kowloon, Hong Kong

    Junkai Chen, Qihao Ou & Zheru Chi

  2. PolyU Shenzhen Research Institute, Shenzhen, China

    Zheru Chi

  3. Department of Computer Science, Chu Hai College of Higher Education, Tuen Mun, Hong Kong

    Hong Fu

Authors
  1. Junkai Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Qihao Ou
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Zheru Chi
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Hong Fu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Junkai Chen.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Chen, J., Ou, Q., Chi, Z. et al. Smile detection in the wild with deep convolutional neural networks. Machine Vision and Applications 28, 173–183 (2017). https://doi.org/10.1007/s00138-016-0817-z

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00138-016-0817-z

Keywords

Search

Navigation