Machine Vision and Applications

, Volume 28, Issue 1–2, pp 173–183 | Cite as

Smile detection in the wild with deep convolutional neural networks

  • Junkai ChenEmail author
  • Qihao Ou
  • Zheru Chi
  • Hong Fu
Original Paper


Smile or happiness is one of the most universal facial expressions in our daily life. Smile detection in the wild is an important and challenging problem, which has attracted a growing attention from affective computing community. In this paper, we present an efficient approach for smile detection in the wild with deep learning. Different from some previous work which extracted hand-crafted features from face images and trained a classifier to perform smile recognition in a two-step approach, deep learning can effectively combine feature learning and classification into a single model. In this study, we apply the deep convolutional network, a popular deep learning model, to handle this problem. We construct a deep convolutional network called Smile-CNN to perform feature learning and smile detection simultaneously. Experimental results demonstrate that although a deep learning model is generally developed for tackling “big data,” the model can also effectively deal with “small data.” We further investigate into the discriminative power of the learned features, which are taken from the neuron activations of the last hidden layer of our Smile-CNN. By using the learned features to train an SVM or AdaBoost classifier, we show that the learned features have impressive discriminative ability. Experiments conducted on the GENKI4K database demonstrate that our approach can achieve a promising performance in smile detection.


Smile detection In the wild Deep learning Feature learning Convolution neural network 



The work reported in this paper was supported by a research grant from National Natural Science Foundation of China (project code: 61473243) and a research grant from the Hong Kong Polytechnic University (project code: 4-BCCJ). Junkai Chen would like to acknowledge a postgraduate scholarship from The Hong Kong Polytechnic University.


  1. 1.
    Lyons, M., Akamatsu, S., Kamachi, M., Gyoba, J.: Coding facial expressions with gabor wavelets. In: Third IEEE International Conference on Automatic Face and Gesture Recognition, Proceedings, pp. 200–205 (1998)Google Scholar
  2. 2.
    Lucey, P., Cohn, J.F., Kanade, T., Saragih, J., Ambadar, Z., Matthews, I.: The Extended Cohn-Kanade Dataset (CK+): a complete dataset for action unit and emotion-specified expression. IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) 2010, 94–101 (2010)Google Scholar
  3. 3.
    Hinton, G.E., Osindero, S., Teh, Y.W.: A fast learning algorithm for deep belief nets. Neural Computation 18, 1527–1554 (2006)MathSciNetCrossRefzbMATHGoogle Scholar
  4. 4.
    LeCun, Y., Boser, B., Denker, J.S., Henderson, D., Howard, R.E., Hubbard, W., et al.: Backpropagation applied to handwritten zip code recognition. Neural Computation 1, 541–551 (1989)CrossRefGoogle Scholar
  5. 5.
    Bengio, Y., Lamblin, P., Popovici, D., Larochelle, H.: Greedy layer-wise training of deep networks. In: Advances in Neural Information Processing Systems, pp. 153–160 (2007)Google Scholar
  6. 6.
    Hubel, D.H., Wiesel, T.N.: Receptive fields, binocular interaction and functional architecture in the cat’s visual cortex. The Journal of Physiology 160, 106–154 (1962)CrossRefGoogle Scholar
  7. 7.
    Serre, T., Kreiman, G., Kouh, M., Cadieu, C., Knoblich, U., Poggio, T.: A quantitative theory of immediate visual recognition. Progress in Brain Research 165, 33–56 (2007)CrossRefGoogle Scholar
  8. 8.
    Liu, M., Li, S., Shan, S., Chen, X.: AU-aware Deep Networks for facial expression recognition. In: 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG), pp. 1–6 (2013)Google Scholar
  9. 9.
    Tang, Y.: Deep Learning using Linear Support Vector Machines. arXiv preprint arXiv:1306.0239 (2013)
  10. 10.
    Whitehill, J., Littlewort, G., Fasel, I., Bartlett, M., Movellan, J.: Toward practical smile detection. IEEE Transactions on Pattern Analysis and Machine Intelligence 31, 2106–2111 (2009)CrossRefGoogle Scholar
  11. 11.
    Shan, C., Gritti, T.: Learning discriminative LBP-histogram bins for Facial Expression Recognition. In: BMVC, pp. 1–10 (2008)Google Scholar
  12. 12.
    Moore, S., Bowden, R.: Local binary patterns for multi-view facial expression recognition. Computer Vision and Image Understanding 115, 541–558 (2011)CrossRefGoogle Scholar
  13. 13.
    Orrite, C., Gañán, A., Rogez, G.: HOG-based decision tree for facial expression classification. In: Iberian Conference on Pattern Recognition and Image Analysis. Springer, Berlin, pp. 176–183 (2009)Google Scholar
  14. 14.
    Zhang, Z., Lyons, M., Schuster, M., Akamatsu, S.: Comparison between geometry-based and Gabor-wavelets-based facial expression recognition using multi-layer perceptron. In: Third IEEE International Conference on Automatic Face and Gesture Recognition Proceedings, pp. 454–459 (1998)Google Scholar
  15. 15.
    Rose, N.: Facial expression classification using gabor and log-gabor filters. In: 7th International Conference on Automatic Face and Gesture Recognition, 2006. pp. 346–350 (2006)Google Scholar
  16. 16.
    Shan, C.: Smile detection by boosting pixel differences. IEEE Transactions on Image Processing 21, 431–436 (2012)MathSciNetCrossRefGoogle Scholar
  17. 17.
    Liu, M., Li, S., Shan, S., Chen, X.: Enhancing Expression Recognition in the Wild with Unlabeled Reference Data. In: Computer Vision–ACCV 2012, pp. 577–588 (2012)Google Scholar
  18. 18.
    Jain, V., Crowley, J.L., Lux, A.: Local binary patterns calculated over Gaussian derivative images. In: Presented at the Pattern Recognition (ICPR), 2014 22nd International Conference on, 2014Google Scholar
  19. 19.
    An, L., Yang, S., Bhanu, B.: Efficient smile detection by extreme learning machine. Neurocomputing 149, 354–363 (2015)CrossRefGoogle Scholar
  20. 20.
    Matsugu, M., Mori, K., Mitari, Y., Kaneda, Y.: Subject independent facial expression recognition with robust face detection using a convolutional neural network. Neural Networks 16, 555–559 (2003)CrossRefGoogle Scholar
  21. 21.
    Devries, T., Biswaranjan, K., Taylor, G.W.: Multi-task learning of facial landmarks and expression. Canadian Conference on Computer and Robot Vision (CRV) 2014, 98–103 (2014)Google Scholar
  22. 22.
    Ijjina, E.P., Mohan, C.K.: Facial expression recognition using kinect depth sensor and convolutional neural networks. In: 2014 13th International Conference on Machine Learning and Applications (ICMLA), pp. 392–396 (2014)Google Scholar
  23. 23.
    Glauner, P.O.: Deep convolutional neural networks for smile recognition, arXiv preprint arXiv:1508.06535 (2015)
  24. 24.
    Mavadati, S.M., Mahoor, M.H., Bartlett, K., Trinh, P., Cohn, J.F.: Disfa: a spontaneous facial action intensity database. IEEE Transactions on Affective Computing 4, 151–160 (2013)CrossRefGoogle Scholar
  25. 25.
    Zhang, K., Huang, Y., Wu, H., Wang, L.: Facial smile detection based on deep learning features, Chinese Academy of Sciences Institute of Automation (2015)Google Scholar
  26. 26.
    Rifai, S., Bengio, Y., Courville, A., Vincent, P., Mirza, M.: Disentangling factors of variation for facial expression recognition. In: Computer Vision—ECCV 2012, pp. 808–822 (2012)Google Scholar
  27. 27.
    Kim, Y., Lee, H., Provost, E.M.: Deep learning for robust feature generation in audiovisual emotion recognition. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) 2013, 3687–3691 (2013)Google Scholar
  28. 28.
    Glorot, X., Bordes, A., Bengio, Y.: Deep sparse rectifier neural networks. In: International Conference on Artificial Intelligence and Statistics, pp. 315–323 (2011)Google Scholar
  29. 29.
    Krizhevsky, A., Sutskever, I., Hinton, G.: Imagenet classification with deep convolutional neural networks. Advances in Neural Information Processing Systems 25, 1106–1114 (2012)Google Scholar
  30. 30.
    Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., Salakhutdinov, R.: Dropout: a simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research 15, 1929–1958 (2014)MathSciNetzbMATHGoogle Scholar
  31. 31.
    Vedaldi, A., Lenc, K.: Matconvnet: Convolutional neural networks for matlab. In: Proceedings of the 23rd ACM International Conference on Multimedia, pp. 689–692 (2015)Google Scholar
  32. 32.
    Huang, F.J., LeCun, Y.: Large-scale learning with SVM and convolutional for generic object categorization. IEEE Computer Society Conference on Computer Vision and Pattern Recognition 2006, 284–291 (2006)Google Scholar
  33. 33.
    Lee, H., Grosse, R., Ranganath, R., Ng, A.Y.: Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations. In: Proceedings of the 26th Annual International Conference on Machine Learning, pp. 609–616 (2009)Google Scholar
  34. 34.
    Chang, C.-C., Lin, C.-J.: LIBSVM: a library for support vector machines. In: ACM Transactions on Intelligent Systems and Technology (TIST), vol. 2 (2011)Google Scholar
  35. 35.
    Freund, Y., Schapire, R.E.: A decision-theoretic generalization of on-line learning and an application to boosting. Journal of Computer and System Sciences 55, 119–139 (1997)Google Scholar
  36. 36.
    Liu, Z., Luo, P., Wang, X., Tang, X.: Deep learning face attributes in the wild. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 3730–3738 (2015)Google Scholar
  37. 37.
    LeCun, Y., Bengio, Y., Hinton, G.: Deep learning. Nature 521, 436–444 (2015)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2016

Authors and Affiliations

  1. 1.Department of Electronic and Information EngineeringThe Hong Kong Polytechnic UniversityKowloonHong Kong
  2. 2.PolyU Shenzhen Research InstituteShenzhenChina
  3. 3.Department of Computer ScienceChu Hai College of Higher EducationTuen MunHong Kong

Personalised recommendations