Facial Expression Analysis Based on High Dimensional Binary Features

  • Samira Ebrahimi KahouEmail author
  • Pierre Froumenty
  • Christopher Pal
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8926)


High dimensional engineered features have yielded high performance results on a variety of visual recognition tasks and attracted significant recent attention. Here, we examine the problem of expression recognition in static facial images. We first present a technique to build high dimensional, \(\sim 60\mathrm{k}\) features composed of dense Census transformed vectors based on locations defined by facial keypoint predictions. The approach yields state of the art performance at 96.8% accuracy for detecting facial expressions on the well known Cohn-Kanade plus (CK+) evaluation and 93.2% for smile detection on the GENKI dataset. We also find that the subsequent application of a linear discriminative dimensionality reduction technique can make the approach more robust when keypoint locations are less precise. We go on to explore the recognition of expressions captured under more challenging pose and illumination conditions. Specifically, we test this representation on the GENKI smile detection dataset. Our high dimensional feature technique yields state of the art performance on both of these well known evaluations.


Facial expression recognition Smile detection High-dimensional feature Census transformation Deep learning GENKI CK+ 


  1. 1.
    Ojala, T., Pietikäinen, M., Harwood, D.: A comparative study of texture measures with classification based on featured distributions. Pattern Recognition 29(1), 51–59 (1996)CrossRefGoogle Scholar
  2. 2.
    Ahonen, T., Hadid, A., Pietikainen, M.: Face description with local binary patterns: Application to face recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence 28(12), 2037–2041 (2006)CrossRefGoogle Scholar
  3. 3.
    Chen, D., Cao, X., Wen, F., Sun, J.: Blessing of dimensionality: High-dimensional feature and its efficient compression for face verification. In: Proceedings of the 2013 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2013, pp. 3025–3032. IEEE Computer Society, Washington, DC (2013)Google Scholar
  4. 4.
    Lu, C., Tang, X.: Surpassing human-level face verification performance on LFW with gaussianface. In: Technical report arXiv:1404.3840 (2014)
  5. 5.
    Sikka, K., Dykstra, K., Sathyanarayana, S., Littlewort, G., Bartlett, M.: Multiple kernel learning for emotion recognition in the wild. In: Proceedings of the 15th ACM on International Conference on Multimodal Interaction, ICMI 2013, pp. 517–524. ACM, New York (2013)Google Scholar
  6. 6.
    Kanade, T., Cohn, J., Tian, Y.: Comprehensive database for facial expression analysis. In: Proceedings of the Fourth IEEE International Conference on Automatic Face and Gesture Recognition, pp. 46–53 (2000)Google Scholar
  7. 7.
    Lucey, P., Cohn, J., Kanade, T., Saragih, J., Ambadar, Z., Matthews, I.: The extended cohn-kanade dataset (CK+): a complete dataset for action unit and emotion-specified expression. In: 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 94–101 (2010)Google Scholar
  8. 8.
    Eisert, P., Girod, B.: Analyzing facial expressions for virtual conferencing. IEEE Computer Graphics and Applications, 70–78 (1998)Google Scholar
  9. 9.
    GENKI-4K: The MPLab GENKI Database, GENKI-4K Subset.
  10. 10.
    Zabih, R., Woodfill, J.: Non-parametric local transforms for computing visual correspondence. In: Eklundh, J.-O. (ed.) ECCV 1994. LNCS, vol. 801, pp. 151–158. Springer, Heidelberg (1994) CrossRefGoogle Scholar
  11. 11.
    Sugiyama, M.: Dimensionality reduction of multimodal labeled data by local fisher discriminant analysis. The Journal of Machine Learning Research 8, 1027–1061 (2007)zbMATHGoogle Scholar
  12. 12.
    Dalal, N., Triggs, B.: Histograms of oriented gradients for human detection. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition, CVPR 2005, vol. 1, pp. 886–893 (June 2005)Google Scholar
  13. 13.
    Lowe, D.: Object recognition from local scale-invariant features. In: The Proceedings of the Seventh IEEE International Conference on Computer Vision, vol. 2, pp. 1150–1157 (1999)Google Scholar
  14. 14.
    Shan, C., Gong, S., McOwan, P.W.: Facial expression recognition based on local binary patterns: A comprehensive study. Image Vision Comput. 27(6), 803–816 (2009)CrossRefGoogle Scholar
  15. 15.
    Freund, Y., Schapire, R.E.: A desicion-theoretic generalization of on-line learning and an application to boosting. In: Vitányi, Paul M.B. (ed.) EuroCOLT 1995. LNCS, vol. 904, pp. 23–37. Springer, Heidelberg (1995) CrossRefGoogle Scholar
  16. 16.
    Dahmane, M., Meunier, J.: Emotion recognition using dynamic grid-based HoG features. In: 2011 IEEE International Conference on Automatic Face Gesture Recognition and Workshops (FG 2011), pp. 884–888 (March 2011)Google Scholar
  17. 17.
    Sikka, Karan, Wu, Tingfan, Susskind, Josh, Bartlett, Marian: Exploring bag of words architectures in the facial expression domain. In: Fusiello, Andrea, Murino, Vittorio, Cucchiara, Rita (eds.) ECCV 2012 Ws/Demos, Part II. LNCS, vol. 7584, pp. 250–259. Springer, Heidelberg (2012) Google Scholar
  18. 18.
    Kahou, S.E., Pal, C., Bouthillier, X., Froumenty, P., Gülçehre, c., Memisevic, R., Vincent, P., Courville, A., Bengio, Y., Ferrari, R.C., Mirza, M., Jean, S., Carrier, P.L., Dauphin, Y., Boulanger-Lewandowski, N., Aggarwal, A., Zumer, J., Lamblin, P., Raymond, J.P., Desjardins, G., Pascanu, R., Warde-Farley, D., Torabi, A., Sharma, A., Bengio, E., Côté, M., Konda, K.R., Wu, Z.: Combining modality specific deep neural networks for emotion recognition in video. In: Proceedings of the 15th ACM on International Conference on Multimodal Interaction, ICMI 2013, pp. 543–550. ACM, New York (2013)Google Scholar
  19. 19.
    Tang, Y.: Deep learning using support vector machines. CoRR abs/1306.0239 (2013)Google Scholar
  20. 20.
    Rifai, S., Bengio, Y., Courville, A., Vincent, P., Mirza, M.: Disentangling factors of variation for facial expression recognition. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (eds.) ECCV 2012, Part VI. LNCS, vol. 7577, pp. 808–822. Springer, Heidelberg (2012) CrossRefGoogle Scholar
  21. 21.
    Jeni, L., Takacs, D., Lorincz, A.: High quality facial expression recognition in video streams using shape related information only. In: 2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops), pp. 2168–2174 (November 2011)Google Scholar
  22. 22.
    Jain, V., Crowley, J.: Smile detection using multi-scale gaussian derivatives. In: 12th WSEAS International Conference on Signal Processing, Robotics and Automation, Cambridge, United Kingdom (February 2013)Google Scholar
  23. 23.
    Shan, C.: Smile detection by boosting pixel differences. Trans. Img. Proc. 21(1), 431–436 (2012)CrossRefGoogle Scholar
  24. 24.
    Liu, M., Li, S., Shan, S., Chen, X.: Enhancing expression recognition in the wild with unlabeled reference data. In: Lee, K.M., Matsushita, Y., Rehg, J.M., Hu, Z. (eds.) ACCV 2012, Part II. LNCS, vol. 7725, pp. 577–588. Springer, Heidelberg (2013) CrossRefGoogle Scholar
  25. 25.
    Matthews, I., Baker, S.: Active appearance models revisited. International Journal of Computer Vision 60(2), 135–164 (2004)CrossRefGoogle Scholar
  26. 26.
    Zhu, X., Ramanan, D.: Face detection, pose estimation, and landmark localization in the wild. In: 2012 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2879–2886 (June 2012)Google Scholar
  27. 27.
    Sun, Y., Wang, X., Tang, X.: Deep convolutional network cascade for facial point detection. In: Proceedings of the 2013 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2013, pp. 3476–3483. IEEE Computer Society, Washington, DC (2013)Google Scholar
  28. 28.
    Krizhevsky, A., Sutskever, I., Hinton, G.: Imagenet classification with deep convolutional neural networks. In: NIPS, pp. 1106–1114 (2012)Google Scholar
  29. 29.
    Dhall, A., Goecke, R., Joshi, J., Wagner, M., Gedeon, T.: Emotion recognition in the wild challenge 2013. In: ACM ICMI (2013)Google Scholar
  30. 30.
    Susskind, J., Anderson, A., Hinton, G.: The toronto face database. Technical report, UTML TR 2010–001, University of Toronto (2010)Google Scholar
  31. 31.
    Carrier, P.L., Courville, A., Goodfellow, I.J., Mirza, M., Bengio, Y.: FER-2013 Face Database. Technical report, 1365, Université de Montréal (2013)Google Scholar
  32. 32.
    Chang, C.C., Lin, C.J.: LIBSVM: A library for support vector machines. ACM Transactions on Intelligent Systems and Technology 2, 27:1–27:27 (2011).
  33. 33.
    Whitehill, J., Littlewort, G., Fasel, I., Bartlett, M., Movellan, J.: Toward practical smile detection. IEEE Transactions on Pattern Analysis and Machine Intelligence 31(11), 2106–2111 (2009)CrossRefGoogle Scholar
  34. 34.
    Viola, P., Jones, M.: Rapid object detection using a boosted cascade of simple features. In: Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, CVPR 2001, vol. 1, pp. I-511–I-518 (2001)Google Scholar
  35. 35.
    Štruc, V., Pavešić, N.: Gabor-based kernel partial-least-squares discrimination features for face recognition. Informatica 20(1), 115–138 (2009)zbMATHGoogle Scholar
  36. 36.
    Štruc, V., Pavešić, N.: Photometric normalization techniques for illumination invariance, pp. 279–300. IGI-Global (2011)Google Scholar
  37. 37.
    Dollár, P.: Piotr’s Image and Video Matlab Toolbox (PMT).

Copyright information

© Springer International Publishing Switzerland 2015

Authors and Affiliations

  • Samira Ebrahimi Kahou
    • 1
    Email author
  • Pierre Froumenty
    • 1
  • Christopher Pal
    • 1
  1. 1.École Polytechique de MontréalUniversité de MontréalMontréalCanada

Personalised recommendations