Machine Vision and Applications

, Volume 29, Issue 3, pp 489–502 | Cite as

A recursive framework for expression recognition: from web images to deep models to game dataset

  • Wei Li
  • Christina Tsangouri
  • Farnaz Abtahi
  • Zhigang Zhu
Original Paper


In this paper, we propose a recursive framework to recognize facial expressions from images in real scenes. Unlike traditional approaches that typically focus on developing and refining algorithms for improving recognition performance on an existing dataset, we integrate three important components in a recursive manner: facial dataset generation, facial expression recognition model building, and interactive interfaces for testing and new data collection. To start with, we first create candid images for facial expression (CIFE) dataset. We then apply a convolutional neural network (CNN) to CIFE and build a CNN model for web image expression classification. In order to increase the expression recognition accuracy, we also fine-tune the CNN model and thus obtain a better CNN facial expression recognition model. Based on the fine-tuned CNN model, we design a facial expression game engine and collect a new and more balanced dataset, GaMo. The images of this dataset are collected from the different expressions our game users make when playing the game. Finally, we run yet another recursive step—a self-evaluation of the quality of the data labeling and propose a self-cleansing mechanism for improve the quality of the data. We evaluate the GaMo and CIFE datasets and show that our recursive framework can help build a better facial expression model for dealing with real scene facial expression tasks.


Facial computing Deep learning Data collection Self-updating 



The first author would also like to thank IBM China Research Laboratory for the summer internship that enables the collection of the CIFE dataset. Special thanks to Ms. Celina M. Cavalluzzi, Director of Day Services, GoodWill, for her assistance in evaluating the game applications by adults with autism spectrum disorders.

Supplementary material

138_2017_904_MOESM1_ESM.avi (111.4 mb)
Supplementary material 1 (avi 114067 KB)
138_2017_904_MOESM2_ESM.avi (41.8 mb)
Supplementary material 2 (avi 42753 KB)


  1. 1.
    Li, W., Li, M., Su, Z., Zhu, Z.: A deep-learning approach to facial expression recognition with candid images. In: 2015 14th IAPR International Conference on Machine Vision Applications (MVA), pp. 279–282. IEEE (2015)Google Scholar
  2. 2.
    Li, W., Abtahi, F., Zhu, Z.: A deep feature based multi-kernel learning approach for video expression recognition. In: Proceedings of the 2015 ACM on International Conference on Multimodal Interaction, pp. 483–490. ACM (2015)Google Scholar
  3. 3.
    Li, W., Farnaz A., Tsangouri, C., Zhu Z.: Towards an “in-the-wild” emotion dataset using a game-based framework. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 75–83 (2016)Google Scholar
  4. 4.
    Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014)
  5. 5.
    Mavadati, S.M., Mahoor, M.H., Bartlett, K., Trinh, P., Cohn, J.F.: Disfa: a spontaneous facial action intensity database. IEEE Trans. Affect. Comput. 4(2), 151–160 (2013)CrossRefGoogle Scholar
  6. 6.
    Kanade, T., Cohn, J.F., Tian, Y.: Comprehensive database for facial expression analysis. In: 4th IEEE International Conference on Automatic Face and Gesture Recognition, 2000. Proceedings, pp. 46–53. IEEE (2000)Google Scholar
  7. 7.
    Cohn, J.F., Ambadar, Z., Ekman, P.: Observer-based measurement of facial expression with the facial action coding system. In: The Handbook of Expression Elicitation and Assessment, pp. 203–221 (2007)Google Scholar
  8. 8.
    Lucey, P., Cohn, J.F., Kanade, T., Saragih, J., Ambadar, Z., Matthews, I.: The extended cohn-kanade dataset (ck+): a complete dataset for action unit and expression-specified expression. In: 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 94–101. IEEE (2010)Google Scholar
  9. 9.
    Pantic, M., Valstar, M., Rademaker, R., Maat, L.: Web-based database for facial expression analysis. In: IEEE International Conference on Multimedia and Expo, 2005. ICME 2005, p. 5. IEEE (2005)Google Scholar
  10. 10.
    Shan, C., Gong, S., McOwan, P.W.: Facial expression recognition based on local binary patterns: a comprehensive study. Image Vis. Comput. 27(6), 803–816 (2009)CrossRefGoogle Scholar
  11. 11.
    Xiao, R., Zhao, Q., Zhang, D., Shi, P.: Facial expression recognition on multiple manifolds. Pattern Recognit. 44(1), 107–116 (2011)CrossRefzbMATHGoogle Scholar
  12. 12.
    Wang, Z., Wang, S., Ji, Q.: Capturing complex spatio-temporal relations among facial muscles for facial expression recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3422–3429 (2013)Google Scholar
  13. 13.
    Sikka, K., Dhall, A., Bartlett, M.: Exemplar hidden Markov models for classification of facial expressions in videos. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 18–25 (2015)Google Scholar
  14. 14.
    Taigman, Y., Yang, M., Ranzato, M.A., Wolf, L.: Deepface: closing the gap to human-level performance in face verification. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1701–1708 (2014)Google Scholar
  15. 15.
    Sun, Y., Wang, X., Tang, X.: Deep learning face representation from predicting 10,000 classes. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1891–1898 (2014)Google Scholar
  16. 16.
    Liu, P., Han, S., Meng, Z., Tong, Y.: Facial expression recognition via a boosted deep belief network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1805–1812 (2014)Google Scholar
  17. 17.
    Kim, Y., Lee, H., Provost, E. M.: Deep learning for robust feature generation in audiovisual expression recognition. In: 2013 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 3687–3691. IEEE (2013)Google Scholar
  18. 18.
    Jung, H., Lee, S., Yim, J., Park, S., Kim, J.: Joint fine-tuning in deep neural networks for facial expression recognition. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2983–2991 (2015)Google Scholar
  19. 19.
    Deng, J., Dong, W., Socher, R., Li, L. J., Li, K., Fei-Fei, L.: Imagenet: a large-scale hierarchical image database. In: IEEE Conference on Computer Vision and Pattern Recognition, 2009. CVPR 2009, pp. 248–255. IEEE (2009)Google Scholar
  20. 20.
    Von Ahn, L., Dabbish, L.: Labeling images with a computer game. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 319–326. ACM (2004)Google Scholar
  21. 21.
    Mouro, A., Magalhes, J.: Competitive affective gaming: winning with a smile. In: Proceedings of the 21st ACM International Conference on Multimedia, pp. 83–92. ACM. Chicago (2013)Google Scholar
  22. 22.
    Deriso, D., Susskind, J., Krieger, L., Bartlett, M.: expression mirror: a novel intervention for autism based on real-time expression recognition. In: Computer Vision-ECCV 2012. Workshops and Demonstrations, pp. 671–674. Springer, Berlin (2012)Google Scholar
  23. 23.
    Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems, pp. 1097–1105 (2012)Google Scholar
  24. 24.
    Girshick, R.: Fast r-cnn. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 1440–1448 (2015)Google Scholar
  25. 25.
    Xie, L., Hong, R., Zhang, B., Tian, Q.: Image classification and retrieval are one. In: Proceedings of the 5th ACM on International Conference on Multimedia Retrieval, pp. 3–10. ACM (2015)Google Scholar

Copyright information

© Springer-Verlag GmbH Germany, part of Springer Nature 2018

Authors and Affiliations

  1. 1.Department of Electrical EngineeringCUNY City CollegeNew YorkUSA
  2. 2.Department of Computer EngineeringCUNY City CollegeNew YorkUSA
  3. 3.Department of Computer ScienceCUNY Graduate CenterNew YorkUSA
  4. 4.Department of Computer ScienceCUNY City College and Graduate CenterNew YorkUSA

Personalised recommendations