Skip to main content
SpringerLink
Log in

Recognition of facial expressions based on CNN features

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Facial expressions are a natural way to communicate emotional states and intentions. In recent years, automatic facial expression recognition (FER) has been studied due to its practical importance in many human-behavior analysis tasks such as interviews, autonomous-driving, medical treatment, among others. In this paper we propose a method for facial expression recognition based on features extracted with convolutional neural networks (CNN), taking advantage of a pre-trained model in similar tasks. Unlike other approaches, the proposed FER method learns from mixed instances taken from different databases with the goal of improving generalization, a major issue in machine learning. Experimental results show that the FER method is able to recognize the six universal expressions with an accuracy above 92% considering five of the widely used databases. In addition, we have extended our method to deal with micro-expressions recognition (MER). In this regard, we propose three strategies to create a temporal-aggregated feature vector: mean, standard deviation and early fusion. In this case, the best result is 78.80% accuracy. Furthermore, we present a prototype system that implements the two proposed methods for FER and MER as a tool that allows to analyze videos.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

References

  1. Aifanti N, Papachristou C, Delopoulos A (2010) The mug facial expression database. In: 11th International Workshop on image Analysis for Multimedia Interactive Services WIAMIS 10, pp 1–4

  2. Aifanti N, Delopoulos A (2014) Linear subspaces for facial expression recognition. Image Commun 29(1):177–188

    Google Scholar 

  3. Dhall A, Goecke R, Lucey S, Gedeon T (2011) Static facial expression analysis in tough conditions: data, evaluation protocol and benchmark. In: 2011 IEEE International Conference on Computer Vision Workshops (ICCV workshops), pp 2106–2112

  4. Dhall A, Goecke R, Joshi J, Wagner M, Gedeon T (2013) Emotion recognition in the wild challenge 2013. In: Proceedings of the 15th ACM on International Conference on Multimodal Interaction, ICMI ’13, pp 509–516

  5. Donato G, Bartlett MS, Hager JC, Ekman P, Sejnowski TJ (1999) Classifying facial actions. IEEE Trans Pattern Anal Intell 21(10):974–989

    Article  Google Scholar 

  6. Ekman P, Friesen WV (1971) Constants across cultures in the face and emotion. J Pers Soc Psychol 17(2):124–129

    Article  Google Scholar 

  7. Fan R-E, Chang K-W, Hsieh C-J, Wang X-R, Lin C-J (2008) Liblinear: A library for large linear classification. J Mach Learn Res 9:1871–1874

    MATH  Google Scholar 

  8. Goodfellow IJ, Erhan D, Carrier PL, Courville A, Mirza M, Hamner B, Cukierski W, Tang Y, Thaler D, Lee D-H, Zhou Y, Ramaiah C, Feng F, Li R, Wang X, Athanasakis D, Shawe-Taylor J, Milakov M, Park J, Ionescu R, Popescu M, Grozea C, Bergstra J, Xie J, Romaszko L, Xu B, Chuang Z, Bengio Y (2015) Challenges in representation learning A report on three machine learning contexts. Neural Netw 64:59–63

    Article  Google Scholar 

  9. Hall M, Frank E, Holmes G, Pfahringer B, Reutemann P, Witten IH (2009) The weka data mining software: an update. SIGKDD Explorations Newsletter 11(1):10–18

    Article  Google Scholar 

  10. Happy SL, Patnaik P, Routray A, Guha R (2017) The indian spontaneous expression database for emotion recognition. IEEE Trans Affect Comput 8(1):131–142

    Article  Google Scholar 

  11. Jain S, Hu C, Aggarwal JK (2011) Facial expression recognition with temporal modeling of shapes. In: 2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops), pp 1642–1649

  12. Kanade T, Tian Y, Cohn JF (2000) Comprehensive database for facial expression analysis. In: Proceedings of the Fourth IEEE International Conference on Automatic Face and Gesture Recognition 2000, FG ’00, pp 46–53

  13. Lecun Y, Bengio Y, Hinton G (2015) Deep learning. Nature 521(7553):436–444

    Article  Google Scholar 

  14. Lee SH, Plataniotis KN, Ro YM (2014) Intra-class variation reduction using training expression images for sparse representation based facial expression recognition. IEEE Trans Affect Comput 5(3):340–351

    Article  Google Scholar 

  15. Li Qiuyu, Zhan Shu, Liangfeng X u, Congzhong W u (2019) Facial micro-expression recognition based on the fusion of deep learning and enhanced optical flow. Multimedia Tools and Applications 78(20):29307–29322

    Article  Google Scholar 

  16. Li X, Hong X, Moilanen A, Huang X, Pfister T, Zhao G, Pietikäinen M (2018) Towards reading hidden emotions: a comparative study of spontaneous micro-expression spotting and recognition methods. IEEE Trans Affect Comput 9(4):563–577

    Article  Google Scholar 

  17. Li X, Pfister T, Huang X, Zhao G, Pietikäinen M (2013) A spontaneous micro-expression database: Inducement, collection and baseline. In: 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG), pp 1–6

  18. Littlewort G, Whitehill J, Wu T, Fasel I, Frank M, Movellan J, Bartlett M (March 2011) The computer expression recognition toolbox (cert). In: Face and Gesture 2011, pp 298–305

  19. Liu M, Li S, Shan S, Chen X (2013) Au-aware deep networks for facial expression recognition. In: 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG), pp 1–6

  20. Liu M, Li S, Shan S, Wang R, Chen X (2015) Deeply learning deformable facial action parts model for dynamic expression analysis. In: Cremers D, Reid I, Saito H, Yang M-H (eds) Computer Vision ACCV 2014: 12th Asian Conference on Computer Vision, Singapore, Singapore, November 1-5, 2014, Revised Selected Papers, Part IV. Springer International Publishing, Cham, pp 143–157

    Chapter  Google Scholar 

  21. Long J, Zhang N, Darrell T (2014) Do convnets learn correspondence?. In: Proceedings of the 27th International Conference on Neural Information Processing Systems, NIPS’14. MIT Press, Cambridge, pp 1601–1609

  22. Lucey P, Cohn JF, Kanade T, Saragih J, Ambadar Z, Matthews I (2010) The extended cohn-kanade dataset (ck+): A complete dataset for action unit and emotion-specified expression. In: 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Workshops, pages 94–101. IEEE Xplore

  23. Lyons M, Akamatsu S, Kamachi M, Gyoba J (1998) Coding facial expressions with gabor wavelets. In: Proceedings of the 3rd. International Conference on Face & Gesture Recognition, FG ’98, pp 200–206

  24. Mayer C, Eggers M, Radig B (2014) Cross-database evaluation for facial expression recognition. Pattern Recognition and Image Analysis 24(1):124–132

    Article  Google Scholar 

  25. Mollahosseini A, Chan D, Mahoor MH (2016) Going deeper in facial expression recognition using deep neural networks. In: 2016 IEEE Winter Conference on Applications of Computer vision (WACV), IEEE, pp 1–10

  26. Mollahosseini A, Hasani B, Salvador MJ, Abdollahi H, Chan D, Mahoor MH (2016) Facial expression recognition from world wild web. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp 58–65

  27. Pantic M, Valstar M, Rademaker R, Maat L (2005) Web-based database for facial expression analysis. In: 2005 IEEE International Conference on Multimedia and Expo, pp 317–321

  28. Parkhi OM, Vedaldi A, Zisserman A (2015) Deep face recognition. In: Proceedings of the British Machine Vision Conference (BMVC)

  29. Peng M, Wang C, Chen T, Liu G, Fu X (2017) Dual temporal scale convolutional neural network for micro-expression recognition. Front Psychol 8 (1745):1745–1745

    Article  Google Scholar 

  30. Peng X, Xia Z, Li L, Feng X (2016) Towards facial expression recognition in the wild A new database and deep recognition system. In: IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), IEEE, pp 1544–1550

  31. Pfister T, Li X, Zhao G, Pietikäinen M (2011) Recognising spontaneous facial micro-expressions. In: 2011 International Conference on Computer Vision, pp 1449–1456

  32. Polikovsky S, Kameda Y, Ohta Y (2009) Facial micro-expressions recognition using high speed camera and 3d-gradient descriptor. In: 3rd International Conference on Imaging for Crime Detection and Prevention (ICDP 2009), pp 1–6

  33. Sadeghi H, Raie A-A (2019) Human vision inspired feature extraction for facial expression recognition. Multimedia Tools and Applications 78(21):30335–30353

    Article  Google Scholar 

  34. Shan Caifeng, Gong Shaogang, McOwan PW (2009) Facial expression recognition based on local binary patterns A comprehensive study. Image Vis Comput 27(6):803–816

    Article  Google Scholar 

  35. Taheri S, Qiu Q, Chellappa R (2014) Structure-preserving sparse decomposition for facial expression analysis. IEEE Trans Image Process 23(8):3590–3603

    Article  MathSciNet  Google Scholar 

  36. Tang Y (2013) Deep learning using support vector machines. CoRR, arXiv:abs/1306.0239

  37. Vedaldi A, Lenc K (2015) Matconvnet – convolutional neural networks for matlab. In: Proceedings of the 25th Annual ACM International Conference on Multimedia

  38. Viola P, Jones MJ (2004) Robust real-time face detection. Int J Comput Vis 57(2):137–154

    Article  Google Scholar 

  39. Wang S-J, Chen H-L, Yan W-J, Chen Y-H, Fu X (2014) Face recognition and micro-expression recognition based on discriminant tensor subspace analysis plus extreme learning machine. Neural Process Lett 39(1):25–43

    Article  Google Scholar 

  40. Wang X, Feng X, Peng J (2011) A novel facial expression database construction method based on web images. In: Proceedings of the Third International Conference on Internet Multimedia Computing and Service, ICIMCS ’11, pp 124–127

  41. Wolf L, Hassner T, Maoz I (2011) Face recognition in unconstrained videos with matched background similarity. In: CVPR 2011, pp 529–534

  42. Wu W, Yin Y, Wang Y, Wang X, Xu D (2018) Facial expression recognition for different pose faces based on special landmark detection. In: 2018 24Th International Conference on Pattern Recognition (ICPR), pp 1524–1529

  43. Yan W-J, Li X, Wang S-J, Zhao G, Liu Y-J, Chen Y-H, Fu X (2014) Casme ii: an improved spontaneous micro-expression database and the baseline evaluation. PLOS One 9(1):1–8

    Google Scholar 

  44. Yan W-J, Wang S-J, Liu Y-J, Wu Q, Fu X (2014) For micro-expression recognition: Database and suggestions. Neurocomputing 136:82–87

    Article  Google Scholar 

  45. Yan W-J, Wu Q, Liu Y-J, Wang S-J, Fu X (2013) Casme database: A dataset of spontaneous micro-expressions collected from neutralized faces. In: 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG), pp 1–7

  46. Yang B, Cao J, Ni R, Zhang Y (2018) Facial expression recognition using weighted mixture deep neural network based on double-channel facial images. IEEE Access 6:4630–4640

    Article  Google Scholar 

  47. Yu Z, Zhang C (2015) Image based static facial expression recognition with multiple deep network learning. In: Proceedings of the 2015 ACM on International Conference on Multimodal Interaction, ICMI ’15, pp 435–442

  48. Zhang X, Mahoor MH, Mohammad Mavadati S (2015) Facial expression recognition using lp-norm mkl multiclass - svm. Mach Vis Appl 26(4):467–483

    Article  Google Scholar 

Download references

Funding

This work has been supported by the CONACyT with scholarships No. 71150 and 214764. The authors also would like to thank sponsor Red temática en Inteligencia Computacional Aplicada (RedICA).

Author information

Authors and Affiliations

  1. Universidad Autónoma de Baja California (UABC), Ensenada, México

    Sonia M. González-Lozoya & Luis Pellegrin

  2. Universidad Politécnica de Puebla (UPPuebla), Puebla, México

    Jorge de la Calleja, Ma. Auxilio Medina & Antonio Benitez-Ruiz

  3. Instituto Nacional de Astrofísica, Óptica y Electrónica (INAOE), Puebla, México

    Hugo Jair Escalante

Authors
  1. Sonia M. González-Lozoya
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jorge de la Calleja
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Luis Pellegrin
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Hugo Jair Escalante
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Ma. Auxilio Medina
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Antonio Benitez-Ruiz
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Sonia M. González-Lozoya.

Ethics declarations

Conflict of interests

All Authors declare that they have no conflict of interest.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

González-Lozoya, S.M., de la Calleja, J., Pellegrin, L. et al. Recognition of facial expressions based on CNN features. Multimed Tools Appl 79, 13987–14007 (2020). https://doi.org/10.1007/s11042-020-08681-4

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-020-08681-4

Keywords

Search

Navigation