Comparison Between Deep Learning Models and Traditional Machine Learning Approaches for Facial Expression Recognition in Ageing Adults

Abstract

Facial expression recognition is one of the most active areas of research in computer vision since one of the non-verbal communication methods by which one understands the mood/mental state of a person is the expression of face. Thus, it has been used in various fields such as human-robot interaction, security, computer graphics animation, and ambient assistance. Nevertheless, it remains a challenging task since existing approaches lack generalizability and almost all studies ignore the effects of facial attributes, such as age, on expression recognition even though the research indicates that facial expression manifestation varies with age. Recently, a lot of progress has been made in this topic and great improvements in classification task were achieved with the emergence of deep learning methods. Such approaches have shown how hierarchies of features can be directly learned from original data, thus avoiding classical hand designed feature extraction methods that generally rely on manual operations with labelled data. However, research papers systematically exploring the performance of existing deep architectures for the task of classifying expression of ageing adults are absent in the literature. In the present work a tentative to try this gap is done considering the performance of three recent deep convolutional neural networks models (VGG-16, AlexNet and GoogLeNet/Inception V1) and evaluating it on four different benchmark datasets (FACES, Lifespan, CIFE, and FER2013 ) which also contain facial expressions performed by elderly subjects. As the baseline, and with the aim of making a comparison, two traditional machine learning approaches based on handcrafted features extraction process are evaluated on the same datasets. Carrying out an exhaustive and rigorous experimentation focused on the concept of “transfer learning”, which consists of replacing the output level of the deep architectures considered with new output levels appropriate to the number of classes (facial expressions), and training three different classifiers (i.e., Random Forest, Support Vector Machine and Linear Regression), VGG-16 deep architecture in combination with Random Forest classifier was found to be the best in terms of accuracy for each dataset and for each considered age-group. Moreover, the experimentation stage showed that the deep learning approach significantly improves the baseline approaches considered, and the most noticeable improvement was obtained when considering facial expressions of ageing adults.

This is a preview of subscription content, access via your institution.

References

  1. [1]

    Zeng Z, Pantic M, Roisman G I, Huang T S. A survey of affect recognition methods: Audio, visual, and spontaneous expressions. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2009, 31(1): 39-58.

    Article  Google Scholar 

  2. [2]

    Pantic M, Rothkrantz L J M. Automatic analysis of facial expressions: The state of the art. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2000, 22(12): 1424-1445.

    Article  Google Scholar 

  3. [3]

    Fasel B, Luettin J. Automatic facial expression analysis: A survey. Pattern Recognition, 2003, 36(1): 259-275.

    MATH  Article  Google Scholar 

  4. [4]

    Carroll J M, Russell J A. Do facial expressions signal specific emotions? Judging emotion from the face in context. Journal of Personality and Social Psychology, 1996, 70(2): 205-218.

    Article  Google Scholar 

  5. [5]

    Rolls E T, Ekman P, Perrett D I et al. Facial expressions of emotion: An old controversy and new findings: Discussion. RSPTB, 335(1273): 69.

    Google Scholar 

  6. [6]

    Shbib R, Zhou S. Facial expression analysis using active shape model. International Journal of Signal Processing, Image Processing and Pattern Recognition, 2015, 8(1): 9-22.

    Article  Google Scholar 

  7. [7]

    Cheon Y, Kim D. Natural facial expression recognition using differential-AAM and manifold learning. Pattern Recognition, 2009, 42(7): 1340-1350.

    MATH  Article  Google Scholar 

  8. [8]

    Soyel H, Demirel H. Facial expression recognition based on discriminative scale invariant feature transform. Electronics Letters, 2010, 46(5): 343-345.

    Article  Google Scholar 

  9. [9]

    Gu W, Xiang C, Venkatesh Y V, Huang D, Lin H. Facial expression recognition using radial encoding of local Gabor features and classifier synthesis. Pattern Recognition, 2012, 45(1): 80-91.

    Article  Google Scholar 

  10. [10]

    Shan C, Gong S, McOwan P W. Facial expression recognition based on local binary patterns: A comprehensive study. Image and Vision Computing, 2009, 27(6): 803-816.

    Article  Google Scholar 

  11. [11]

    Chen J, Chen Z, Chi Z, Fu H. Facial expression recognition based on facial components detection and HOG features. In Proc. the Scientific Cooperations International Workshops on Electrical and Computer Engineering Subfields, Aug. 2014, pp.884-888.

  12. [12]

    Guo G, Guo R, Li X. Facial expression recognition influenced by human ageing. IEEE Transactions on Affective Computing, 2013, 4(3): 291-298.

    Article  Google Scholar 

  13. [13]

    Wang S, Wu S, Gao Z, Ji Q. Facial expression recognition through modeling age-related spatial patterns. Multimedia Tools and Applications, 2016, 75(7): 3937-3954.

    Article  Google Scholar 

  14. [14]

    Malatesta C Z, Izard C E. The facial expression of emotion: Young, middle-aged, and older adult expressions. In Emotion in Adult Development, Malatesta C Z, Izard C E (eds.), Sage Publications, 1984, pp.253-273.

  15. [15]

    Malatesta-Magai C, Jonas R, Shepard B, Culver L C. Type A behavior pattern and emotion expression in younger and older adults. Psychology and Aging, 1992, 7(4): 551-561.

    Article  Google Scholar 

  16. [16]

    Malatesta C Z, Fiore M J, Messina J J. Affect, personality, and facial expressive characteristics of older people. Psychology and Aging, 1987, 2(1): 64-69.

    Article  Google Scholar 

  17. [17]

    Lozano-Monasor E, López M T, Vigo-Bustos F, Fernández- Caballero A. Facial expression recognition in ageing adults: From lab to ambient assisted living. Journal of Ambient Intelligence and Humanized Computing, 2017, 8(4): 567-578.

    Article  Google Scholar 

  18. [18]

    LeCun Y, Bengio Y, Hinton G. Deep learning. Nature, 2015, 521(7553): 436-444.

    Article  Google Scholar 

  19. [19]

    Yu D, Deng L. Deep learning and its applications to signal and information processing [Exploratory DSP]. IEEE Signal Processing Magazine, 2011, 28(1): 145-154.

  20. [20]

    Li S, Deng W. Deep facial expression recognition: A survey. arXiv:1804.08348, 2018. https://arxiv.org/abs/1804.08348, Dec. 2019.

  21. [21]

    Ginne R, Jariwala K. Facial expression recognition using CNN: A survey. International Journal of Advances in Electronics and Computer Science, 2018, 5(3): 13-16.

    Google Scholar 

  22. [22]

    Goodfellow I J, Erhan D, Carrier P L et al. Challenges in representation learning: A report on three machine learning contests. In Proc. the 20th International Conference on Neural Information Processing, Nov. 2013, pp.117-124.

  23. [23]

    Kahou S E, Pal C, Bouthillier X et al. Combining modality specific deep neural networks for emotion recognition in video. In Proc. the 15th ACM on International Conference on Multimodal Interaction, Dec. 2013, pp.543-550.

  24. [24]

    Liu M, Wang R, Li S, Shan S, Huang Z, Chen X. Combining multiple kernel methods on Riemannian manifold for emotion recognition in the wild. In Proc. the 16th International Conference on Multimodal Interaction, Nov. 2014, pp.494-501.

  25. [25]

    Simonyan K, Zisserman A. Very deep convolutional networks for large-scale image recognition. arXiv:1409.1556, 2014. https://arxiv.org/abs/1409.1556, Dec. 2019.

  26. [26]

    Krizhevsky A, Sutskever I, Hinton G E. ImageNet classification with deep convolutional neural networks. In Proc. the 26th Annual Conference on Neural Information Processing Systems, Dec. 2012, pp.1106-1114.

  27. [27]

    Szegedy C, Liu W, Jia Y et al. Going deeper with convolutions. In Proc. the 2015 IEEE Conference on Computer Vision and Pattern Recognition, June 2015, pp.1-9.

  28. [28]

    Viola P, Jones M J. Robust real-time face detection. International Journal of Computer Vision, 2004, 57(2): 137-154.

    Article  Google Scholar 

  29. [29]

    Zuiderveld K. Contrast limited adaptive histogram equalization. In Graphics Gems IV, Heckbert P S (ed.), Academic Press Professional, 1994, pp.474-485.

  30. [30]

    Hubel D H, Wiesel T N. Receptive fields and functional architecture of monkey striate cortex. The Journal of Physiology, 1968, 195(1): 215-243.

    Article  Google Scholar 

  31. [31]

    Pan S J, Yang Q. A survey on transfer learning. IEEE Transactions on Knowledge and Data Engineering, 2010, 22(10): 1345-1359.

    Article  Google Scholar 

  32. [32]

    Russakovsky O, Deng J, Su H et al. ImageNet large scale visual recognition challenge. International Journal of Computer Vision, 2015, 115(3): 211-252.

    MathSciNet  Article  Google Scholar 

  33. [33]

    Lin M, Chen Q, Yan S. Network in network. arXiv:1312.4400, 2013. https://arxiv.org/abs/1312.4400, Dec. 2019.

  34. [34]

    Milborrow S, Nicolls F. Active shape models with SIFT descriptors and MARS. In Proc. the 9th International Conference on Computer Vision Theory and Applications, Jan. 2014, pp.380-387.

  35. [35]

    Shan C, Gong S, McOwan P W. Facial expression recognition based on local binary patterns: A comprehensive study. Image and Vision Computing, 2009, 27(6): 803-816.

    Article  Google Scholar 

  36. [36]

    Ebner N. C, Riediger M, Lindenberger U. FACES — A database of facial expressions in young, middle-aged, and older women and men: Development and validation. Behavior Research Methods, 2010, 42(1): 351-362.

    Article  Google Scholar 

  37. [37]

    Minear M, Park D C. A lifespan database of adult facial stimuli. Behavior Research Methods, Instruments, & Computers, 2004, 36(4): 630-633.

    Article  Google Scholar 

  38. [38]

    Li W, Li M, Su Z, Zhu Z. A deep-learning approach to facial expression recognition with candid images. In Proc. the 14th IAPR International Conference on Machine Vision Applications, May 2015, pp.279-282.

  39. [39]

    Goodfellow I J, Erhan D, Carrier P L et al. Challenges in representation learning: A report on three machine learning contests. In Proc. the 20th International Conference on Neural Information Processing, Nov. 2013, pp.117-124.

  40. [40]

    Wu T, Turaga P, Chellappa R. Age estimation and face verification across ageing using landmarks. IEEE Transactions on Information Forensics and Security, 2012, 7(6): 1780-1788.

    Article  Google Scholar 

  41. [41]

    Giannopoulos P, Perikos I, Hatzilygeroudis I. Deep learning approaches for facial emotion recognition: A case study on FER-2013. In Advances in Hybridization of Intelligent Methods: Models, Systems and Applications, Hatzilygeroudis I, Palade V (eds.), Springer, 2018, pp.1-16.

  42. [42]

    Georgescu M I, Ionescu R T, Popescu M. Local learning with deep and handcrafted features for facial expression recognition. arXiv:1804.10892, 2018. https://arxiv.org/pdf/1804.10892.pdf, Dec. 2019.

  43. [43]

    Abadi M, Barham P, Chen J et al. TensorFlow: A system for large-scale machine learning. In Proc. the 12th USENIX Symposium on Operating Systems Design and Implementation, Nov. 2016, pp.265-283.

  44. [44]

    Caroppo A, Leone A, Siciliano P. Facial expression recognition in ageing adults: A comparative study. In Ambient Assisted Living, Leone A, Caroppo A, Rescio G et al. (eds.), Springer, 2018, pp.349-359.

  45. [45]

    Li W, Tsangouri C, Abtahi F, Zhu Z. A recursive framework for expression recognition: From web images to deep models to game dataset. Machine Vision and Applications, 2018, 29(3): 489-502.

    Article  Google Scholar 

  46. [46]

    Wang X, Wang X, Ni Y. Unsupervised domain adaptation for facial expression recognition using generative adversarial networks. Computational Intelligence and Neuroscience, 2018, Article No. 7208794.

  47. [47]

    Ionescu R T, Popescu M, Grozea C. Local learning to improve bag of visual words model for facial expression recognition. In Proc. the 2013 ICML Workshop on Challenges in Representation Learning, June 2013.

  48. [48]

    Benitez-Quiroz C F, Srinivasan R, Feng Q, Wang Y, Martinez A M. EmotioNet challenge: Recognition of facial expressions of emotion in the wild. arXiv:1703.01210, 2017. https://arxiv.org/abs/1703.01210, Dec. 2019.

  49. [49]

    Mollahosseini A, Hasani B, Mahoor M H. AffectNet: A database for facial expression, valence, and arousal computing in the wild. IEEE Transactions on Affective Computing, 2019, 10(1): 18-31

    Article  Google Scholar 

  50. [50]

    He K, Zhang X, Ren S, Sun J. Deep residual learning for image recognition. In Proc. the 2016 IEEE Conference on Computer Vision and Pattern Recognition, June 2016, pp.770-778.

  51. [51]

    Szegedy C, Vanhoucke V, Ioffe S, Shlens J, Wojna Z. Rethinking the inception architecture for computer vision. In Proc. the 2016 IEEE Conference on Computer Vision and Pattern Recognition, June 2016, pp.2818-2826.

  52. [52]

    Szegedy C, Ioffe S, Vanhoucke V, Alemi A A. Inception-v4, Inception-ResNet and the impact of residual connections on learning. In Proc. the 31st AAAI Conference on Artificial Intelligence, February 2017, pp.4278-4284.

  53. [53]

    Chollet F. Xception: Deep learning with depthwise separable convolutions. In Proc. the 2017 IEEE Conference on Computer Vision and Pattern Recognition, July 2017, pp.1800-1827.

Download references

Author information

Affiliations

Authors

Corresponding author

Correspondence to Andrea Caroppo.

Electronic supplementary material

ESM 1

(PDF 1681 kb)

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Caroppo, A., Leone, A. & Siciliano, P. Comparison Between Deep Learning Models and Traditional Machine Learning Approaches for Facial Expression Recognition in Ageing Adults. J. Comput. Sci. Technol. 35, 1127–1146 (2020). https://doi.org/10.1007/s11390-020-9665-4

Download citation

Keywords

  • computer vision
  • deep learning
  • facial expression
  • machine learning
  • ageing adult