Skip to main content

Subject-dependent selection of geometrical features for spontaneous emotion recognition

Abstract

Facial expressions are among the most powerful ways to reveal the emotional state. Therefore, Facial Expression Recognition (FER) has been widely introduced to wide fields of applications, such as security, psychotherapy, neuromarketing, and advertisement. Feature extraction and selection are two essential key issues for the design of efficient FER systems. However, most of the previous studies focused on implementing static feature selection methods. Although these methods have shown promising results, they still present weaknesses, especially when dealing with spontaneous expressions. This is mainly due to the specificity of each face, which makes the facial emotion display differs from one subject to another. To address this problem, we propose a face-based dynamic feature selection of two geometric features sub-classes, namely linear and eccentricity features. This combination provides a better understanding of the facial transformation during the emotion display. Moreover, the suggested selection method takes into consideration the subject’s general facial structure, muscle movements, and head position. The performed experiments, using the CK+ and the DISFA datasets, have showed that the proposed method outperforms the state-of-the-art techniques and maintains superior performance with cross-dataset validation. In fact, the accuracy of facial expression recognition by the proposed method reaches 97.72% and 91,26% on the CK+ and the DISFA datasets, respectively.

This is a preview of subscription content, access via your institution.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

References

  1. Amminger G, Schaefer M, Papageorgiou K, Klier C, Schlogelhofer M, Mossaheb N, Werneck-Rohrer S, Nelson B, Mcgorry P (2012) Emotion recognition in individuals at clinical high-risk for schizophrenia. Schizophr Bull 38(5):1030–1039

    Article  Google Scholar 

  2. Arora M, Kumar M (2021) Autofer: Pca and pso based automatic facial emotion recognition. Multimed Tools Appl 80:3039–3049. https://doi.org/10.1007/s11042-020-09726-4

    Article  Google Scholar 

  3. Bandrabur A, Florea L, Florea C, Mancas M (2015) Emotion identification by facial landmarks dynamics analysis. International Conference on Intelligent Computer Communication and Processing (ICCP). https://doi.org/10.1109/ICCP.2015.7312688

  4. Bansal M, Kumar M, Kumar M (2020) Xgboost: 2d-object recognition using shape descriptors and extreme gradient boosting classifier. In: Proceedings of the international conference on computational methods and data engineering, pp 207–222

  5. Bansal M, Kumar M, Kumar M (2021) 2d object recognition: a comparative analysis of sift, surf and orb feature descriptors. Multimed Tools Appl 80:18839–18857. https://doi.org/10.1007/s11042-021-10646-0

    Article  Google Scholar 

  6. Bejaoui H, Ghazouani H, Barhoumi W (2017) Fully automated facial expression recognition using 3d morphable model and mesh-local binary pattern, pp 39–50 https://doi.org/10.1007/978-3-319-70353-4_4

  7. Bejaoui H, Ghazouani H, Barhoumi W (2019) Sparse coding-based representation of lbp difference for 3d/4d facial expression recognition. Multimed Tools Appl 78:22773–22796. https://doi.org/10.1007/s11042-019-7632-2

    Article  Google Scholar 

  8. Ben-Hur A, Weston J (2010) A user’s guide to support vector machines. Methods in molecular biology (Clifton, N.J.) 609:223–39. https://doi.org/10.1007/978-1-60327-241-4_13

    Article  Google Scholar 

  9. Brown G, Pocock A, Zhao M-J, Lujan M (2012) Conditional likelihood maximisation: a unifying framework for information theoretic feature selection. J Mach Learn Res 13(1):27–66

    MathSciNet  MATH  Google Scholar 

  10. Butalia AH, Ingle M, Kulkarni SJ (2012) Facial expression recognition for security. Int J Mod Eng Res Technol 2(4):1449–1453

    Google Scholar 

  11. Candra H, Yuwono M, Chai R, Nguyen HT, Su S (2016) Classification of facial-emotion expression in the application of psychotherapy using viola-jones and edge-histogram of oriented gradient. 2016 38th Annu Int Conf of the IEEE Eng Med Biol Soc 38(5):423–426. https://doi.org/10.1109/EMBC.2016.7590730

    Article  Google Scholar 

  12. Cao N, Ton-That A, Choi H-I (2016) An effective facial expression recognition approach for intelligent game systems. Int J Comput Vis Robot 6(3):223–234. https://doi.org/10.1504/IJCVR.2016.077353

    Article  Google Scholar 

  13. Chen J, Chen D, Gong Y, Yu M, Zhang K, Wang L (2012) Facial expression recognition using geometric and appearance features, pp 29–33. https://doi.org/10.1145/2382336.2382345

  14. Chen M, Cheng J, Zhang Z, Li Y, Zhang Y (2021) Facial expression recognition method combined with attention mechanism. Mobile Information Systems pp 2021. https://doi.org/10.1155/2021/5608340

  15. Chhabra P, Garg NK, Kumar M (2020) Content-based image retrieval system using orb and sift features. Neural Comput Applic 32:2725–2733. https://doi.org/10.1007/s00521-018-3677-9

    Article  Google Scholar 

  16. Dailey M, Cottrell G, Padgett C, Adolphs R (2002) Empath: a neural network that categorizes facial expressions. J Cogn Neurosci, pp 1158–73. https://doi.org/10.1162/089892902760807177

  17. Dalal N, Triggs B (2005) Histograms of oriented gradients for human detection. Int Conf Comp Vision Pattern Recognit (CVPR ’05), Jun 2005, San Diego, United States, pp 886–893

  18. Darwin C (1872) The expression of emotions, vol 19. (1–12), pp 399

  19. Datta S, Sen D, Balasubramanian R (2017) Integrating geometric and textural features for facial emotion classification using svm frameworks, pp 619–628. https://doi.org/10.1007/978-981-10-2104-6_55

  20. Desrosiers P, Daoudi M, Devanne M (2016) Novel generative model for facial expressions based on statistical shape analysis of landmarks trajectories. In: 23rd International Conference on Pattern Recognition (ICPR). https://doi.org/10.1109/ICPR.2016.7899760

  21. Dibeklioğlu H, Salah AA, Gevers T (2015) Recognition of genuine smiles. IEEE Trans Multimed 17(3):279–294

    Article  Google Scholar 

  22. Dibekliouglu H, Salah AA, Gevers T (2012) Are you really smiling at me? spontaneous versus posed enjoyment smiles pp 525–538

  23. Ekman P (2003) Darwin, deception, and facial expression. Ann N Y Acad Sci 1000(1):205–221. https://doi.org/10.1196/annals.1280.010

    Article  Google Scholar 

  24. Ekman P (2009) Telling lies: Clues to deceit in the marketplace, politics and marriage (revised edition)

  25. Ekman P, Friesen WV (1978) Facial action coding system: a technique for the measurement of facial movement

  26. Ekman P, Friesen W (1982) Felt, false, and miserable smiles. J Nonverbal Behav 6(4):238–252. https://doi.org/10.1007/BF00987191

    Article  Google Scholar 

  27. Ekundayo O, Viriri S (2021) Facial expression recognition: a review of trends and techniques. IEEE Access, pp 1–1. https://doi.org/10.1109/ACCESS.2021.3113464

  28. Fernandes J, Matos L, Aragao M (2016) Geometrical approaches for facial expression recognition using support vector machines. 016 29th SIBGRAPI Conference on Graphics, Patterns and Images, pp 347–354. https://doi.org/10.1109/SIBGRAPI.2016.055

  29. Ferreira A, Figueiredo M (2012) Boosting algorithms: a review of methods, theory, and applications. Ensemble Machine Learning: Methods and Applications 3:35–85. https://doi.org/10.1007/978-1-4419-9326-7_2

    Article  Google Scholar 

  30. Fölster M, Hess U, Werheid K (2014) Facial age affects emotional expression decoding. Frontiers in Psychology vol 5. https://doi.org/10.3389/fpsyg.2014.00030

  31. Friedman JH (1996) Another approach to polychotomous classification Department of Statistics, Stanford University, pp 1452–1459

  32. Gharsalli S, Laurent H, Emile B, Desquesnes X (2015) Various fusion schemes to recognize simulated and spontaneous emotions. VISAPP 2015 - 10th International Conference on Computer Vision Theory and Applications; VISIGRAPP, Proceedings vol 2. https://doi.org/10.5220/0005312804240431

  33. Ghazouani H (2021) A genetic programming-based feature selection and fusion for facial expression recognition. Appl Soft Comput 103:107173

    Article  Google Scholar 

  34. Ghimire D, Jeong S, Lee J, Park S (2017) Facial expression recognition based on local region specific features and support vector machines. Multimed Tools Appl, vol 76. https://doi.org/10.1007/s11042-016-3418-y

  35. Gidudu A, Hulley G, Marwala T (2007) Image classification using svms: One-against-one vs one-against-all. arXiv:0711.2914

  36. Goren D, Wilson HR (2006) Quantifying facial expression recognition across viewing conditions. Vis Res 46:1253–1262. https://doi.org/10.1016/j.visres.2005.10.028

    Article  Google Scholar 

  37. Goren D, Wilson HR (2006) Quantifying facial expression recognition across viewing conditions. Vis Res 46(8):1253–1262. https://doi.org/10.1016/j.visres.2005.10.028

    Article  Google Scholar 

  38. Guo H, Zhang X-H, Liang J, Yan W-J (2018) The dynamic features of lip corners in genuine and posed smiles. Front Psychol 9:202. https://doi.org/10.3389/fpsyg.2018.00202

    Article  Google Scholar 

  39. Gupta O, Raviv D, Raskar R (2017) Illumination invariants in deep video expression recognition. Pattern Recognition, pp 76. https://doi.org/10.1016/j.patcog.2017.10.017

  40. Guyon I, Elisseeff A (2003) An introduction to variable and feature selection. J Mach Learn Res 3:1157–1182

    MATH  Google Scholar 

  41. Hall M (2000) Correlation-based feature selection for discrete and numeric class machine learning. In: Proceedings of the Seventeenth International Conference on Machine Learning (ICML 2000), Stanford University, Stanford, CA,USA, pp 359–366

  42. Hall M (2000) Correlation-based feature selection for machine learning. Department of Computer Science

  43. Hamelin N, Moujahid OE, Thaichon P (2017) Emotion and advertising effectiveness: a novel facial expression analysis approach. J Retail Consum Serv 36:103–111. https://doi.org/10.1016/j.jretconser.2017.01.001

    Article  Google Scholar 

  44. Hassaballah M, Saddam Bekhet AAMR, Zhang G (2019) Facial features detection and localization. Recent Adv Comput Vision Stud Comput Intell 804:33–59. https://doi.org/10.1007/978-3-030-03000-1_2

    Article  Google Scholar 

  45. He M, Wang S, Liu Z, Chen X (2013) Analyses of the differences between posed and spontaneous facial expressions. Humaine Association Conference on Affective Computing and Intelligent Interaction, pp 79–84. https://doi.org/10.1109/ACII.2013.20

  46. Horn B, Schunck B (1981) Determining optical flow. Artif Intell 17:185–203. https://doi.org/10.1016/0004-3702(81)90024-2

    Article  Google Scholar 

  47. Hsu C-W , Lin C-J (2002) A comparison of methods for multiclass support vector machines. IEEE Trans Neural Netw 13 (2):415–425. https://doi.org/10.1109/72.991427

    Article  Google Scholar 

  48. Huynh X-P, Kim Y-G (2017) Discrimination between genuine versus fake emotion using long-short term memory with parametric bias and facial landmarks pp 3065–3072

  49. Iqbal MT, Ryu B, Ramirez Rivera A, Makhmudkhujaev F, Chae O, Bae S-H (2020) Facial expression recognition with active local shape pattern and learned-size block representations. IEEE Trans Affect Comput. https://doi.org/10.1109/TAFFC.2020.2995432

  50. Jia S, Wang S, Hu C, Webster PJ, Li X (2021) Detection of genuine and posed facial expressions of emotion: databases and methods. Front Psychol 11:3818. https://doi.org/10.3389/fpsyg.2020.580287

    Article  Google Scholar 

  51. Kazemi V, Sullivan J (2014) One millisecond face alignment with an ensemble of regression trees. In: Comput Vis Pattern Recognit (June 2014, Columbus, Ohio) CVPR’14 IEEE, pp 1867–1874

  52. Kumar M, Chhabra P, Garg NK (2018) An efficient content based image retrieval system using bayesnet and k-nn. Multimed Tools Appl 77:21557–21570. https://doi.org/10.1007/s11042-017-5587-8

    Article  Google Scholar 

  53. Kumar A, Kaur A, Kumar M (2019) Face detection techniques: a review. Artif Intell Rev 52:927–948. https://doi.org/10.1007/s10462-018-9650-2

    Article  Google Scholar 

  54. Kumar A, Kumar M, Kaur A (2021) Face detection in still images under occlusion and non-uniform illumination. Multimed Tools Appl 80:14565–14590. https://doi.org/10.1007/s11042-020-10457-9

    Article  Google Scholar 

  55. Lajevardi S, Hussain Z (2012) Automatic facial expression recognition: Feature extraction and selection. A. Signal, Image and Video Processing 6:159–169. https://doi.org/10.1007/s11760-010-0177-5

    Article  Google Scholar 

  56. Lee S, Baddar W, Ro Y (2016) Collaborative expression representation using peak expression and intra class variation face images for practical subject-independent emotion recognition in videos. Pattern Recognition, vol 54. https://doi.org/10.1016/j.patcog.2015.12.016

  57. Li L, Yuan Y, Li M, Xu H, Li R, Lu S (2019) Subject independent facial expression recognition: Cross-connection and spatial pyramid pooling convolutional neural network. IVSP 2019,: Proceedings of the 2019, International Conference on Image, Video and Signal Processing, pp 85–92. https://doi.org/10.1145/3317640.3317662

  58. Littlewort G, Bartlett M.S, Fasel I, Susskind J, Movellan J. (2006) Dynamics of facial expression extracted automatically from video. Image Vis Comput 24:615–625. https://doi.org/10.1016/j.imavis.2005.09.011

    Article  Google Scholar 

  59. Littlewort G, Frank M, Lainscsek C, Fasel I, Movellan J (2006) Automatic recognition of facial actions in spontaneous expressions. Journal of Multimedia, vol 1(6) https://doi.org/10.4304/jmm.1.6.22-35

  60. Littlewort G, Lainscsek C, Fasel I, Movellan J (2004) Machine learning methods for fully automatic recognition of facial expressions and facial actions. Conf Proc - IEEE Int Conf Syst Man Cybern 1:592–597. https://doi.org/10.1109/ICSMC.2004.1398364

    Article  Google Scholar 

  61. Liu J, Bai M, Jiang N , Cheng R, Li X, Wang Y, Yu D (2021) Interclass interference suppression in multi-class problems. Applied Sciences vol 11. https://doi.org/10.3390/app11010450

  62. Liu P, Han S, Meng Z, Tong Y (2014) Facial expression recognition via a boosted deep belief network. Proc IEEE Conf Comput Vis Pattern Recognit, pp 1805–1812. https://doi.org/10.1109/CVPR.2014.233

  63. Liu M, Li S, Shan S, Chen X. (2015) Au-inspired deep networks for facial expression feature learning. Neurocomputing, vol 159. https://doi.org/10.1016/j.neucom.2015.02.011

  64. Lucey P, Cohn J, Kanade T, Saragih J, Ambadar Z, Matthews I (2010) The extended cohn-kanade dataset (ck+): A complete dataset for action unit and emotion-specified expression. IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Workshops, CVPRW 2010:94–101. https://doi.org/10.1109/CVPRW.2010.5543262

    Article  Google Scholar 

  65. Makhmudkhujaev F, Iqbal MT, Ryu B, Chae O (2019) Local directional-structural pattern for person-independent facial expression recognition. Turk J Elec Eng Comp Sci 27:516–531. https://doi.org/10.3906/elk-1804-58

    Article  Google Scholar 

  66. Mavadati S, Mahoor M, Bartlett K, Trinh P, Cohn J (2013) Disfa: a spontaneous facial action intensity database. IEEE Trans Affect Comput 4(2):151–160. https://doi.org/10.1109/T-AFFC.2013.4

    Article  Google Scholar 

  67. Miao Y-Q, Araujo R, Kamel MS (2012) Cross-domain facial expression recognition using supervised kernel mean matching, vol 2012

  68. Namba S, Makihara S, Kabir R, Miyatani M, Nakao T (2016) Spontaneous facial expressions are different from posed facial expressions: Morphological properties and dynamic sequences. Current Psychology, pp 1–13. https://doi.org/10.1007/s12144-016-9448-9

  69. Novakovic J, Minic M, Veljovic A (2011) Classification accuracy of neural networks with pca in emotion recognition. Theory Appl Math Comput Sci 1:11–16

    Google Scholar 

  70. Park S, Lee K, Lim J-A, Ko H, Kim T, Lee J-I, Kim H, Han S-J, Kim J-S, Park S et al (2020) Differences in facial expressions between spontaneous and posed smiles: automated method by action units and three-dimensional facial landmarks. Sensors 20(4):1199

    Article  Google Scholar 

  71. Peng H, Long F, Ding C (2005) Feature selection based on mutual information: Criteria of max-dependency, max-relevance, and min-redundancy. IEEE Trans Pattern Anal Mach Intell 27:1226–38. https://doi.org/10.1109/TPAMI.2005159

    Article  Google Scholar 

  72. Rabiu H, Saripan MI, Mashohor S, Marhaban MH (2012) 3d facial expression recognition using maximum relevance minimum redundancy geometrical features. EURASIP Journal on Advances in Signal Processing. https://doi.org/10.1186/1687-6180-2012-213

  73. Sadeghi H, Raie A, Mohammadi MR (2013) Facial expression recognition using geometric normalization and appearance representation. Iranian Conference on Machine Vision and Image Processing, MVIP. https://doi.org/10.1109/IranianMVIP.2013.6779970

  74. Sagonas C, Antonakos E, Tzimiropoulos G, Zafeiriou S, Pantic M (2016) 300 faces in-the-wild challenge: database and results. Image Vision Comput 47:3–18. https://doi.org/10.1016/j.imavis.2016.01.002

    Article  Google Scholar 

  75. Sagonas C, Zafeiriou S (2013) Facial point annotations. https://ibug.doc.ic.ac.uk/resources/facial-point-annotations/. Accessed date 24th September 2020

  76. Said C, Haxby J, Todorov A (2011) Brain systems for assessing the affective value of faces. Philosophical transactions of the Royal Society of London. Series B. Biological sciences vol 366. pp 1660–70. https://doi.org/10.1098/rstb.2010.0351

  77. Samadiani N, Huang G, Cai B, Luo W, Chi C-H, Xiang Y, He J (2019) A review on automatic facial expression recognition systems assisted by multimodal sensor data. Sensors (Basel) vol 19(8). https://doi.org/10.3390/s19081863

  78. Saxen F, Werner P, Al-Hamadi A (2017) Real vs. fake emotion challenge: Learning to rank authenticity from facial activity descriptors, pp 3073–3078

  79. Sen D, Datta S, Balasubramanian R (2019) Facial emotion classification using concatenated geometric and textural features. Multimedia Tools and Applications vol 78. https://doi.org/10.1007/s11042-018-6537-9

  80. Shan C, Gong S, McOwan PW (2009) Facial expression recognition based on local binary patterns: a comprehensive study. Image Vis Comput 27:803–816. https://doi.org/10.1016/j.imavis.2008.08.005

    Article  Google Scholar 

  81. Shen L, Bai L (2004) Adaboost gabor feature selection for classification

  82. Shreem S, Sheikh Abdullah S, Nazri MZA, Alzaqebah M (2012) Hybridizing relief, mrmr filters and ga wrapper approaches for gene selection. J Theor Appl Inf Technol 46:1034–1039

    Google Scholar 

  83. Singh S, Ahuja U, Kumar M, Kumar K, Sachdeva M (2021) Face mask detection using yolov3 and faster r-cnn models: Covid-19 environment. Multimed Tools Appl 80:19753–19768. https://doi.org/10.1007/s11042-021-10711-8

    Article  Google Scholar 

  84. Song F, Guo Z, Mei D (2010) Feature selection using principal component analysis. Conference: System Science, Engineering Design and Manufacturing Informatization (ICSEM) 1:27–30. https://doi.org/10.1109/ICSEM.2010.14

    Article  Google Scholar 

  85. Sormaz M, Young AW, Andrews TJ (2016) Contributions of feature shapes and surface cues to the recognition of facial expressions. Vis Res 127:1–10 . https://doi.org/10.1016/j.visres.2016.07.002

    Article  Google Scholar 

  86. Tottenham N, Tanaka J, Leon A, Mccarry T, Nurse M, Hare T, Marcus D, Westerlund A, Casey B, Nelson C (2009) The nimstim set of facial expressions: Judgments from untrained research participants. Psychiatry Res 168:242–9. https://doi.org/10.1016/j.psychres.2008.05.006

    Article  Google Scholar 

  87. Valstar M, Pantic M. (2006) Fully automatic facial action unit detection and temporal analysis’, paper presented. IEEE Conf Comput Vis Pattern Recog Work. https://doi.org/10.1109/CVPRW.2006.85

  88. Valstar M, Pantic M (2010) Induced disgust, happiness and surprise: an addition to the mmi facial expression database, Proc Int’l Conf Language Resources and Evaluation, workshop emotion, pp 65–70

  89. Valstar M, Pantic M, Ambadar Z, Cohn J (2006) Spontaneous vs. posed facial behavior: automatic analysis of brow actions. Applied Physics Letters - APPL PHYS LETT, pp 162–170. https://doi.org/10.1145/1180995.1181031

  90. Wang S, Wu C, He M, Wang J, Ji Q (2015) Posed and spontaneous expression recognition through modeling their spatial patterns. Mach Vis Appl 26(2–3):219–231. https://doi.org/10.1007/s00138-015-0657-2

    Article  Google Scholar 

  91. Wu T, Butko N, Ruvolo P, Whitehill J, Movellan J. (2011) Action unit recognition transfer across datasets, pp 889–896

  92. Xin Beh K, Meng Goh K (2019) Micro-expression spotting using facial landmarks. IEEE 15th International Colloquium on Signal Processing and Its Applications (CSPA) https://doi.org/10.1109/CSPA.2019.8696059

  93. Zhang K, Huang Y, Du Y, Wang L (2017) Facial expression recognition based on deep evolutional spatial-temporal networks. IEEE transactions on image processing : a publication of the IEEE Signal Processing Society. https://doi.org/10.1109/TIP.2017.2689999

  94. Zhang L, Tjondronegoro D, Chandran V (2012) Discovering the best feature extraction and selection algorithms for spontaneous facial expression recognition. Proceedings - IEEE International Conference on Multimedia and Expo, pp 1027–1032. https://doi.org/10.1109/ICME.2012.97

  95. Zhu J, Zou H, Rosset S, Hastie T (2009) Multi-class adaboost, statistics and its interface. J Comput Syst Sci 2:349–360

    MATH  Google Scholar 

  96. Zraqou J, Alkhadour W, Al-Nu’aimi AA-T (2013) An efficient approach for recognizing and tracking spontaneous facial expressions. 013 Second International Conference on E-Learning and E-Technologies in Education (ICEEE), pp 304–307. https://doi.org/10.1109/ICeLeTE.2013.6644393

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Walid Barhoumi.

Ethics declarations

Conflict of Interests

The authors have no relevant financial or non-financial interests to disclose.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Ones Sidhom, Haythem Ghazouani and Walid Barhoumi contributed equally to this work.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Sidhom, O., Ghazouani, H. & Barhoumi, W. Subject-dependent selection of geometrical features for spontaneous emotion recognition. Multimed Tools Appl (2022). https://doi.org/10.1007/s11042-022-13380-3

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s11042-022-13380-3

Keywords

  • Facial expression recognition
  • Feature extraction
  • Geometrical features
  • Face-based feature selection
  • Subject-dependency