Abstract
Facial expressions are among the most powerful ways to reveal the emotional state. Therefore, Facial Expression Recognition (FER) has been widely introduced to wide fields of applications, such as security, psychotherapy, neuromarketing, and advertisement. Feature extraction and selection are two essential key issues for the design of efficient FER systems. However, most of the previous studies focused on implementing static feature selection methods. Although these methods have shown promising results, they still present weaknesses, especially when dealing with spontaneous expressions. This is mainly due to the specificity of each face, which makes the facial emotion display differs from one subject to another. To address this problem, we propose a face-based dynamic feature selection of two geometric features sub-classes, namely linear and eccentricity features. This combination provides a better understanding of the facial transformation during the emotion display. Moreover, the suggested selection method takes into consideration the subject’s general facial structure, muscle movements, and head position. The performed experiments, using the CK+ and the DISFA datasets, have showed that the proposed method outperforms the state-of-the-art techniques and maintains superior performance with cross-dataset validation. In fact, the accuracy of facial expression recognition by the proposed method reaches 97.72% and 91,26% on the CK+ and the DISFA datasets, respectively.
Similar content being viewed by others
References
Amminger G, Schaefer M, Papageorgiou K, Klier C, Schlogelhofer M, Mossaheb N, Werneck-Rohrer S, Nelson B, Mcgorry P (2012) Emotion recognition in individuals at clinical high-risk for schizophrenia. Schizophr Bull 38(5):1030–1039
Arora M, Kumar M (2021) Autofer: Pca and pso based automatic facial emotion recognition. Multimed Tools Appl 80:3039–3049. https://doi.org/10.1007/s11042-020-09726-4
Bandrabur A, Florea L, Florea C, Mancas M (2015) Emotion identification by facial landmarks dynamics analysis. International Conference on Intelligent Computer Communication and Processing (ICCP). https://doi.org/10.1109/ICCP.2015.7312688
Bansal M, Kumar M, Kumar M (2020) Xgboost: 2d-object recognition using shape descriptors and extreme gradient boosting classifier. In: Proceedings of the international conference on computational methods and data engineering, pp 207–222
Bansal M, Kumar M, Kumar M (2021) 2d object recognition: a comparative analysis of sift, surf and orb feature descriptors. Multimed Tools Appl 80:18839–18857. https://doi.org/10.1007/s11042-021-10646-0
Bejaoui H, Ghazouani H, Barhoumi W (2017) Fully automated facial expression recognition using 3d morphable model and mesh-local binary pattern, pp 39–50 https://doi.org/10.1007/978-3-319-70353-4_4
Bejaoui H, Ghazouani H, Barhoumi W (2019) Sparse coding-based representation of lbp difference for 3d/4d facial expression recognition. Multimed Tools Appl 78:22773–22796. https://doi.org/10.1007/s11042-019-7632-2
Ben-Hur A, Weston J (2010) A user’s guide to support vector machines. Methods in molecular biology (Clifton, N.J.) 609:223–39. https://doi.org/10.1007/978-1-60327-241-4_13
Brown G, Pocock A, Zhao M-J, Lujan M (2012) Conditional likelihood maximisation: a unifying framework for information theoretic feature selection. J Mach Learn Res 13(1):27–66
Butalia AH, Ingle M, Kulkarni SJ (2012) Facial expression recognition for security. Int J Mod Eng Res Technol 2(4):1449–1453
Candra H, Yuwono M, Chai R, Nguyen HT, Su S (2016) Classification of facial-emotion expression in the application of psychotherapy using viola-jones and edge-histogram of oriented gradient. 2016 38th Annu Int Conf of the IEEE Eng Med Biol Soc 38(5):423–426. https://doi.org/10.1109/EMBC.2016.7590730
Cao N, Ton-That A, Choi H-I (2016) An effective facial expression recognition approach for intelligent game systems. Int J Comput Vis Robot 6(3):223–234. https://doi.org/10.1504/IJCVR.2016.077353
Chen J, Chen D, Gong Y, Yu M, Zhang K, Wang L (2012) Facial expression recognition using geometric and appearance features, pp 29–33. https://doi.org/10.1145/2382336.2382345
Chen M, Cheng J, Zhang Z, Li Y, Zhang Y (2021) Facial expression recognition method combined with attention mechanism. Mobile Information Systems pp 2021. https://doi.org/10.1155/2021/5608340
Chhabra P, Garg NK, Kumar M (2020) Content-based image retrieval system using orb and sift features. Neural Comput Applic 32:2725–2733. https://doi.org/10.1007/s00521-018-3677-9
Dailey M, Cottrell G, Padgett C, Adolphs R (2002) Empath: a neural network that categorizes facial expressions. J Cogn Neurosci, pp 1158–73. https://doi.org/10.1162/089892902760807177
Dalal N, Triggs B (2005) Histograms of oriented gradients for human detection. Int Conf Comp Vision Pattern Recognit (CVPR ’05), Jun 2005, San Diego, United States, pp 886–893
Darwin C (1872) The expression of emotions, vol 19. (1–12), pp 399
Datta S, Sen D, Balasubramanian R (2017) Integrating geometric and textural features for facial emotion classification using svm frameworks, pp 619–628. https://doi.org/10.1007/978-981-10-2104-6_55
Desrosiers P, Daoudi M, Devanne M (2016) Novel generative model for facial expressions based on statistical shape analysis of landmarks trajectories. In: 23rd International Conference on Pattern Recognition (ICPR). https://doi.org/10.1109/ICPR.2016.7899760
Dibeklioğlu H, Salah AA, Gevers T (2015) Recognition of genuine smiles. IEEE Trans Multimed 17(3):279–294
Dibekliouglu H, Salah AA, Gevers T (2012) Are you really smiling at me? spontaneous versus posed enjoyment smiles pp 525–538
Ekman P (2003) Darwin, deception, and facial expression. Ann N Y Acad Sci 1000(1):205–221. https://doi.org/10.1196/annals.1280.010
Ekman P (2009) Telling lies: Clues to deceit in the marketplace, politics and marriage (revised edition)
Ekman P, Friesen WV (1978) Facial action coding system: a technique for the measurement of facial movement
Ekman P, Friesen W (1982) Felt, false, and miserable smiles. J Nonverbal Behav 6(4):238–252. https://doi.org/10.1007/BF00987191
Ekundayo O, Viriri S (2021) Facial expression recognition: a review of trends and techniques. IEEE Access, pp 1–1. https://doi.org/10.1109/ACCESS.2021.3113464
Fernandes J, Matos L, Aragao M (2016) Geometrical approaches for facial expression recognition using support vector machines. 016 29th SIBGRAPI Conference on Graphics, Patterns and Images, pp 347–354. https://doi.org/10.1109/SIBGRAPI.2016.055
Ferreira A, Figueiredo M (2012) Boosting algorithms: a review of methods, theory, and applications. Ensemble Machine Learning: Methods and Applications 3:35–85. https://doi.org/10.1007/978-1-4419-9326-7_2
Fölster M, Hess U, Werheid K (2014) Facial age affects emotional expression decoding. Frontiers in Psychology vol 5. https://doi.org/10.3389/fpsyg.2014.00030
Friedman JH (1996) Another approach to polychotomous classification Department of Statistics, Stanford University, pp 1452–1459
Gharsalli S, Laurent H, Emile B, Desquesnes X (2015) Various fusion schemes to recognize simulated and spontaneous emotions. VISAPP 2015 - 10th International Conference on Computer Vision Theory and Applications; VISIGRAPP, Proceedings vol 2. https://doi.org/10.5220/0005312804240431
Ghazouani H (2021) A genetic programming-based feature selection and fusion for facial expression recognition. Appl Soft Comput 103:107173
Ghimire D, Jeong S, Lee J, Park S (2017) Facial expression recognition based on local region specific features and support vector machines. Multimed Tools Appl, vol 76. https://doi.org/10.1007/s11042-016-3418-y
Gidudu A, Hulley G, Marwala T (2007) Image classification using svms: One-against-one vs one-against-all. arXiv:0711.2914
Goren D, Wilson HR (2006) Quantifying facial expression recognition across viewing conditions. Vis Res 46:1253–1262. https://doi.org/10.1016/j.visres.2005.10.028
Goren D, Wilson HR (2006) Quantifying facial expression recognition across viewing conditions. Vis Res 46(8):1253–1262. https://doi.org/10.1016/j.visres.2005.10.028
Guo H, Zhang X-H, Liang J, Yan W-J (2018) The dynamic features of lip corners in genuine and posed smiles. Front Psychol 9:202. https://doi.org/10.3389/fpsyg.2018.00202
Gupta O, Raviv D, Raskar R (2017) Illumination invariants in deep video expression recognition. Pattern Recognition, pp 76. https://doi.org/10.1016/j.patcog.2017.10.017
Guyon I, Elisseeff A (2003) An introduction to variable and feature selection. J Mach Learn Res 3:1157–1182
Hall M (2000) Correlation-based feature selection for discrete and numeric class machine learning. In: Proceedings of the Seventeenth International Conference on Machine Learning (ICML 2000), Stanford University, Stanford, CA,USA, pp 359–366
Hall M (2000) Correlation-based feature selection for machine learning. Department of Computer Science
Hamelin N, Moujahid OE, Thaichon P (2017) Emotion and advertising effectiveness: a novel facial expression analysis approach. J Retail Consum Serv 36:103–111. https://doi.org/10.1016/j.jretconser.2017.01.001
Hassaballah M, Saddam Bekhet AAMR, Zhang G (2019) Facial features detection and localization. Recent Adv Comput Vision Stud Comput Intell 804:33–59. https://doi.org/10.1007/978-3-030-03000-1_2
He M, Wang S, Liu Z, Chen X (2013) Analyses of the differences between posed and spontaneous facial expressions. Humaine Association Conference on Affective Computing and Intelligent Interaction, pp 79–84. https://doi.org/10.1109/ACII.2013.20
Horn B, Schunck B (1981) Determining optical flow. Artif Intell 17:185–203. https://doi.org/10.1016/0004-3702(81)90024-2
Hsu C-W , Lin C-J (2002) A comparison of methods for multiclass support vector machines. IEEE Trans Neural Netw 13 (2):415–425. https://doi.org/10.1109/72.991427
Huynh X-P, Kim Y-G (2017) Discrimination between genuine versus fake emotion using long-short term memory with parametric bias and facial landmarks pp 3065–3072
Iqbal MT, Ryu B, Ramirez Rivera A, Makhmudkhujaev F, Chae O, Bae S-H (2020) Facial expression recognition with active local shape pattern and learned-size block representations. IEEE Trans Affect Comput. https://doi.org/10.1109/TAFFC.2020.2995432
Jia S, Wang S, Hu C, Webster PJ, Li X (2021) Detection of genuine and posed facial expressions of emotion: databases and methods. Front Psychol 11:3818. https://doi.org/10.3389/fpsyg.2020.580287
Kazemi V, Sullivan J (2014) One millisecond face alignment with an ensemble of regression trees. In: Comput Vis Pattern Recognit (June 2014, Columbus, Ohio) CVPR’14 IEEE, pp 1867–1874
Kumar M, Chhabra P, Garg NK (2018) An efficient content based image retrieval system using bayesnet and k-nn. Multimed Tools Appl 77:21557–21570. https://doi.org/10.1007/s11042-017-5587-8
Kumar A, Kaur A, Kumar M (2019) Face detection techniques: a review. Artif Intell Rev 52:927–948. https://doi.org/10.1007/s10462-018-9650-2
Kumar A, Kumar M, Kaur A (2021) Face detection in still images under occlusion and non-uniform illumination. Multimed Tools Appl 80:14565–14590. https://doi.org/10.1007/s11042-020-10457-9
Lajevardi S, Hussain Z (2012) Automatic facial expression recognition: Feature extraction and selection. A. Signal, Image and Video Processing 6:159–169. https://doi.org/10.1007/s11760-010-0177-5
Lee S, Baddar W, Ro Y (2016) Collaborative expression representation using peak expression and intra class variation face images for practical subject-independent emotion recognition in videos. Pattern Recognition, vol 54. https://doi.org/10.1016/j.patcog.2015.12.016
Li L, Yuan Y, Li M, Xu H, Li R, Lu S (2019) Subject independent facial expression recognition: Cross-connection and spatial pyramid pooling convolutional neural network. IVSP 2019,: Proceedings of the 2019, International Conference on Image, Video and Signal Processing, pp 85–92. https://doi.org/10.1145/3317640.3317662
Littlewort G, Bartlett M.S, Fasel I, Susskind J, Movellan J. (2006) Dynamics of facial expression extracted automatically from video. Image Vis Comput 24:615–625. https://doi.org/10.1016/j.imavis.2005.09.011
Littlewort G, Frank M, Lainscsek C, Fasel I, Movellan J (2006) Automatic recognition of facial actions in spontaneous expressions. Journal of Multimedia, vol 1(6) https://doi.org/10.4304/jmm.1.6.22-35
Littlewort G, Lainscsek C, Fasel I, Movellan J (2004) Machine learning methods for fully automatic recognition of facial expressions and facial actions. Conf Proc - IEEE Int Conf Syst Man Cybern 1:592–597. https://doi.org/10.1109/ICSMC.2004.1398364
Liu J, Bai M, Jiang N , Cheng R, Li X, Wang Y, Yu D (2021) Interclass interference suppression in multi-class problems. Applied Sciences vol 11. https://doi.org/10.3390/app11010450
Liu P, Han S, Meng Z, Tong Y (2014) Facial expression recognition via a boosted deep belief network. Proc IEEE Conf Comput Vis Pattern Recognit, pp 1805–1812. https://doi.org/10.1109/CVPR.2014.233
Liu M, Li S, Shan S, Chen X. (2015) Au-inspired deep networks for facial expression feature learning. Neurocomputing, vol 159. https://doi.org/10.1016/j.neucom.2015.02.011
Lucey P, Cohn J, Kanade T, Saragih J, Ambadar Z, Matthews I (2010) The extended cohn-kanade dataset (ck+): A complete dataset for action unit and emotion-specified expression. IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Workshops, CVPRW 2010:94–101. https://doi.org/10.1109/CVPRW.2010.5543262
Makhmudkhujaev F, Iqbal MT, Ryu B, Chae O (2019) Local directional-structural pattern for person-independent facial expression recognition. Turk J Elec Eng Comp Sci 27:516–531. https://doi.org/10.3906/elk-1804-58
Mavadati S, Mahoor M, Bartlett K, Trinh P, Cohn J (2013) Disfa: a spontaneous facial action intensity database. IEEE Trans Affect Comput 4(2):151–160. https://doi.org/10.1109/T-AFFC.2013.4
Miao Y-Q, Araujo R, Kamel MS (2012) Cross-domain facial expression recognition using supervised kernel mean matching, vol 2012
Namba S, Makihara S, Kabir R, Miyatani M, Nakao T (2016) Spontaneous facial expressions are different from posed facial expressions: Morphological properties and dynamic sequences. Current Psychology, pp 1–13. https://doi.org/10.1007/s12144-016-9448-9
Novakovic J, Minic M, Veljovic A (2011) Classification accuracy of neural networks with pca in emotion recognition. Theory Appl Math Comput Sci 1:11–16
Park S, Lee K, Lim J-A, Ko H, Kim T, Lee J-I, Kim H, Han S-J, Kim J-S, Park S et al (2020) Differences in facial expressions between spontaneous and posed smiles: automated method by action units and three-dimensional facial landmarks. Sensors 20(4):1199
Peng H, Long F, Ding C (2005) Feature selection based on mutual information: Criteria of max-dependency, max-relevance, and min-redundancy. IEEE Trans Pattern Anal Mach Intell 27:1226–38. https://doi.org/10.1109/TPAMI.2005159
Rabiu H, Saripan MI, Mashohor S, Marhaban MH (2012) 3d facial expression recognition using maximum relevance minimum redundancy geometrical features. EURASIP Journal on Advances in Signal Processing. https://doi.org/10.1186/1687-6180-2012-213
Sadeghi H, Raie A, Mohammadi MR (2013) Facial expression recognition using geometric normalization and appearance representation. Iranian Conference on Machine Vision and Image Processing, MVIP. https://doi.org/10.1109/IranianMVIP.2013.6779970
Sagonas C, Antonakos E, Tzimiropoulos G, Zafeiriou S, Pantic M (2016) 300 faces in-the-wild challenge: database and results. Image Vision Comput 47:3–18. https://doi.org/10.1016/j.imavis.2016.01.002
Sagonas C, Zafeiriou S (2013) Facial point annotations. https://ibug.doc.ic.ac.uk/resources/facial-point-annotations/. Accessed date 24th September 2020
Said C, Haxby J, Todorov A (2011) Brain systems for assessing the affective value of faces. Philosophical transactions of the Royal Society of London. Series B. Biological sciences vol 366. pp 1660–70. https://doi.org/10.1098/rstb.2010.0351
Samadiani N, Huang G, Cai B, Luo W, Chi C-H, Xiang Y, He J (2019) A review on automatic facial expression recognition systems assisted by multimodal sensor data. Sensors (Basel) vol 19(8). https://doi.org/10.3390/s19081863
Saxen F, Werner P, Al-Hamadi A (2017) Real vs. fake emotion challenge: Learning to rank authenticity from facial activity descriptors, pp 3073–3078
Sen D, Datta S, Balasubramanian R (2019) Facial emotion classification using concatenated geometric and textural features. Multimedia Tools and Applications vol 78. https://doi.org/10.1007/s11042-018-6537-9
Shan C, Gong S, McOwan PW (2009) Facial expression recognition based on local binary patterns: a comprehensive study. Image Vis Comput 27:803–816. https://doi.org/10.1016/j.imavis.2008.08.005
Shen L, Bai L (2004) Adaboost gabor feature selection for classification
Shreem S, Sheikh Abdullah S, Nazri MZA, Alzaqebah M (2012) Hybridizing relief, mrmr filters and ga wrapper approaches for gene selection. J Theor Appl Inf Technol 46:1034–1039
Singh S, Ahuja U, Kumar M, Kumar K, Sachdeva M (2021) Face mask detection using yolov3 and faster r-cnn models: Covid-19 environment. Multimed Tools Appl 80:19753–19768. https://doi.org/10.1007/s11042-021-10711-8
Song F, Guo Z, Mei D (2010) Feature selection using principal component analysis. Conference: System Science, Engineering Design and Manufacturing Informatization (ICSEM) 1:27–30. https://doi.org/10.1109/ICSEM.2010.14
Sormaz M, Young AW, Andrews TJ (2016) Contributions of feature shapes and surface cues to the recognition of facial expressions. Vis Res 127:1–10 . https://doi.org/10.1016/j.visres.2016.07.002
Tottenham N, Tanaka J, Leon A, Mccarry T, Nurse M, Hare T, Marcus D, Westerlund A, Casey B, Nelson C (2009) The nimstim set of facial expressions: Judgments from untrained research participants. Psychiatry Res 168:242–9. https://doi.org/10.1016/j.psychres.2008.05.006
Valstar M, Pantic M. (2006) Fully automatic facial action unit detection and temporal analysis’, paper presented. IEEE Conf Comput Vis Pattern Recog Work. https://doi.org/10.1109/CVPRW.2006.85
Valstar M, Pantic M (2010) Induced disgust, happiness and surprise: an addition to the mmi facial expression database, Proc Int’l Conf Language Resources and Evaluation, workshop emotion, pp 65–70
Valstar M, Pantic M, Ambadar Z, Cohn J (2006) Spontaneous vs. posed facial behavior: automatic analysis of brow actions. Applied Physics Letters - APPL PHYS LETT, pp 162–170. https://doi.org/10.1145/1180995.1181031
Wang S, Wu C, He M, Wang J, Ji Q (2015) Posed and spontaneous expression recognition through modeling their spatial patterns. Mach Vis Appl 26(2–3):219–231. https://doi.org/10.1007/s00138-015-0657-2
Wu T, Butko N, Ruvolo P, Whitehill J, Movellan J. (2011) Action unit recognition transfer across datasets, pp 889–896
Xin Beh K, Meng Goh K (2019) Micro-expression spotting using facial landmarks. IEEE 15th International Colloquium on Signal Processing and Its Applications (CSPA) https://doi.org/10.1109/CSPA.2019.8696059
Zhang K, Huang Y, Du Y, Wang L (2017) Facial expression recognition based on deep evolutional spatial-temporal networks. IEEE transactions on image processing : a publication of the IEEE Signal Processing Society. https://doi.org/10.1109/TIP.2017.2689999
Zhang L, Tjondronegoro D, Chandran V (2012) Discovering the best feature extraction and selection algorithms for spontaneous facial expression recognition. Proceedings - IEEE International Conference on Multimedia and Expo, pp 1027–1032. https://doi.org/10.1109/ICME.2012.97
Zhu J, Zou H, Rosset S, Hastie T (2009) Multi-class adaboost, statistics and its interface. J Comput Syst Sci 2:349–360
Zraqou J, Alkhadour W, Al-Nu’aimi AA-T (2013) An efficient approach for recognizing and tracking spontaneous facial expressions. 013 Second International Conference on E-Learning and E-Technologies in Education (ICEEE), pp 304–307. https://doi.org/10.1109/ICeLeTE.2013.6644393
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of Interests
The authors have no relevant financial or non-financial interests to disclose.
Additional information
Publisher’s note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Ones Sidhom, Haythem Ghazouani and Walid Barhoumi contributed equally to this work.
Rights and permissions
About this article
Cite this article
Sidhom, O., Ghazouani, H. & Barhoumi, W. Subject-dependent selection of geometrical features for spontaneous emotion recognition. Multimed Tools Appl 82, 2635–2661 (2023). https://doi.org/10.1007/s11042-022-13380-3
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11042-022-13380-3