Abstract
Facial expression recognition has always been a challenging issue due to the inconsistencies in the complexity of samples and variability of between expression categories. Many facial expression recognition methods train a classification model and then use this model to identify all test samples, without considering the complexity of each test sample. They are inconsistent with human cognition laws such as the principle of simplicity, so that they are easily under-learned and then are difficult to identify test samples correctly. Hence, this paper proposed a new facial expression recognition method sensing the complexity of test samples, which can nicely solve the problem of the inconsistent distribution of samples complexity. It firstly divided the training data into the hard subset and the easy subset for classification according to the complexity of samples for expression recognition. Subsequently, these two subsets are applied to train two classifiers. Instead of using the same classifier to predict all test samples, our method assigned each test sample to the corresponding classifier based on the complexity of the test sample. The experimental results demonstrated the effectiveness of the proposed method and obtained a significant improvements of the recognition performance on benchmark datasets.
Similar content being viewed by others
Change history
27 July 2020
The original version of this article unfortunately contained a mistake. Graphs c, d and e are missing in Figure 4. The correct and complete graphs of Figure 4 is shown here.
References
Sun Y, Wen G (2017) Cognitive facial expression recognition with constrained dimensionality reduction. Neurocomputing 230(2016):397–408
Friesen WV, Ekman P (1983) EMFACS-7: Emotional Facial Action Coding System
Siddiqi MH (2018) Accurate and robust facial expression recognition system using real-time YouTube-based datasets[J]. Appl Intell 48(9):2912–2929
Lopes AT, de Aguiar E, De Souza AF, Oliveira-Santos T (2017) Facial expression recognition with convolutional neural networks: coping with few data and the training sample order. Pattern Recogn 61:610–628
Wen G, Wei J, Wang J, Zhou T, Chen L (2013) Cognitive gravitation model for classification on small noisy data. Neurocomputing 118:245–252
Baruchello G (2015) A classification of classic, gestalt psychology and the tropes of rthetoric. New idea Pscychol 26:10–24
Smith MR, Martinez T, Giraud-Carrier C (2014) An instance level analysis of data complexity. Mach Learn 95:7225–7256
Brun AL, Britto AS Jr, Oliveira LS, Enembreck F, Sabourin R (2018) A framework for dynamic classifier selection oriented by the classification problem difficulty[J]. Pattern Recogn 76:175–190
Wang Z, Ruan Q, An G (2016) Facial expression recognition using sparse local fisher discriminant analysis. Neurocomputing 174:756–766
Savran A, Cao H, Nenkova A, Verma R (2015) Temporal Bayesian Fusion for Affect Sensing: Combining Video, Audio, and Lexical Modalities. IEEE Trans. Cybern
Zhang W, Shan S, Gao W, Chen X, Zhang H (2005) Local Gabor Binary Pattern Histogram Sequence (LGBPHS): A novel non-statistical model for face representation and recognition,” in Proceedings of the IEEE International Conference on Computer Vision
Dahmane M, Meunier J (2011) Emotion recognition using dynamic grid-based HoG features. in 2011 IEEE International Conference on Automatic Face and Gesture Recognition and Workshops, FG 2011
Berretti S, Ben Amor B, Daoudi M, Del Bimbo A (2011) 3D facial expression recognition using SIFT descriptors of automatically detected keypoints. Vis Comput
Jaffar MA (2017) Facial expression recognition using hybrid texture features based ensemble classifier. Int J Adv Comput Sci Appl 8(6):449–453
Ghimire D, Jeong S, Lee J, Park SH (2017) Facial expression recognition based on local region specific features and support vector machines. Multimed Tools Appl 76(6):7803–7821
Lajevardi SM, Hussain ZM (2012) Automatic facial expression recognition: feature extraction and selection. Signal, Image Video Proc 6(1):159–169
Shan C, Gritti T (2008) Learning discriminative LBP-histogram bins for facial expression recognition. Proc Br Mach Vis Conf:27.1–27.10
Khan S, Chen L, Yan H (2017) Co-clustering to reveal salient facial features for expression recognition. IEEE Trans Affect Comput 3045(c):1–14
Pantic M, Patras I (2006) Dynamics of facial expression: recognition of facial actions and their temporal segments from face profile image sequences. IEEE Trans Syst Man, Cybern Part B Cybern 36(2):433–449
Tong Y, Liao W, Ji Q (2007) Facial action unit recognition by exploiting their dynamic and semantic relationships. IEEE Trans Pattern Anal Mach Intell 29(10):1683–1699
Liu M, Li S, Shan S, Chen X (2015) AU-inspired deep networks for facial expression feature learning. Neurocomputing 159(1):126–136
Ranzato M, Susskind J, Mnih V, Hinton G (2011) On deep generative models with applications to recognition. Cvpr 2011:2857–2864
Deng J, Dong W, Socher R, Li L-J, Li K, Li F-F (2009) ImageNet: a large-scale hierarchical image database. Cvpr:248–255
Huang G, Liu Z, Maaten LVD, Weinberger KQ (2017) Densely connected convolutional networks. 2017 IEEE Conf Comput Vis Pattern Recognit:2261–2269
Li J, Lam EY (2015) Facial expression recognition using deep neural networks. Imaging Syst Tech (IST), 2015 IEEE Int Conf:1–6
Yu Z, Zhang C (2015) Image based static facial expression recognition with multiple deep network learning. Proc 2015 ACM Int Conf Multimodal Interact - ICMI ‘15:435–442
Tang Y (2013) Deep learning using linear support vector machines. Comput Therm Sci
Xu M, Cheng W, Zhao Q, Ma L, Xu F (2015) Facial expression recognition based on transfer learning from deep convolutional networks. 2015 11th Int Conf Nat Comput:702–708
Ng H-W, Nguyen VD, Vonikakis V, Winkler S (2015) Deep learning for emotion recognition on small datasets using transfer learning. Proc 2015 ACM Int Conf Multimodal Interact - ICMI ‘15:443–449
Li D, Wen G (2017) MRMR-based ensemble pruning for facial expression recognition. Multimed Tools Appl
Li D, Wen G, Hou Z, Huan E, Hu Y, Li H (2018) RTCRelief-F: an effective clustering and ordering-based ensemble pruning algorithm for facial expression recognition. Knowl Inf Syst:1–32
Ding H, Zhou SK, Chellappa R (2017) “FaceNet2ExpNet: Regularizing a Deep Face Recognition Net for Expression Recognition,” Proc. - 12th IEEE Int. Conf. Autom. Face Gesture Recognition, FG 2017 - 1st Int. Work. Adapt. Shot Learn. Gesture Underst. Prod. ASL4GUP 2017, Biometrics Wild, Bwild 2017, Heteroge, pp. 118–126
Al-Shabi M, Cheah WP, Connie T (2016) Facial Expression Recognition Using a Hybrid CNN– SIFT Aggregator. Int Work Multi-disciplinary Trends Artif Intell
Zhang T, Zheng W, Cui Z, Zong Y, Yan J (2016) A deep neural network-driven feature learning method for multi-view facial expression recognition [J]. IEEE Trans Multimed 18(12):2528–2536
Lopes AT, Aguiar ED, Souza AF, Oliveira-Santos T (2017) Facial expression recognition with convolutional neural networks: coping with few data and the training sample order [J]. Pattern Recogn 61:610–628
Chen J, Xu R, Liu L (2018) Deep peak-neutral difference feature for facial expression recognition[J]. Multimed Tools Appl. https://doi.org/10.1007/s11042-018-5909-5
Fang T, Zhao X, Ocegueda O, Shah SK, Kakadiaris IA (2011) 3D facial expression recognition: a perspective on promises and challenges[C]. IEEE Int Conf Autom Face Gesture Recog 28:603–610
Zhen Q, Huang D, Wang Y, Chen L (2016) Muscular movement model-based automatic 3D/4D facial expression recognition[J]. IEEE Trans Multimed 18(7):1438–1450
Dapogny A, Bailly K, Dubuisson S (2017) Dynamic pose-robust facial expression recognition by multi-view pairwise conditional random forests [J]. IEEE Trans on Affect Comput 99:1–14
Drira H, Ben Amor B, Daoudi M, Srivastava A, Berretti S (2012) 3D dynamic expression recognition based on a novel deformation vector field and random Forest[C]. IEEE Int Conf Patt Recog:1104–1107
Ben Amor B, Drira H, Berretti S, Daoudi M, Srivastava A (2017) 4D facial expression recognition by learning geometric deformations[J]. IEEE Trans Cybernet 44(12):2443–2457
Yao Y, Huang D, Yang X, Wang Y, Chen L (2018) Texture and Geometry Scattering Representation based Facial Expression Recognition in 2D+3D Videos [J], ACM Transactions on Multimedia Computing and Applications
Joan B, Stephane M (2013) Invariant scattering Nonvolution networks[J]. IEEE Trans Pattern Anal Mach Intell 35(8):1872–1886
Yang X, Huang D, Wang Y, Chen L (2015) Automatic 3D Facial Expression Recognition using Geometric Scattering Representation[C]. IEEE International Conference on Automatic Face and Gesture Recognition
Liu Y, Zeng J, Shan S, Zheng Z (2018) Multi-channel pose-aware convolution neural networks for multi-view facial expression recognition[C], 13th IEEE International Conference on Automatic Face & Gesture Recognition
Li W, Huang D, Li H, Wang Y (2018) Automatic 4D Facial Expression Recognition using Dynamic Geometrical Image Network[C], 13th IEEE International Conference on Automatic Face & Gesture Recognition
Mendialdua I, Martínez-Otzeta JM, Rodriguez-Rodriguez I, Ruiz-Vazquez T, Sierra B (2015) Dynamic selection of the best base classifier in one versus one[J]. Knowl-Based Syst 85:298–310
Didaci L, Giacinto G, Roli F, Marcialis GL (2005) A study on the performances of dynamic classifier selection based on local accuracy estimation[J]. Pattern Recogn 38(11):2188–2191
Xiao J, Xie L, He C, Jiang X (2012) Dynamic classifier ensemble model for customer classification with imbalanced class distribution[J]. Expert Syst Appl 39:3668–3675
Cavalin PR, Sabourin R, Suen CY (2012) Logid: an adaptive framework combining local and global incremental learning for dynamic selection of ensembles of HMMs[J]. Pattern Recogn 45(9):3544–3556
Szepannek G, Bischl B, Weihs C (2009) On the combination of locally optimal pairwise classifiers [J]. Eng Appl Artif Intell 22:79–85
de Souza BF, de Carvalho A, Calvo R, Ishii RP (2006) Multiclass svm model selection using particle swarm optimization[C]. Sixth Int Conf Hybrid Intel Syst, IEEE:31
Rafael MO (2015) Cruz, Robert Sabourin, George D.C. Cavalcanti, Tsang Ing Ren, META-DES: a dynamic ensemble selection framework using META-learning [J]. Pattern Recogn 48:1925–1935
Xu C, Du PF, Feng ZY, Meng ZP, Cao TY, Dong CC (2013) Multi-modal emotion recognition fusing video and audio [J]. Appl Math Inform Sci 7(2):455–462
Wang Y, Yang X, Zou J (2013) Research of emotion recognition based on speech and facial expression[J]. Inst Adv Eng Sci 11(1):83–90
Wang SF, He S, Wu Y, He MH, Ji Q (2014) Fusion of visible and thermal images for facial expression recognition [J]. Front Comput Sci 8(2):232–242
Majumder A, Behera L, Subramanian VK (2018) Automatic facial expression recognition system using deep network-based data fusion [J]. IEEE Trans Cybernet 48(1):103–114
Sun YC, Yu J (2017) Facial expression recognition by fusing Gabor and local binary pattern features [J]. Multimed Model 10133:209–220
Wang WC, Chang FL, Liu YL, Wu XJ (2017) Expression recognition method based on evidence theory and local texture [J]. Multimed Tools Appl 76(5):7365–7379
Sun B, Li LD, Zhou GY et al (2016) Facial expression recognition in the wild based on multimodal texture features [J]. J Electron Imaging 25(6)
Wen GH, Hou Z, Li HH, Li DY, Jiang LJ, Xun EY (2017) Ensemble of deep neural networks with probability-based fusion for facial expression recognition [J]. Cogn Comput 9(5):597–610
Wen GH, Li HH, Li DY (2015) An ensemble convolutional echo state networks for facial expression recognition [C]. 2015 International Conference on Affective Computing and Intelligent Interaction (ACII), Xian, China 873–878
Li D, Wen G, Hou Z, Huan E, Hu Y, Li H (2018) RTCRelief-F: An effective clustering and ordering-based ensemble pruning algorithm for facial expression recognition[J]. Knowl Inf Syst:1–32
Sun M, Liu K, Hong Q (2017) An ECOC approach for microarray data classification based on minimizing feature related complexities. 10th Int Symp Comput Intell Des 3:300–303
Pujol O, Radeva P, Vitrià J (2006) Discriminant ECOC: a heuristic method for application dependent design of error correcting output codes. IEEE Trans Pattern Anal Mach Intell 28(6):1007–1012
Mansilla EB, Ho TK (2004) On classifier domains of competence. Proc - Int Conf Pattern Recognit 1:136–139
de Souto MCP, Lorena AC, Spolaor N, Costa IG (2010) Complexity measures of supervised classifications tasks: a case study for cancer gene expression data. Int Jt Conf Neural Networks:1–7
Gui L, Baltrusaitis T, Morency L-P (2017) Curriculum learning for facial expression recognition. 12th IEEE Int Conf Autom Face Gesture Recognit:505–511
Wu S, Zhong S, Liu Y (2017) Deep residual learning for image steganalysis. Multimed Tools Appl:1–17
Goodfellow IJ et al (2015) Challenges in representation learning: a report on three machine learning contests. Neural Netw 64:59–63
Lyons MJ (1999) Automatic classification of single facial images. IEEE Trans Pattern Anal Mach Intell 21(12):1357–1362
Lucey P, Cohn JF, Kanade T, Saragih J, Ambadar Z, Matthews I (2010) The extended cohn-kande dataset (CK+): a complete facial expression dataset for action unit and emotions pecified expression. Cvprw:94–101
Mollahosseini A, Chan D, Mahoor MH (2016) Going deeper in facial expression recognition using deep neural networks. IEEE Winter Conf Appl Comput Vis 1:–10
Wu C, Wang S (2015) Multi-instance hidden Markov model for facial expression recognition. Int Conf Autom Face Gesture Recog
Happy SL, Routray A (2015) Automatic facial expression recognition using features of salient facial patches. IEEE Trans Affect Comput
Yan H (2018) Collaborative discriminative multi-metric learning for facial expression recognition in video. Pattern Recogn 75:1339–1351
Barman A, Dutta P (2017) Facial expression recognition using distance and shape signature features. Pattern Recogn Lett 0:1–8
Sariyanidi E, Gunes H, Cavallaro A (2017) “Learning Bases of Activity for Facial Expression Recognition,” IEEE Trans. Image Process
Chen X, Yang X, Wang M, Zou J (2017) Convolution neural network for automatic facial expression recognition. Proc IEEE Int Conf Appl Syst Innov Appl Syst Innov Mod Technol ICASI 2017 814–817
Liu M, Shan S, Wang R, Chen X (2014) Learning expressionlets on spatio-temporal manifold for dynamic facial expression recognition in Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition
Deaney W, Venter I, Ghaziasgar M, Dodds R (2017) A comparison of facial feature representation methods for automatic facial expression recognition. Proc South African Inst Comput Sci Inf Technol 10:1–10
Guo Y, Tao D, Yu J, Hao X, Li Y, Tao D (2016) Deep Neural Networks with Relativity Learning for facial expression recognition,” in 2016 IEEE International Conference on Multimedia and Expo Workshop, ICMEW
Kim B-K, Roh J, Dong S-Y, Lee S-Y (2016) Hierarchical committee of deep convolutional neural networks for robust facial expression recognition. J Multimodal User Interfaces
Wei W, Yang X-L, Zhou B et al (2012) Combined energy minimization for image reconstruction from few views. Math Probl Eng
Wei W, Yang X-L, Shen P-Y et al (2012) Holes detection in anisotropic Sensornets: topological methods. Int J Distribut Sensor Netw
Wei W, Qi Y (2011) Information potential fields navigation wireless adoc sensor networks. Sensors 11(5):4794–4807
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher’s note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Chang, T., Li, H., Wen, G. et al. Facial expression recognition sensing the complexity of testing samples. Appl Intell 49, 4319–4334 (2019). https://doi.org/10.1007/s10489-019-01491-8
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10489-019-01491-8