Advertisement

Applied Intelligence

, Volume 49, Issue 12, pp 4319–4334 | Cite as

Facial expression recognition sensing the complexity of testing samples

  • Tianyuan Chang
  • Huihui LiEmail author
  • Guihua Wen
  • Yang Hu
  • Jiajiong Ma
Article
First Online:
  • 366 Downloads

Abstract

Facial expression recognition has always been a challenging issue due to the inconsistencies in the complexity of samples and variability of between expression categories. Many facial expression recognition methods train a classification model and then use this model to identify all test samples, without considering the complexity of each test sample. They are inconsistent with human cognition laws such as the principle of simplicity, so that they are easily under-learned and then are difficult to identify test samples correctly. Hence, this paper proposed a new facial expression recognition method sensing the complexity of test samples, which can nicely solve the problem of the inconsistent distribution of samples complexity. It firstly divided the training data into the hard subset and the easy subset for classification according to the complexity of samples for expression recognition. Subsequently, these two subsets are applied to train two classifiers. Instead of using the same classifier to predict all test samples, our method assigned each test sample to the corresponding classifier based on the complexity of the test sample. The experimental results demonstrated the effectiveness of the proposed method and obtained a significant improvements of the recognition performance on benchmark datasets.

Keywords

Facial expression recognition Sample complexity Convolutional neural network Gestalt principle 
This is a preview of subscription content, log in to check access.

Notes

References

  1. 1.
    Sun Y, Wen G (2017) Cognitive facial expression recognition with constrained dimensionality reduction. Neurocomputing 230(2016):397–408Google Scholar
  2. 2.
    Friesen WV, Ekman P (1983) EMFACS-7: Emotional Facial Action Coding SystemGoogle Scholar
  3. 3.
    Siddiqi MH (2018) Accurate and robust facial expression recognition system using real-time YouTube-based datasets[J]. Appl Intell 48(9):2912–2929Google Scholar
  4. 4.
    Lopes AT, de Aguiar E, De Souza AF, Oliveira-Santos T (2017) Facial expression recognition with convolutional neural networks: coping with few data and the training sample order. Pattern Recogn 61:610–628Google Scholar
  5. 5.
    Wen G, Wei J, Wang J, Zhou T, Chen L (2013) Cognitive gravitation model for classification on small noisy data. Neurocomputing 118:245–252Google Scholar
  6. 6.
    Baruchello G (2015) A classification of classic, gestalt psychology and the tropes of rthetoric. New idea Pscychol 26:10–24Google Scholar
  7. 7.
    Smith MR, Martinez T, Giraud-Carrier C (2014) An instance level analysis of data complexity. Mach Learn 95:7225–7256MathSciNetGoogle Scholar
  8. 8.
    Brun AL, Britto AS Jr, Oliveira LS, Enembreck F, Sabourin R (2018) A framework for dynamic classifier selection oriented by the classification problem difficulty[J]. Pattern Recogn 76:175–190Google Scholar
  9. 9.
    Wang Z, Ruan Q, An G (2016) Facial expression recognition using sparse local fisher discriminant analysis. Neurocomputing 174:756–766Google Scholar
  10. 10.
    Savran A, Cao H, Nenkova A, Verma R (2015) Temporal Bayesian Fusion for Affect Sensing: Combining Video, Audio, and Lexical Modalities. IEEE Trans. CybernGoogle Scholar
  11. 11.
    Zhang W, Shan S, Gao W, Chen X, Zhang H (2005) Local Gabor Binary Pattern Histogram Sequence (LGBPHS): A novel non-statistical model for face representation and recognition,” in Proceedings of the IEEE International Conference on Computer VisionGoogle Scholar
  12. 12.
    Dahmane M, Meunier J (2011) Emotion recognition using dynamic grid-based HoG features. in 2011 IEEE International Conference on Automatic Face and Gesture Recognition and Workshops, FG 2011Google Scholar
  13. 13.
    Berretti S, Ben Amor B, Daoudi M, Del Bimbo A (2011) 3D facial expression recognition using SIFT descriptors of automatically detected keypoints. Vis ComputGoogle Scholar
  14. 14.
    Jaffar MA (2017) Facial expression recognition using hybrid texture features based ensemble classifier. Int J Adv Comput Sci Appl 8(6):449–453Google Scholar
  15. 15.
    Ghimire D, Jeong S, Lee J, Park SH (2017) Facial expression recognition based on local region specific features and support vector machines. Multimed Tools Appl 76(6):7803–7821Google Scholar
  16. 16.
    Lajevardi SM, Hussain ZM (2012) Automatic facial expression recognition: feature extraction and selection. Signal, Image Video Proc 6(1):159–169Google Scholar
  17. 17.
    Shan C, Gritti T (2008) Learning discriminative LBP-histogram bins for facial expression recognition. Proc Br Mach Vis Conf:27.1–27.10Google Scholar
  18. 18.
    Khan S, Chen L, Yan H (2017) Co-clustering to reveal salient facial features for expression recognition. IEEE Trans Affect Comput 3045(c):1–14Google Scholar
  19. 19.
    Pantic M, Patras I (2006) Dynamics of facial expression: recognition of facial actions and their temporal segments from face profile image sequences. IEEE Trans Syst Man, Cybern Part B Cybern 36(2):433–449Google Scholar
  20. 20.
    Tong Y, Liao W, Ji Q (2007) Facial action unit recognition by exploiting their dynamic and semantic relationships. IEEE Trans Pattern Anal Mach Intell 29(10):1683–1699Google Scholar
  21. 21.
    Liu M, Li S, Shan S, Chen X (2015) AU-inspired deep networks for facial expression feature learning. Neurocomputing 159(1):126–136Google Scholar
  22. 22.
    Ranzato M, Susskind J, Mnih V, Hinton G (2011) On deep generative models with applications to recognition. Cvpr 2011:2857–2864Google Scholar
  23. 23.
    Deng J, Dong W, Socher R, Li L-J, Li K, Li F-F (2009) ImageNet: a large-scale hierarchical image database. Cvpr:248–255Google Scholar
  24. 24.
    Huang G, Liu Z, Maaten LVD, Weinberger KQ (2017) Densely connected convolutional networks. 2017 IEEE Conf Comput Vis Pattern Recognit:2261–2269Google Scholar
  25. 25.
    Li J, Lam EY (2015) Facial expression recognition using deep neural networks. Imaging Syst Tech (IST), 2015 IEEE Int Conf:1–6Google Scholar
  26. 26.
    Yu Z, Zhang C (2015) Image based static facial expression recognition with multiple deep network learning. Proc 2015 ACM Int Conf Multimodal Interact - ICMI ‘15:435–442Google Scholar
  27. 27.
    Tang Y (2013) Deep learning using linear support vector machines. Comput Therm SciGoogle Scholar
  28. 28.
    Xu M, Cheng W, Zhao Q, Ma L, Xu F (2015) Facial expression recognition based on transfer learning from deep convolutional networks. 2015 11th Int Conf Nat Comput:702–708Google Scholar
  29. 29.
    Ng H-W, Nguyen VD, Vonikakis V, Winkler S (2015) Deep learning for emotion recognition on small datasets using transfer learning. Proc 2015 ACM Int Conf Multimodal Interact - ICMI ‘15:443–449Google Scholar
  30. 30.
    Li D, Wen G (2017) MRMR-based ensemble pruning for facial expression recognition. Multimed Tools ApplGoogle Scholar
  31. 31.
    Li D, Wen G, Hou Z, Huan E, Hu Y, Li H (2018) RTCRelief-F: an effective clustering and ordering-based ensemble pruning algorithm for facial expression recognition. Knowl Inf Syst:1–32Google Scholar
  32. 32.
    Ding H, Zhou SK, Chellappa R (2017) “FaceNet2ExpNet: Regularizing a Deep Face Recognition Net for Expression Recognition,” Proc. - 12th IEEE Int. Conf. Autom. Face Gesture Recognition, FG 2017 - 1st Int. Work. Adapt. Shot Learn. Gesture Underst. Prod. ASL4GUP 2017, Biometrics Wild, Bwild 2017, Heteroge, pp. 118–126Google Scholar
  33. 33.
    Al-Shabi M, Cheah WP, Connie T (2016) Facial Expression Recognition Using a Hybrid CNN– SIFT Aggregator. Int Work Multi-disciplinary Trends Artif IntellGoogle Scholar
  34. 34.
    Zhang T, Zheng W, Cui Z, Zong Y, Yan J (2016) A deep neural network-driven feature learning method for multi-view facial expression recognition [J]. IEEE Trans Multimed 18(12):2528–2536Google Scholar
  35. 35.
    Lopes AT, Aguiar ED, Souza AF, Oliveira-Santos T (2017) Facial expression recognition with convolutional neural networks: coping with few data and the training sample order [J]. Pattern Recogn 61:610–628Google Scholar
  36. 36.
    Chen J, Xu R, Liu L (2018) Deep peak-neutral difference feature for facial expression recognition[J]. Multimed Tools Appl.  https://doi.org/10.1007/s11042-018-5909-5 Google Scholar
  37. 37.
    Fang T, Zhao X, Ocegueda O, Shah SK, Kakadiaris IA (2011) 3D facial expression recognition: a perspective on promises and challenges[C]. IEEE Int Conf Autom Face Gesture Recog 28:603–610Google Scholar
  38. 38.
    Zhen Q, Huang D, Wang Y, Chen L (2016) Muscular movement model-based automatic 3D/4D facial expression recognition[J]. IEEE Trans Multimed 18(7):1438–1450Google Scholar
  39. 39.
    Dapogny A, Bailly K, Dubuisson S (2017) Dynamic pose-robust facial expression recognition by multi-view pairwise conditional random forests [J]. IEEE Trans on Affect Comput 99:1–14Google Scholar
  40. 40.
    Drira H, Ben Amor B, Daoudi M, Srivastava A, Berretti S (2012) 3D dynamic expression recognition based on a novel deformation vector field and random Forest[C]. IEEE Int Conf Patt Recog:1104–1107Google Scholar
  41. 41.
    Ben Amor B, Drira H, Berretti S, Daoudi M, Srivastava A (2017) 4D facial expression recognition by learning geometric deformations[J]. IEEE Trans Cybernet 44(12):2443–2457Google Scholar
  42. 42.
    Yao Y, Huang D, Yang X, Wang Y, Chen L (2018) Texture and Geometry Scattering Representation based Facial Expression Recognition in 2D+3D Videos [J], ACM Transactions on Multimedia Computing and ApplicationsGoogle Scholar
  43. 43.
    Joan B, Stephane M (2013) Invariant scattering Nonvolution networks[J]. IEEE Trans Pattern Anal Mach Intell 35(8):1872–1886Google Scholar
  44. 44.
    Yang X, Huang D, Wang Y, Chen L (2015) Automatic 3D Facial Expression Recognition using Geometric Scattering Representation[C]. IEEE International Conference on Automatic Face and Gesture RecognitionGoogle Scholar
  45. 45.
    Liu Y, Zeng J, Shan S, Zheng Z (2018) Multi-channel pose-aware convolution neural networks for multi-view facial expression recognition[C], 13th IEEE International Conference on Automatic Face & Gesture RecognitionGoogle Scholar
  46. 46.
    Li W, Huang D, Li H, Wang Y (2018) Automatic 4D Facial Expression Recognition using Dynamic Geometrical Image Network[C], 13th IEEE International Conference on Automatic Face & Gesture RecognitionGoogle Scholar
  47. 47.
    Mendialdua I, Martínez-Otzeta JM, Rodriguez-Rodriguez I, Ruiz-Vazquez T, Sierra B (2015) Dynamic selection of the best base classifier in one versus one[J]. Knowl-Based Syst 85:298–310Google Scholar
  48. 48.
    Didaci L, Giacinto G, Roli F, Marcialis GL (2005) A study on the performances of dynamic classifier selection based on local accuracy estimation[J]. Pattern Recogn 38(11):2188–2191zbMATHGoogle Scholar
  49. 49.
    Xiao J, Xie L, He C, Jiang X (2012) Dynamic classifier ensemble model for customer classification with imbalanced class distribution[J]. Expert Syst Appl 39:3668–3675Google Scholar
  50. 50.
    Cavalin PR, Sabourin R, Suen CY (2012) Logid: an adaptive framework combining local and global incremental learning for dynamic selection of ensembles of HMMs[J]. Pattern Recogn 45(9):3544–3556Google Scholar
  51. 51.
    Szepannek G, Bischl B, Weihs C (2009) On the combination of locally optimal pairwise classifiers [J]. Eng Appl Artif Intell 22:79–85Google Scholar
  52. 52.
    de Souza BF, de Carvalho A, Calvo R, Ishii RP (2006) Multiclass svm model selection using particle swarm optimization[C]. Sixth Int Conf Hybrid Intel Syst, IEEE:31Google Scholar
  53. 53.
    Rafael MO (2015) Cruz, Robert Sabourin, George D.C. Cavalcanti, Tsang Ing Ren, META-DES: a dynamic ensemble selection framework using META-learning [J]. Pattern Recogn 48:1925–1935Google Scholar
  54. 54.
    Xu C, Du PF, Feng ZY, Meng ZP, Cao TY, Dong CC (2013) Multi-modal emotion recognition fusing video and audio [J]. Appl Math Inform Sci 7(2):455–462Google Scholar
  55. 55.
    Wang Y, Yang X, Zou J (2013) Research of emotion recognition based on speech and facial expression[J]. Inst Adv Eng Sci 11(1):83–90Google Scholar
  56. 56.
    Wang SF, He S, Wu Y, He MH, Ji Q (2014) Fusion of visible and thermal images for facial expression recognition [J]. Front Comput Sci 8(2):232–242MathSciNetGoogle Scholar
  57. 57.
    Majumder A, Behera L, Subramanian VK (2018) Automatic facial expression recognition system using deep network-based data fusion [J]. IEEE Trans Cybernet 48(1):103–114Google Scholar
  58. 58.
    Sun YC, Yu J (2017) Facial expression recognition by fusing Gabor and local binary pattern features [J]. Multimed Model 10133:209–220Google Scholar
  59. 59.
    Wang WC, Chang FL, Liu YL, Wu XJ (2017) Expression recognition method based on evidence theory and local texture [J]. Multimed Tools Appl 76(5):7365–7379Google Scholar
  60. 60.
    Sun B, Li LD, Zhou GY et al (2016) Facial expression recognition in the wild based on multimodal texture features [J]. J Electron Imaging 25(6)Google Scholar
  61. 61.
    Wen GH, Hou Z, Li HH, Li DY, Jiang LJ, Xun EY (2017) Ensemble of deep neural networks with probability-based fusion for facial expression recognition [J]. Cogn Comput 9(5):597–610Google Scholar
  62. 62.
    Wen GH, Li HH, Li DY (2015) An ensemble convolutional echo state networks for facial expression recognition [C]. 2015 International Conference on Affective Computing and Intelligent Interaction (ACII), Xian, China 873–878Google Scholar
  63. 63.
    Li D, Wen G, Hou Z, Huan E, Hu Y, Li H (2018) RTCRelief-F: An effective clustering and ordering-based ensemble pruning algorithm for facial expression recognition[J]. Knowl Inf Syst:1–32Google Scholar
  64. 64.
    Sun M, Liu K, Hong Q (2017) An ECOC approach for microarray data classification based on minimizing feature related complexities. 10th Int Symp Comput Intell Des 3:300–303Google Scholar
  65. 65.
    Pujol O, Radeva P, Vitrià J (2006) Discriminant ECOC: a heuristic method for application dependent design of error correcting output codes. IEEE Trans Pattern Anal Mach Intell 28(6):1007–1012Google Scholar
  66. 66.
    Mansilla EB, Ho TK (2004) On classifier domains of competence. Proc - Int Conf Pattern Recognit 1:136–139Google Scholar
  67. 67.
    de Souto MCP, Lorena AC, Spolaor N, Costa IG (2010) Complexity measures of supervised classifications tasks: a case study for cancer gene expression data. Int Jt Conf Neural Networks:1–7Google Scholar
  68. 68.
    Gui L, Baltrusaitis T, Morency L-P (2017) Curriculum learning for facial expression recognition. 12th IEEE Int Conf Autom Face Gesture Recognit:505–511Google Scholar
  69. 69.
    Wu S, Zhong S, Liu Y (2017) Deep residual learning for image steganalysis. Multimed Tools Appl:1–17Google Scholar
  70. 70.
    Goodfellow IJ et al (2015) Challenges in representation learning: a report on three machine learning contests. Neural Netw 64:59–63Google Scholar
  71. 71.
    Lyons MJ (1999) Automatic classification of single facial images. IEEE Trans Pattern Anal Mach Intell 21(12):1357–1362Google Scholar
  72. 72.
    Lucey P, Cohn JF, Kanade T, Saragih J, Ambadar Z, Matthews I (2010) The extended cohn-kande dataset (CK+): a complete facial expression dataset for action unit and emotions pecified expression. Cvprw:94–101Google Scholar
  73. 73.
    Mollahosseini A, Chan D, Mahoor MH (2016) Going deeper in facial expression recognition using deep neural networks. IEEE Winter Conf Appl Comput Vis 1:–10Google Scholar
  74. 74.
    Wu C, Wang S (2015) Multi-instance hidden Markov model for facial expression recognition. Int Conf Autom Face Gesture RecogGoogle Scholar
  75. 75.
    Happy SL, Routray A (2015) Automatic facial expression recognition using features of salient facial patches. IEEE Trans Affect ComputGoogle Scholar
  76. 76.
    Yan H (2018) Collaborative discriminative multi-metric learning for facial expression recognition in video. Pattern Recogn 75:1339–1351Google Scholar
  77. 77.
    Barman A, Dutta P (2017) Facial expression recognition using distance and shape signature features. Pattern Recogn Lett 0:1–8Google Scholar
  78. 78.
    Sariyanidi E, Gunes H, Cavallaro A (2017) “Learning Bases of Activity for Facial Expression Recognition,” IEEE Trans. Image ProcessGoogle Scholar
  79. 79.
    Chen X, Yang X, Wang M, Zou J (2017) Convolution neural network for automatic facial expression recognition. Proc IEEE Int Conf Appl Syst Innov Appl Syst Innov Mod Technol ICASI 2017 814–817Google Scholar
  80. 80.
    Liu M, Shan S, Wang R, Chen X (2014) Learning expressionlets on spatio-temporal manifold for dynamic facial expression recognition in Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern RecognitionGoogle Scholar
  81. 81.
    Deaney W, Venter I, Ghaziasgar M, Dodds R (2017) A comparison of facial feature representation methods for automatic facial expression recognition. Proc South African Inst Comput Sci Inf Technol 10:1–10Google Scholar
  82. 82.
    Guo Y, Tao D, Yu J, Hao X, Li Y, Tao D (2016) Deep Neural Networks with Relativity Learning for facial expression recognition,” in 2016 IEEE International Conference on Multimedia and Expo Workshop, ICMEWGoogle Scholar
  83. 83.
    Kim B-K, Roh J, Dong S-Y, Lee S-Y (2016) Hierarchical committee of deep convolutional neural networks for robust facial expression recognition. J Multimodal User InterfacesGoogle Scholar
  84. 84.
    Wei W, Yang X-L, Zhou B et al (2012) Combined energy minimization for image reconstruction from few views. Math Probl EngGoogle Scholar
  85. 85.
    Wei W, Yang X-L, Shen P-Y et al (2012) Holes detection in anisotropic Sensornets: topological methods. Int J Distribut Sensor NetwGoogle Scholar
  86. 86.
    Wei W, Qi Y (2011) Information potential fields navigation wireless adoc sensor networks. Sensors 11(5):4794–4807Google Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2019

Authors and Affiliations

  1. 1.South China University of TechnologyGuangzhouChina
  2. 2.Guangdong Research Center of Artificial Intelligence and Traditional Chinese Medicine EngineeringSouth China University of TechnologyGuangzhouChina

Personalised recommendations

Cite article

Buy options