Deep peak-neutral difference feature for facial expression recognition

  • Jingying Chen
  • Ruyi Xu
  • Leyuan Liu


Facial expression recognition (FER) is important in vision-related applications. Deep neural networks demonstrate impressive performance for face recognition; however, it should be noted that this method relies heavily on a great deal of manually labeled training data, which is not available for facial expressions in real-world applications. Hence, we propose a powerful facial feature called deep peak–neutral difference (DPND) for FER. DPND is defined as the difference between two deep representations of the fully expressive (peak) and neutral facial expression frames. The difference tends to emphasize the facial parts that are changed in the transition from the neutral to the expressive face and to eliminate the face identity information retained in the fine-tuned deep neural network for facial expression, the network has been trained on large-scale face recognition dataset. Furthermore, unsupervised clustering and semi-supervised classification methods are presented to automatically acquire the neutral and peak frames from the expression sequence. The proposed facial expression feature achieved encouraging results on public databases, which suggests that it has strong potential to recognize facial expressions in real-world applications.


Facial expression recognition Facial-expression feature Deep neutral network 



This work was supported by the National Social Science Foundation of China (Grant no. 16BSH107).


  1. 1.
    Almaev TR, Valstar MF (2013) Local gabor binary patterns from three orthogonal planes for automatic facial expression recognition. In: Affective Computing and Intelligent Interaction (ACII), 2013 Humaine Association Conference IEEE, pp 356–361Google Scholar
  2. 2.
    An L, Yang S, Bhanu B (2015) Efficient smile detection by extreme learning machine. Neurocomputing 149:354–363CrossRefGoogle Scholar
  3. 3.
    Arthur D, Vassilvitskii S (2007) "K-means++: the advantages of careful seeding." SODA ‘07: proceedings of the eighteenth annual ACM-SIAM symposium on discrete Algorithms, pp. 1027–1035Google Scholar
  4. 4.
    Chang CC, Lin CJ (2011) LIBSVM: a library for support vector machines. ACM Trans Intell Syst Technol 27(2):1–27CrossRefGoogle Scholar
  5. 5.
    Chao WL, Ding JJ, Liu JZ (2015) Facial expression recognition based on improved local binary pattern and class-regularized locality preserving projection. Signal Process 117:1–10CrossRefGoogle Scholar
  6. 6.
    Chen J, Chen D, Gong Y, et al (2012) Facial expression recognition using geometric and appearance features. In: Proceedings of the 4th international conference on internet multimedia computing and service. ACM, pp 29–33Google Scholar
  7. 7.
    Chen J, Luo N, Liu Y, Liu L, Zhang K, Kolodziej J (2016) A hybrid intelligence-aided approach to affect-sensitive e-learning. Computing 98(1–2):215–233MathSciNetCrossRefGoogle Scholar
  8. 8.
    Corneanu CA, Simon MO, Cohn JF, Guerrero SE (2016) Survey on rgb, 3d, thermal, and multimodal approaches for facial expression recognition: history, trends, and affect-related applications. IEEE Trans Pattern Anal Mach Intell 38(8):1548–1568CrossRefGoogle Scholar
  9. 9.
    Dapogny A, Bailly K, Dubuisson S (2017) Dynamic pose-robust facial expression recognition by multi-view pairwise conditional random forests. IEEE Trans Affect Comput 99:1–14CrossRefGoogle Scholar
  10. 10.
    Ding GG, Guo YC, Zhou JL (2016) Large-scale cross-modality search via collective matrix factorization hashing. IEEE Transactions Image Processing 25(11):5427–5440MathSciNetCrossRefGoogle Scholar
  11. 11.
    Ding GG, Zhou JL, Guo YC (2017) Large-scale image retrieval with sparse embedded hashing. Neurocomputing 257:24–36CrossRefGoogle Scholar
  12. 12.
    Ding H, Zhou S K, Chellappa R (2017) Facenet2expnet: Regularizing a deep face recognition net for expression recognition. In: Automatic Face & Gesture Recognition (FG 2017), 2017 12th IEEE International Conference. IEEE, pp 118–126Google Scholar
  13. 13.
    Guo YC, Ding GG, Liu L (2017) Learning to hash with optimized anchor embedding for scalable retrieval. IEEE Transactions Image Processing 26(3):1344–1354MathSciNetCrossRefGoogle Scholar
  14. 14.
    Guo YC, Ding GG, Han JG (2017) Robust quantization for general similarity search. IEEE Transactions Image Processing, pp 949–963Google Scholar
  15. 15.
    Guo YC, Ding GG, Han JG (2017) Zero-shot learning with transferred samples. IEEE Transactions Image Processing 26(7):3277–3290MathSciNetCrossRefGoogle Scholar
  16. 16.
    Jung H, Lee S, Yim J, et al (2015) Joint fine-tuning in deep neural networks for facial expression recognition. In: Proceedings of the IEEE International Conference on Computer Vision IEEE, pp 2983–2991Google Scholar
  17. 17.
    Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. In: Advances in neural information processing systems. NIPS, pp 1097–1105Google Scholar
  18. 18.
    Lee S, Plataniotis K, Ro Y (2014) Intra-class variation reduction using training expression images for sparse representation based facial expression recognition. IEEE Trans Affect Comput 5(3):340–351CrossRefGoogle Scholar
  19. 19.
    Levi G, Hassner T (2015) Emotion recognition in the wild via convolutional neural networks and mapped binary patterns. In: Proceedings of the 2015 ACM on international conference on multimodal interaction. ACM, pp 503–510Google Scholar
  20. 20.
    Li YF, Zhou ZH (2015) Towards making unlabeled data never hurt. IEEE Trans Pattern Anal Mach Intell 37(1):175–188CrossRefGoogle Scholar
  21. 21.
    Li H, Lin Z, Shen X, et al (2015) A convolutional neural network cascade for face detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition IEEE pp 5325–5334Google Scholar
  22. 22.
    Liu M, Li S, Shan S, et al (2013) Au-aware deep networks for facial expression recognition. In: Automatic Face and Gesture Recognition (FG), 2013 10th IEEE International Conference and Workshops. IEEE, pp 1–6Google Scholar
  23. 23.
    Lopes AT, Aguiar ED, Souza AFD, Oliveira-Santos T (2017) Facial expression recognition with convolutional neural networks: coping with few data and the training sample order. Pattern Recogn 61:610–628CrossRefGoogle Scholar
  24. 24.
    Lu X, Yuan Y, Zheng X (2013) Image super-resolution via double sparsity regularized manifold learning. IEEE Trans Circuits Syst Video Technol 23(12):2022–2033CrossRefGoogle Scholar
  25. 25.
    Lu X, Yuan Y, Yan P (2014) Alternatively constrained dictionary learning for image super-resolution. IEEE Trans Cybern 44(3):366–377CrossRefGoogle Scholar
  26. 26.
    Lu X, Yuan Y, Zheng X (2017) Jointly dictionary learning for change detection in multispectral imagery. IEEE Transactions on Cybernetics (IEEE) 47(4):884–897CrossRefGoogle Scholar
  27. 27.
    Lu X, Li X, Zheng X (2017) Latent semantic minimal hashing for image retrieval. IEEE Transactions on Image processing (IEEE) 26(1):355–368MathSciNetCrossRefGoogle Scholar
  28. 28.
    Luo Z, Liu L, Chen J, et al (2016) Spontaneous smile recognition for interest detection. In: Proceedings of the Chinese Conference on Pattern Recogntion IEEE, pp 119–130Google Scholar
  29. 29.
    Mohammadi MR, Fatemizadeh E, Mahoor MH (2014) PCA-based dictionary building for accurate facial expression recognition via sparse representation. J Vis Commun Image Represent 25(5):1082–1092CrossRefGoogle Scholar
  30. 30.
    Mollahosseini A, Graitzer G, Borts E, et al (2014) Expressionbot: an emotive lifelike robotic face for face-to-face communication. In: Humanoid Robots (Humanoids), 2014 14th IEEE-RAS International Conference IEEE, pp 1098–1103Google Scholar
  31. 31.
    Mollahosseini A, Chan D, Mahoor MH (2016) Going deeper in facial expression recognition using deep neural networks. In: Applications of Computer Vision (WACV), 2016 I.E. Winter Conference. IEEE, pp 1–10Google Scholar
  32. 32.
    Ng HW, Nguyen VD, Vonikakis V, et al (2015) Deep learning for emotion recognition on small datasets using transfer learning. Proceedings of the 2015 ACM on international conference on multimodal interaction. ACM, pp 443–449Google Scholar
  33. 33.
    Parkhi O M, Vedaldi A, Zisserman A (2015) Deep face recognition. In: Proceedings of the British machine vision conference. BMVC, pp 1–12Google Scholar
  34. 34.
    Scherer S, Stratou G, Mahmoud M, et al (2013) Automatic behavior descriptors for psychological disorder analysis. In: Automatic Face and Gesture Recognition (FG), 2013 10th IEEE International Conference and Workshops. IEEE, pp 1–8Google Scholar
  35. 35.
    Szegedy C, Liu W, Jia Y, et al (2015) Going deeper with convolutions. In: Proceedings of the IEEE conference on computer vision and pattern recognition IEEE, pp 1–9Google Scholar
  36. 36.
    Tang Y (2013) Deep learning using linear support vector machines. arXiv preprint arXiv:1306,0239Google Scholar
  37. 37.
  38. 38.
    Wang Z, Ying Z (2012) Facial expression recognition based on local phase quantization and sparse representation. In: Natural Computation (ICNC), 2012 Eighth International Conference IEEE, pp 222–225Google Scholar
  39. 39.
    Yao XW, Han JW, Cheng G (2016) Semantic annotation of high-resolution satellite images via weakly supervised learning. IEEE Trans Geosci Remote Sens 54:3660–3671CrossRefGoogle Scholar
  40. 40.
    Yao XW, Han JW, Zhang DW (2017) Revisiting co-saliency detection: a novel approach based on two-stage multi-view spectral rotation co-clustering. IEEE Trans Image Process 26:3196–3209MathSciNetCrossRefGoogle Scholar
  41. 41.
    Yin L, Wei X, Sun Y, et al (2006) A 3D facial expression database for facial behavior research. In: automatic face and gesture recognition, 2006 FGR 2006 7th international conference. IEEE, pp 211–216Google Scholar
  42. 42.
    Zhang X, Mahoor M, Mavadati S (2015) Facial expression recognition using {l} _ {p} -norm MKL multiclass-SVM. Machine Vision & Applications 26(4):467–483CrossRefGoogle Scholar
  43. 43.
    Zhang DW, Han JW, Li C (2015) Detection of co-salient objects by looking deep and wide. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2994–3002Google Scholar
  44. 44.
    Zhang T, Zheng W, Cui Z, Zong Y, Yan J, Yan K (2016) A deep neural network-driven feature learning method for multi-view facial expression recognition. IEEE Transactions on Multimedia 18(12):2528–2536CrossRefGoogle Scholar
  45. 45.
    Zhang DW, Han JW, Jiang L (2017) Revealing event saliency in unconstrained video collection. IEEE Trans Image Process 26:1746–1758MathSciNetCrossRefGoogle Scholar
  46. 46.
    Zhang DW, Meng DY, Han JW (2017) Co-saliency detection via a self-paced multiple-instance learning framework. IEEE Trans Pattern Anal Mach Intell 39:865–878CrossRefGoogle Scholar
  47. 47.
    Zhao X, Liang X, Liu L, et al (2016) Peak-piloted deep network for facial expression recognition. In: European conference on computer vision. Springer International Publishing, pp 425–442Google Scholar
  48. 48.
    Zhao J, Han J, Shao L (2017) Unconstrained face recognition using a set-to-set distance measure on deep learned features. IEEE Transactions on Circuits and Systems for Video Technology, (99) :1–11Google Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2018

Authors and Affiliations

  1. 1.National Engineering Research Center for E-LearningCentral China Normal UniversityWuhanChina

Personalised recommendations