Adaptively Weighted Facial Expression Recognition by Feature Fusion Under Intense Illumination Condition

  • Yuechuan Sun
  • Jun Yu
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10639)


Accurate and robust facial expression recognition under complex environment is a challenging task. In this paper, we propose an adaptively weighted facial expression recognition approach to overcome the intense illumination difficulty by fusing diverse illumination invariant appearance features. First, a novel neural-network-based adaptive weight assignment strategy is designed to eliminate the adverse illumination variations efficiently and effectively. Then, a feature fusion strategy is developed to combine two of the most successful illumination invariant appearance descriptors, namely Gabor and Local Binary Patterns (LBP), for giving comprehensive and robust description of facial expressions. Extensive experiments demonstrate the superiority of the proposed approach on the common used CK+ dataset, especially the adaptive weight assignment for the significant improvement of recognition accuracy under extreme and intense illumination conditions.


Facial expression recognition Illumination Feature fusion Adaptive weight 



This work is supported by the National Natural Science Foundation of China (61572450, 61303150), Anhui Provincial Natural Science Foundation (1708085QF138), the Fundamental Research Funds for the Central Universities (WK2350000002), the Open Funding Project of State Key Lab of Virtual Reality Technology and Systems, Beihang University (BUAA-VR-16KF-12), the Open Funding Project of State Key Lab of Novel Software Technology, Nanjing University (KFKT2016B08).


  1. 1.
    Zou, X., Kittler, J., Messer, K.: Illumination invariant face recognition: a survey. In: IEEE International Conference on Biometrics: Theory, Applications, and Systems, pp. 1–8 (2007)Google Scholar
  2. 2.
    Jobson, D.J., Rahman, Z., Woodell, G.A.: A multiscale retinex for bridging the gap between color images and the human observation of scenes. IEEE Trans. Image Process. 6(7), 965–976 (1997)CrossRefGoogle Scholar
  3. 3.
    Wang, H., Li, S.Z., Wang, Y.: Face recognition under varying lighting conditions using self quotient image. In: IEEE International Conference on Automatic Face and Gesture Recognition, pp. 819–824 (2004)Google Scholar
  4. 4.
    Ou, J., Bai, X.B., Pei, Y., Ma, L., Liu, W.: Automatic facial expression recognition using gabor filter and expression analysis. In: International Conference on Computer Modeling and Simulation, vol. 2, pp. 215–218 (2010)Google Scholar
  5. 5.
    Shan, C., Gong, S., McOwan, P.W.: Facial expression recognition based on local binary patterns: a comprehensive study. Image Vis. Comput. 27(6), 803–816 (2009)CrossRefGoogle Scholar
  6. 6.
    Paysan, P., Knothe, R., Amberg, B., Romdhani, S., Vetter, T.: A 3D face model for pose and illumination invariant face recognition. In: IEEE International Conference on Advanced Video and Signal Based Surveillance, pp. 296–301 (2009)Google Scholar
  7. 7.
    Wang, S., Liu, Z., Lv, S., Lv, Y., Wu, G., Peng, P., Chen, F., Wang, X.: A natural visible and infrared facial expression database for expression recognition and emotion inference. IEEE Trans. Multimedia 12(7), 682–691 (2010)CrossRefGoogle Scholar
  8. 8.
    Di, W., Zhang, L., Zhang, D., Pan, Q.: Studies on hyperspectral face recognition in visible spectrum with feature band selection. IEEE Trans. Syst. Man Cybern. Part A Syst. Hum. 40(6), 1354–1361 (2010)CrossRefGoogle Scholar
  9. 9.
    Ahonen, T., Hadid, A., Pietikäinen, M.: Face description with local binary patterns: application to face recognition. IEEE Trans. Pattern Anal. Mach. Intell. 28(12), 2037–2041 (2006)CrossRefMATHGoogle Scholar
  10. 10.
    Mohamed, A.A., Gavrilova, M.L., Yampolskiy, R.V.: Artificial face recognition using wavelet adaptive LBP with directional statistical features. In: International Conference on Cyberworlds, pp. 23–28 (2012)Google Scholar
  11. 11.
    Viola, P., Jones, M.: Rapid object detection using a boosted cascade of simple features. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition, vol. 1, pp. 511–518 (2001)Google Scholar
  12. 12.
    Ojala, T., Pietikäinen, M., Mäenpää, T.: Multiresolution gray-scale and rotation invariant texture classification with local binary patterns. IEEE Trans. Pattern Anal. Mach. Intell. 24(7), 971–987 (2002)CrossRefMATHGoogle Scholar
  13. 13.
    Yang, J., Yang, J., Zhang, D., Lu, J.: Feature fusion: parallel strategy vs. serial strategy. Pattern Recogn. 36(6), 1369–1381 (2003)CrossRefMATHGoogle Scholar
  14. 14.
    Ekman, P., Friesen, W.: Facial Action Coding System: A Technique for the Measurement of Facial Movement. Consulting Psychologists, San Francisco (1978)Google Scholar
  15. 15.
    Lucey, P., Cohn, J.F., Kanade, T., Saragih, J., Ambadar, Z., Matthews, I.: The extended Cohn-Kanade dataset (CK+): a complete dataset for action unit and emotion-specified expression. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, pp. 94C–101 (2010)Google Scholar
  16. 16.
    Chen, J., Chen, Z., Chi, Z., Fu, H.: Facial expression recognition in video with multiple feature fusion. IEEE Trans. Affect. Comput. PP, 1 (2016)Google Scholar
  17. 17.
    Happy, S., Routray, A.: Automatic facial expression recognition using features of salient facial patches. IEEE Trans. Affect. Comput. 6(1), 1–12 (2015)CrossRefGoogle Scholar
  18. 18.
    Mollahosseini, A., Chan, D., Mahoor, M.H.: Going deeper in facial expression recognition using deep neural networks. In: IEEE Winter Conference on Applications of Computer Vision, pp. 1–10 (2016)Google Scholar
  19. 19.
    Jung, H., Lee, S., Yim, J., Park, S., Kim, J.: Joint fine-tuning in deep neural networks for facial expression recognition. In: International Conference on Computer Vision, pp. 2983–2991 (2015)Google Scholar
  20. 20.
    Liu, M., Li, S., Shan, S., Wang, R., Chen, X.: Deeply learning deformable facial action parts model for dynamic expression analysis. In: Asian Conference on Computer Vision, pp. 143–157 (2014)Google Scholar
  21. 21.
    Zavaschi, T.H., Britto, A.S., Oliveira, L.E., Koerich, A.L.: Fusion of feature sets and classifiers for facial expression recognition. Expert Syst. Appl. 40(2), 646–655 (2013)CrossRefGoogle Scholar
  22. 22.
    Jain, S., Hu, C., Aggarwal, J.K.: Facial expression recognition with temporal modeling of shapes. In: International Conference on Computer Vision Workshops, pp. 1642–1649 (2011)Google Scholar
  23. 23.
    Liu, M., Li, S., Shan, S., Chen, X.: AU-aware deep networks for facial expression recognition. In: IEEE International Conference and Workshops on Automatic Face and Gesture Recognition, pp. 1–6 (2013)Google Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  1. 1.Department of AutomationUniversity of Science and Technology of ChinaHefeiChina

Personalised recommendations