Facial Expression-Aware Face Frontalization

Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10113)


Face frontalization is a rising technique for view-invariant face analysis. It enables a non-frontal facial image to recover its general facial appearances to frontal view. A few pioneering works have been proposed very recently. However, face frontalization with detailed facial expression recovering is still very challenging due to the non-linear relationships between head-pose and expression variations. In this paper, we propose a novel facial expression-aware face frontalization method aiming at reconstructing the frontal view while maintaining vivid appearances with regards to facial expressions. First of all, we design multiple face shape models as the reference templates in order to fit in with various shape of facial expressions. Each template describes a set of typical facial actions referred to Facial Action Coding System (FACS). Then a template matching strategy is applied by measuring a weighted Chi Square error such that the input image can be matched with the most approximate template. Finally, Robust Statistical face Frontalization (RSF) method is employed for the task of frontal view recovery. This method is validated on a spontaneous facial expression database and the experimental results show that the proposed method outperforms the state-of-the-art methods.


Face Frontalization Facial Action Coding System (FACS) Head Pose Approximate Template Alternating Direction Method Of Multipliers (ADMM) 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.



This work was supported by EU seventh framework programme under grant agreement No. 611391, DREAM, and the EPSRC project, 4D Facial Sensing and Modelling (EP/N025849/1).


  1. 1.
    Valstar, M., Jiang, B., Mehu, M., Pantic, M., Scherer, K.: The first facial expression recognition and analysis challenge. In: IEEE International Conference and Workshops on Automatic Face and Gesture Recognition, pp. 921–926 (2011)Google Scholar
  2. 2.
    Sariyanidi, E., Gunes, H., Cavallaro, A.: Automatic analysis of facial affect: a survey of registration, representation and recognition. IEEE Trans. Pattern Anal. Mach. Intell. 37, 1113–1133 (2015)CrossRefGoogle Scholar
  3. 3.
    Zeng, Z., Pantic, M., Roisman, G.I., Huang, T.S.: A survey of affect recognition methods: Audio, visual, and spontaneous expressions. IEEE Trans. Pattern Anal. Mach. Intell. 31, 39–58 (2009)CrossRefGoogle Scholar
  4. 4.
    Moore, S., Bowden, R.: Local binary patterns for multi-view facial expression recognition. Comput. Vis. Image Underst. 115, 541–558 (2011)CrossRefGoogle Scholar
  5. 5.
    Hesse, N., Gehrig, T., Gao, H., Ekenel, H.K.: Multi-view facial expression recognition using local appearance features. Int. J. Comput. Vis. 83, 178–194 (2011)Google Scholar
  6. 6.
    Eleftheriadis, S., Rudovic, O., Pantic, M.: Discriminative shared Gaussian processes for multiview and view-invariant facial expression recognition. IEEE Trans. Image Process. 24, 189–204 (2015)MathSciNetCrossRefGoogle Scholar
  7. 7.
    Rudovic, O., Pantic, M., Patras, I.: Coupled Gaussian processes for pose-invariant facial expression recognition. IEEE Trans. Pattern Anal. Mach. Intell. 35, 1357–1369 (2013)CrossRefGoogle Scholar
  8. 8.
    Tariq, U., Yang, J., Huang, T.S.: Multi-view facial expression recognition analysis with generic sparse coding feature. In: Fusiello, A., Murino, V., Cucchiara, R. (eds.) ECCV 2012. LNCS, vol. 7585, pp. 578–588. Springer, Heidelberg (2012). doi: 10.1007/978-3-642-33885-4_58 CrossRefGoogle Scholar
  9. 9.
    Tariq, U., Yang, J., Huang, T.S.: Supervised super-vector encoding for facial expression recognition. Pattern Recogn. Lett. 46, 89–95 (2014)CrossRefGoogle Scholar
  10. 10.
    Lowe, D.G.: Object recognition from local scale-invariant features. In: Proceedings of 7th IEEE International Conference on Computer Vision, vol. 2, pp. 1150–1157 (1999)Google Scholar
  11. 11.
    Jeni, L.A., Cohn, J.F., Kanade, T.: Dense 3D face alignment from 2D videos in real-time. In: IEEE International Conference and Workshops on Automatic Face and Gesture Recognition, pp. 1–8 (2015)Google Scholar
  12. 12.
    J. Roth, Y.T., Liu, X.: Unconstrained 3D face reconstruction. In: IEEE International Conference Computer Vision Workshops, pp. 2606–2615 (2015)Google Scholar
  13. 13.
    Hassner, T., Harel, S., Paz, E., Enbar, R.: Effective face frontalization in unconstrained images. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 4295–4304 (2015)Google Scholar
  14. 14.
    Ho, H.T., Chellappa, R.: Pose-invariant face recognition using markov random fields. IEEE Trans. Image Process. 22, 1573–1584 (2013)MathSciNetCrossRefGoogle Scholar
  15. 15.
    Sagonas, C., Panagakis, Y., Zafeiriou, S., Pantic, M.: Robust statistical face frontalization. In: Proceedings of IEEE International Conference on Computer Vision, pp. 3871–3879 (2015)Google Scholar
  16. 16.
    Taheri, S., Qiu, Q., Chellappa, R.: Structure-preserving sparse decomposition for facial expression analysis. IEEE Trans. Image Process. 23, 3590–3603 (2014)MathSciNetCrossRefGoogle Scholar
  17. 17.
    Xiong, X., la Torre, F.D.: Supervised descent method and its applications to face alignment. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 532–539 (2013)Google Scholar
  18. 18.
    Xue, M., Liu, W., Li, L.: Person-independent facial expression recognition via hierarchical classification. In: IEEE International Conference on Intelligent Sensors, Sensor Networks and Information Processing, pp. 449–454 (2013)Google Scholar
  19. 19.
    Zhang, Z., Ganesh, A., Liang, X., Ma, Y.: TILT: transform invariant low-rank textures. Int. J. Comput. Vis. 99, 1–24 (2012)MathSciNetCrossRefzbMATHGoogle Scholar
  20. 20.
    Cootes, T.F., Taylor, C.J., Cooper, D.H., Graham, J.: Active shape model-their training and application. Comput. Vis. Image Underst. 61, 38–59 (1995)CrossRefGoogle Scholar
  21. 21.
    Metthews, I., Baker, S.: Active appearance model revisited. Int. J. Comput. Vis. 60, 135–164 (2004)CrossRefGoogle Scholar
  22. 22.
    Dhall, A., Goecke, R., Lucey, S., Gedeon, T.: Static facial expression analysis in tough conditions: Data evaluation protocol and benchmark. In: IEEE International Conference Computer Vision Workshops, pp. 2106–2112 (2011)Google Scholar
  23. 23.
    Huang, D., Shan, C., Ardabilian, M., Wang, Y., Chen, L.: Local binary patterns and its application to facial image analysis: a survey. IEEE Trans. Syst. Man Cybern. Part C Appl. Rev. 41, 765–781 (2011)CrossRefGoogle Scholar
  24. 24.
    Liu, M., Li, S., Chen, X.: AU-aware deep networks for facial expression recognition. In: IEEE International Conference and Workshops on Automatic Face and Gesture Recognition, pp. 1–6 (2013)Google Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  1. 1.School of Creative TechnologiesUniversity of PortsmouthPortsmouthUK
  2. 2.Ocean University of ChinaQingdaoChina
  3. 3.School of ComputingUniversity of PortsmouthPortsmouthUK

Personalised recommendations