Advertisement

Robust Face Alignment Using a Mixture of Invariant Experts

  • Oncel Tuzel
  • Tim K. MarksEmail author
  • Salil Tambe
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9909)

Abstract

Face alignment, which is the task of finding the locations of a set of facial landmark points in an image of a face, is useful in widespread application areas. Face alignment is particularly challenging when there are large variations in pose (in-plane and out-of-plane rotations) and facial expression. To address this issue, we propose a cascade in which each stage consists of a mixture of regression experts. Each expert learns a customized regression model that is specialized to a different subset of the joint space of pose and expressions. The system is invariant to a predefined class of transformations (e.g., affine), because the input is transformed to match each expert’s prototype shape before the regression is applied. We also present a method to include deformation constraints within the discriminative alignment framework, which makes our algorithm more robust. Our algorithm significantly outperforms previous methods on publicly available face alignment datasets.

Supplementary material

419978_1_En_50_MOESM1_ESM.pdf (742 kb)
Supplementary material 1 (pdf 742 KB)
419978_1_En_50_MOESM2_ESM.mp4 (22.4 mb)
Supplementary material 2 (mp4 22902 KB)
419978_1_En_50_MOESM3_ESM.mp4 (13.1 mb)
Supplementary material 3 (mp4 13424 KB)
419978_1_En_50_MOESM4_ESM.mp4 (8.9 mb)
Supplementary material 4 (mp4 9132 KB)

References

  1. 1.
    Cootes, T.F., Taylor, C.J., Cooper, D.H., Graham, J.: Active shape models-their training and application. Comput. Vis. Image Understand. 61(1), 38–59 (1995)CrossRefGoogle Scholar
  2. 2.
    Cootes, T.F., Edwards, G.J., Taylor, C.J.: Active appearance models. IEEE Trans. Pattern Anal. Mach. Intell. 23(6), 681–685 (2001)CrossRefGoogle Scholar
  3. 3.
    Sauer, P., Cootes, T.F., Taylor, C.J.: Accurate regression procedures for active appearance models. In: BMVC, pp. 1–11 (2011)Google Scholar
  4. 4.
    Sung, J., Kim, D.: Adaptive active appearance model with incremental learning. Pattern Recogn. Lett. 30(4), 359–367 (2009)CrossRefGoogle Scholar
  5. 5.
    Tzimiropoulos, G., Pantic, M.: Optimization problems for fast AAM fitting in-the-wild. In: Proceedings of IEEE International Conference on Computer Vision (ICCV) (2013)Google Scholar
  6. 6.
    Romdhani, S., Gong, S., Psarrou, A., et al.: A multi-view nonlinear active shape model using kernel PCA. BMVC 10, 483–492 (1999)Google Scholar
  7. 7.
    Cootes, T.F., Wheeler, G.V., Walker, K.N., Taylor, C.J.: View-based active appearance models. Image Vis. Comput. 20(9), 657–664 (2002)CrossRefGoogle Scholar
  8. 8.
    Asthana, A., Marks, T., Jones, M., Tieu, K., M.V., R.: Fully automatic pose-invariant face recognition via 3d pose normalization. In: IEEE International Conference on Computer Vision (ICCV), pp. 937–944, November 2011Google Scholar
  9. 9.
    Cootes, T.F., Wheeler, G.V., Walker, K.N., Taylor, C.J.: Coupled-view active appearance models. In: BMVC, pp. 52–61 (2000)Google Scholar
  10. 10.
    Hu, C., Xiao, J., Matthews, I., Baker, S., Cohn, J.F., Kanade, T.: Fitting a single active appearance model simultaneously to multiple images. In: BMVC, pp. 1–10 (2004)Google Scholar
  11. 11.
    Cristinacce, D., Cootes, T.F.: Feature detection and tracking with constrained local models. In: BMVC (2006)Google Scholar
  12. 12.
    Cristinacce, D., Cootes, T.F.: Boosted regression active shape models. In: BMVC, pp. 1–10 (2007)Google Scholar
  13. 13.
    Zhou, F., Brandt, J., Lin, Z.: Exemplar-based graph matching for robust facial landmark localization. In: 2013 IEEE International Conference on Computer Vision (ICCV), pp. 1025–1032. IEEE (2013)Google Scholar
  14. 14.
    Smith, B.M., Zhang, L.: Joint face alignment with non-parametric shape models. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (eds.) ECCV 2012. LNCS, vol. 7574, pp. 43–56. Springer, Heidelberg (2012). doi: 10.1007/978-3-642-33712-3_4 Google Scholar
  15. 15.
    Tzimiropoulos, G., Pantic, M.: Gauss-newton deformable part models for face alignment in-the-wild. In: 2014 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1851–1858. IEEE (2014)Google Scholar
  16. 16.
    Tuzel, O., Porikli, F., Meer, P.: Learning on lie groups for invariant detection and tracking. In: IEEE Conference on Computer Vision and Pattern Recognition, 2008, CVPR 2008, pp. 1–8. IEEE (2008)Google Scholar
  17. 17.
    Xiong, X., De la Torre, F.: Supervised descent method and its applications to face alignment. In: 2013 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 532–539. IEEE (2013)Google Scholar
  18. 18.
    Asthana, A., Zafeiriou, S., Cheng, S., Pantic, M.: Incremental face alignment in the wild. In: 2014 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1859–1866. IEEE (2014)Google Scholar
  19. 19.
    Liu, X.: Discriminative face alignment. IEEE Trans. Pattern Anal. Machine Intell. 31(11), 1941–1954 (2009)CrossRefGoogle Scholar
  20. 20.
    Kazemi, V., Sullivan, J.: Face alignment with part-based modeling. In: Proceedings of the British Machine Vision Conference, p. 27:1. BMVA Press (2011)Google Scholar
  21. 21.
    Saragih, J.M., Lucey, S., Cohn, J.F.: Deformable model fitting by regularized landmark mean-shift. Int. J. Comput. Vis. 91(2), 200–215 (2011)MathSciNetCrossRefzbMATHGoogle Scholar
  22. 22.
    Dollár, P., Welinder, P., Perona, P.: Cascaded pose regression. In: 2010 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1078–1085. IEEE (2010)Google Scholar
  23. 23.
    Ren, S., Cao, X., Wei, Y., Sun, J.: Face alignment at 3000 fps via regressing local binary features. In: 2014 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1685–1692. IEEE (2014)Google Scholar
  24. 24.
    Kazemi, V., Sullivan, J.: One millisecond face alignment with an ensemble of regression trees. In: 2014 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1867–1874. IEEE (2014)Google Scholar
  25. 25.
    Cao, X., Wei, Y., Wen, F., Sun, J.: Face alignment by explicit shape regression. Int. J. Comput. Vis. 107(2), 177–190 (2014)MathSciNetCrossRefGoogle Scholar
  26. 26.
    Tzimiropoulos, G.: Project-out cascaded regression with an application to face alignment. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2015Google Scholar
  27. 27.
    Yan, J., Lei, Z., Yi, D., Li, S.Z.: Learn to combine multiple hypotheses for accurate face alignment. In: 2013 IEEE International Conference on Computer Vision Workshops (ICCVW), pp. 392–396. IEEE (2013)Google Scholar
  28. 28.
    Cootes, T.F., Ionita, M.C., Lindner, C., Sauer, P.: Robust and accurate shape model fitting using random forest regression voting. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (eds.) ECCV 2012. LNCS, vol. 7578, pp. 278–291. Springer, Heidelberg (2012). doi: 10.1007/978-3-642-33786-4_21 CrossRefGoogle Scholar
  29. 29.
    Alabort-i Medina, J., Antonakos, E., Booth, J., Snape, P., Zafeiriou, S.: Menpo: A comprehensive platform for parametric image alignment and visual deformable models. In: Proceedings of the ACM International Conference on Multimedia, MM 2014, pp. 679–682. ACM, New York (2014)Google Scholar
  30. 30.
    Xiong, X., De la Torre, F.: Global supervised descent method. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2015Google Scholar
  31. 31.
    Xiao, S., Yan, S., Kassim, A.A.: Facial landmark detection via progressive initialization. In: The IEEE International Conference on Computer Vision (ICCV) Workshops, December 2015Google Scholar
  32. 32.
    Zhang, J., Kan, M., Shan, S., Chen, X.: Leveraging datasets with varying annotations for face alignment via deep regression network. In: The IEEE International Conference on Computer Vision (ICCV), December 2015Google Scholar
  33. 33.
    Zhou, E., Fan, H., Cao, Z., Jiang, Y., Yin, Q.: Extensive facial landmark localization with coarse-to-fine convolutional network cascade. In: ICCV Workshop (2013)Google Scholar
  34. 34.
    Zhang, J., Shan, S., Kan, M., Chen, X.: Coarse-to-fine auto-encoder networks (CFAN) for real-time face alignment. In: ECCV (2014)Google Scholar
  35. 35.
    Zhu, S., Li, C., Change Loy, C., Tang, X.: Face alignment by coarse-to-fine shape searching. In: CVPR (2015)Google Scholar
  36. 36.
    Burgos-Artizzu, X.P., Perona, P., Dollar, P.: Robust face landmark estimation under occlusion. In: The IEEE International Conference on Computer Vision (ICCV), December 2013Google Scholar
  37. 37.
    Yu, X., Lin, Z., Brandt, J., Metaxas, D.N.: Consensus of regression for occlusion-robust facial feature localization. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8692, pp. 105–118. Springer, Heidelberg (2014). doi: 10.1007/978-3-319-10593-2_8 Google Scholar
  38. 38.
    Rao, A., Miller, D., Rose, K., Gersho, A.: Mixture of experts regression modeling by deterministic annealing. IEEE Trans. Sig. Process. 45(11), 2811–2820 (1997)CrossRefGoogle Scholar
  39. 39.
    Sagonas, C., Tzimiropoulos, G., Zafeiriou, S., Pantic, M.: A semi-automatic methodology for facial landmark annotation. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR-Workshops), 5th Workshop on Analysis and Modeling of Faces and Gestures (AMFG2013), Portland Oregon, USA, June 2013Google Scholar
  40. 40.
    Sagonas, C., Tzimiropoulos, G., Zafeiriou, S., Pantic, M.: 300 faces in-the-wild challenge: the first facial landmark localization challenge. In: Proceedings of IEEE International Conference on Computer Vision (ICCV-Workshops), 300 Faces in-the-Wild Challenge (300-W), Sydney, Australia, December 2013Google Scholar
  41. 41.
    Zhu, X., Ramanan, D.: Face detection, pose estimation, and landmark localization in the wild. In: 2012 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2879–2886. IEEE (2012)Google Scholar
  42. 42.
    Lowe, D.G.: Distinctive image features from scale-invariant keypoints. Int. J. Comput. Vis. 60(2), 91–110 (2004)CrossRefGoogle Scholar
  43. 43.
    Čech, J., Franc, V., Uřičář, M., Matas, J.: Multi-view facial landmark detection by using a 3D shape model. Image Vis. Comput. 47, 60–70 (2016)CrossRefGoogle Scholar
  44. 44.
    Belhumeur, P.N., Jacobs, D.W., Kriegman, D., Kumar, N.: Localizing parts of faces using a consensus of exemplars. In: 2011 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 545–552. IEEE (2011)Google Scholar
  45. 45.
    Le, V., Brandt, J., Lin, Z., Bourdev, L., Huang, T.S.: Interactive facial feature localization. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (eds.) ECCV 2012. LNCS, vol. 7574, pp. 679–692. Springer, Heidelberg (2012). doi: 10.1007/978-3-642-33712-3_49 Google Scholar
  46. 46.
    Yang, H., Jia, X., Loy, C.C., Robinson, P.: An empirical study of recent face alignment methods. CoRR abs/1511.05049 (2015)Google Scholar

Copyright information

© Springer International Publishing AG 2016

Authors and Affiliations

  1. 1.Mitsubishi Electric Research Labs (MERL)CambridgeUSA
  2. 2.Intel CorporationSanta ClaraUSA

Personalised recommendations