Fully Automated and Highly Accurate Dense Correspondence for Facial Surfaces

  • Carl Martin Grewe
  • Stefan Zachow
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9914)


We present a novel framework for fully automated and highly accurate determination of facial landmarks and dense correspondence, e.g. a topologically identical mesh of arbitrary resolution, across the entire surface of 3D face models. For robustness and reliability of the proposed approach, we are combining 2D landmark detectors and 3D statistical shape priors with a variational matching method. Instead of matching faces in the spatial domain only, we employ image registration to align the 2D parametrization of the facial surface to a planar template we call the Unified Facial Parameter Domain (ufpd). This allows us to simultaneously match salient photometric and geometric facial features using robust image similarity measures while reasonably constraining geometric distortion in regions with less significant features. We demonstrate the accuracy of the dense correspondence established by our framework on the BU3DFE database with 2500 facial surfaces and show, that our framework outperforms current state-of-the-art methods with respect to the fully automated location of facial landmarks.


Dense face matching Face shape and appearance models Markerless motion capture 



Special thanks go to Olaf Hellwich, Gabriel Le Roux, The Anh Pham, Sven-Kristofer Pilz, Honglei Wang, and Martin Zänker for their valuable contribution and support. Our research is funded by the Image Knowledge Gestaltung. Cluster of Excellence at the Humboldt-Universität zu Berlin, with financial support from the German Research Foundation as a part of the Excellence Initiative.

Supplementary material

434776_1_En_38_MOESM1_ESM.pdf (9 mb)
Supplementary material 1 (pdf 9165 KB)


  1. 1.
    Brunton, A., Salazar, A., Bolkart, T., Wuhrer, S.: Review of statistical shape spaces for 3D data with comparative analysis for human faces. Comput. Vis. Image Underst. 128, 1–17 (2014). ElsevierCrossRefGoogle Scholar
  2. 2.
    Yin, L., Wei, X., Sun, Y., Wang, J., Rosato, M.J.: A 3D facial expression database for facial behavior research. In: IEEE International Conference Automatic Face and Gesture Recognition (FGR), pp. 211–216 (2006)Google Scholar
  3. 3.
    Savran, A., Alyüz, N., Dibeklioğlu, H., Çeliktutan, O., Gökberk, B., Sankur, B., Akarun, L.: Bosphorus database for 3D face analysis. In: Schouten, B., Juul, N.C., Drygajlo, A., Tistarelli, M. (eds.) BIOID 2008. LNCS, vol. 5372, pp. 47–56. Springer, Heidelberg (2008)CrossRefGoogle Scholar
  4. 4.
    Paysan, P., Knothe, R., Amberg, B., Romdhani, S., Vetter, T.: A 3D face model for pose and illumination invariant face recognition. In: IEEE International Conference On Advanced Video and Signal Based Surveillance (AVSS), pp. 296–301. IEEE (2009)Google Scholar
  5. 5.
    Booth, J., Roussos, A., Zafeiriou, S., Ponniah, A., Dunaway, D.: A 3D morphable model learnt from 10,000 faces. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2016)Google Scholar
  6. 6.
    Brunton, A., Bolkart, T., Wuhrer, S.: Multilinear wavelets: a statistical shape space for human faces. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014, Part I. LNCS, vol. 8689, pp. 297–312. Springer, Heidelberg (2014)Google Scholar
  7. 7.
    Cao, C., Weng, Y., Zhou, S., Tong, Y., Zhou, K.: Facewarehouse: a 3D facial expression database for visual computing. IEEE Trans. Vis. Comput. Graph. 20, 413–425 (2014). IEEECrossRefGoogle Scholar
  8. 8.
    Zafeiriou, S., Zhang, C., Zhang, Z.: A survey on face detection in the wild: past, present and future. Comput. Vis. Image Underst. 138, 1–24 (2015)CrossRefGoogle Scholar
  9. 9.
    Cootes, T.F., Ionita, M.C., Lindner, C., Sauer, P.: Robust and accurate shape model fitting using random forest regression voting. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (eds.) ECCV 2012, Part VII. LNCS, vol. 7578, pp. 278–291. Springer, Heidelberg (2012)CrossRefGoogle Scholar
  10. 10.
    Ren, S., Cao, X., Wei, Y., Sun, J.: Face alignment at 3000 fps via regressing local binary features. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1685–1692 (2014)Google Scholar
  11. 11.
    Kazemi, V., Sullivan, J.: One millisecond face alignment with an ensemble of regression trees. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1867–1874 (2014)Google Scholar
  12. 12.
    de la Torre, F., Chu, W.S., Xiong, X., Vicente, F., Ding, X., Cohn, J.: Intraface. In: IEEE International Conference Automatic Face and Gesture Recognition (FGR), vol. 1, pp. 1–8. IEEE (2015)Google Scholar
  13. 13.
    Segundo, M.P., Silva, L., Bellon, O.R.P., Queirolo, C.: Automatic face segmentation and facial landmark detection in range images. IEEE Trans. Syst. Man Cybern. B: Cybern. 40, 1319–1330 (2010). IEEECrossRefGoogle Scholar
  14. 14.
    Salazar, A., Wuhrer, S., Shu, C., Prieto, F.: Fully automatic expression-invariant face correspondence. Mach. Vis. Appl. 25, 859–879 (2014). SpringerCrossRefGoogle Scholar
  15. 15.
    Gilani, Z.S., Shafait, F., Mian, A.: Shape-based automatic detection of a large number of 3D facial landmarks. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4639–4648 (2015)Google Scholar
  16. 16.
    Matthews, I., Baker, S.: Active appearance models revisited. Int. J. Comput. Vis. 60(2), 135–164 (2004)CrossRefGoogle Scholar
  17. 17.
    Theobald, B.J., Matthews, I., Mangini, M., Spies, J.R., Brick, T.R., Cohn, J.F., Boker, S.M.: Mapping and manipulating facial expression. Lang. Speech 52(2–3), 369–386 (2009)CrossRefGoogle Scholar
  18. 18.
    Litke, N., Droske, M., Rumpf, M., Schröder, P.: An image processing approach to surface matching. Symp. Geom. Process. (SGP) 255, 207–216 (2005)Google Scholar
  19. 19.
    Bronstein, A.M., Bronstein, M.M., Kimmel, R.: Generalized multidimensional scaling: a framework for isometry-invariant partial surface matching. Proc. Natl Acad. Sci. 103, 1168–1172 (2006)MathSciNetCrossRefzbMATHGoogle Scholar
  20. 20.
    Windheuser, T., Schlickewei, U., Schmidt, F.R., Cremers, D.: Geometrically consistent elastic matching of 3D shapes: a linear programming solution. In: 2011 International Conference on Computer Vision (ICCV), pp. 2134–2141. IEEE (2011)Google Scholar
  21. 21.
    Dubrovina, A., Kimmel, R.: Approximately isometric shape correspondence by matching pointwise spectral features and global geodesic structures. In: Advances in Adaptive Data Analysis, vol. 3, pp. 203–228. World Scientific (2011)Google Scholar
  22. 22.
    Van Kaick, O., Zhang, H., Hamarneh, G., Cohen-Or, D.: A survey on shape correspondence. Comput. Graph. Forum 30(6), 1681–1707 (2011)CrossRefGoogle Scholar
  23. 23.
    Besl, P.J., McKay, N.D.: Method for registration of 3-D shapes. In: Robotics-DL tentative, International Society for Optics and Photonics, pp. 586–606 (1992)Google Scholar
  24. 24.
    Myronenko, A., Song, X.: Point set registration: coherent point drift. IEEE Trans. Pattern Anal. Mach. Intell. 32, 2262–2275 (2010)CrossRefGoogle Scholar
  25. 25.
    Lamecker, H., Seeba, M., Hege, H.C., Deuhard, P.: A 3D statistical shape model of the pelvic bone for segmentation. In: Proceedings SPIE Medical Imaging, vol. 5370, pp. 1341–1351 (2004)Google Scholar
  26. 26.
    Duy, N.T., Lamecker, H., Kainmueller, D., Zachow, S.: Automatic detection and classification of teeth in CT data. In: Ayache, N., Delingette, H., Golland, P., Mori, K. (eds.) MICCAI 2012. LNCS, vol. 7510, pp. 609–616. Springer, Heidelberg (2012). doi: 10.1007/978-3-642-33415-3_75 CrossRefGoogle Scholar
  27. 27.
    Ehlke, M., Ramm, H., Lamecker, H., Hege, H.C., Zachow, S.: Fast generation of virtual x-ray images for reconstruction of 3D anatomy. IEEE Trans. Vis. Comput. Graph. 19(12), 2673–2682 (2013)CrossRefGoogle Scholar
  28. 28.
    Blanz, V., Vetter, T.: Face recognition based on fitting a 3D morphable model. IEEE Trans. Pattern Anal. Mach. Intell. 25, 1063–1074 (2003). IEEECrossRefGoogle Scholar
  29. 29.
    Blanz, V., Vetter, T.: A morphable model for the synthesis of 3D faces. In: Proceedings of the Annual Conference on Computer Graphics and Interactive Techniques, pp. 187–194. ACM Press/Addison-Wesley Publishing Co. (1999)Google Scholar
  30. 30.
    Bradley, D., Heidrich, W., Popa, T., Sheffer, A.: High resolution passive facial performance capture. ACM Trans. Graph. (TOG) 29, 1–10 (2010). ACMCrossRefGoogle Scholar
  31. 31.
    Beeler, T., Hahn, F., Bradley, D., Bickel, B., Beardsley, P., Gotsman, C., Sumner, R.W., Gross, M.: High-quality passive facial performance capture using anchor frames. ACM Trans. Graph. (TOG) 30, 75–85 (2011). ACMCrossRefGoogle Scholar
  32. 32.
    Fyffe, G., Jones, A., Alexander, O., Ichikari, R., Debevec, P.: Driving high-resolution facial scans with video performance capture. ACM Trans. Graph. (TOG) 34, 8 (2014). ACMCrossRefGoogle Scholar
  33. 33.
    Kaiser, M., Störmer, A., Arsić, D., Rigoll, G.: Non-rigid registration of 3D facial surfaces with robust outlier detection. In: IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 1–6 (2009)Google Scholar
  34. 34.
    Savran, A., Sankur, B.: Non-rigid registration of 3D surfaces by deformable 2D triangular meshes. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 1–6 (2008)Google Scholar
  35. 35.
    Milborrow, S., Nicolls, F.: Active shape models with SIFT descriptors and MARS. In: International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISAPP), pp. 380–387 (2014)Google Scholar
  36. 36.
    Beeler, T., Bickel, B., Beardsley, P., Sumner, B., Gross, M.: High-quality single-shot capture of facial geometry. ACM Trans. Graph. (TOG) 29, 41–50 (2010). ACMCrossRefGoogle Scholar
  37. 37.
    Kazhdan, M., Bolitho, M., Hoppe, H.: Poisson surface reconstruction. In: Proceedings of the Fourth Eurographics Symposium on Geometry Processing, vol. 7, pp. 61–70 (2006)Google Scholar
  38. 38.
    Pérez, P., Gangnet, M., Blake, A.: Poisson image editing. ACM Trans. Graph. (TOG) 22, 313–318 (2003). ACMCrossRefGoogle Scholar
  39. 39.
    Kälberer, F., Nieser, M., Polthier, K.: Quadcover-surface parameterization using branched coverings. Comput. Graph. Forum 26, 375–384 (2007). WileyCrossRefzbMATHGoogle Scholar
  40. 40.
    Bishop, C.M.: Pattern Recognition and Machine Learning. Springer, New York (2006)zbMATHGoogle Scholar
  41. 41.
    Floater, M.S.: Parametrization and smooth approximation of surface triangulations. Comput. Aided Geom. Des. 14, 231–250 (1997). ElsevierMathSciNetCrossRefzbMATHGoogle Scholar
  42. 42.
    Floater, M.S.: Mean value coordinates. Comput. Aided Geom. Des. 20, 19–27 (2003). ElsevierMathSciNetCrossRefzbMATHGoogle Scholar
  43. 43.
    Klein, S., Staring, M.: elastix the manual v4.7. Technical report, Image Sciences Institute, University Medical Center, Utrecht (2014)Google Scholar
  44. 44.
    Staring, M., Klein, S., Pluim, J.P.: A rigidity penalty term for nonrigid registration. In: Medical Physics, vol. 34, pp. 4098–4108. American Association of Physicists in Medicine (2007)Google Scholar
  45. 45.
    Klein, S., Staring, M., Murphy, K., Viergever, M., Pluim, J.P., et al.: elastix: a toolbox for intensity-based medical image registration. IEEE Trans. Med. Imaging 29, 196–205 (2010). IEEECrossRefGoogle Scholar
  46. 46.
    Ekman, P., Friesen, W.V.: Unmasking the face: a guide to recognizing emotions from facial clues (2003)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2016

Authors and Affiliations

  1. 1.Mathematics for Life and Materials SciencesZuse Institute BerlinBerlinGermany

Personalised recommendations