Semantic 3D Reconstruction of Heads

  • Fabio ManincheddaEmail author
  • Christian Häne
  • Bastien Jacquet
  • Amaël Delaunoy
  • Marc Pollefeys
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9910)


We present a novel approach that jointly reconstructs the geometry of a human head and semantically segments it into labels such as skin, hair and eyebrows. In order to get faithful reconstructions from data captured in uncontrolled environments, we propose to adapt a recently introduced implicit volumetric surface normal based shape prior formulation. Shape prior based approaches critically rely on an accurate alignment between the data and the prior to succeed. To this end, we propose an automatic alignment procedure for the used shape prior formulation. We evaluate our alignment procedure thoroughly and show head reconstruction results on challenging datasets.


Face Head Semantic Multi-label Shape prior Alignment 



This project is supported by Grant 16703.1 PFES-ES of CTI Switzerland and the Swiss National Science Foundation under Project Nr. 143422.

Supplementary material

Supplementary material 1 (mp4 15457 KB)

419981_1_En_40_MOESM2_ESM.pdf (5.9 mb)
Supplementary material 2 (pdf 6011 KB)


  1. 1.
    Agarwal, S., Mierle, K., Others: Ceres solver.
  2. 2.
    Beeler, T., Bickel, B., Beardsley, P., Sumner, B., Gross, M.: High-quality single-shot capture of facial geometry. ACM Trans. Graph. (ToG) 29(4), 40 (2010)CrossRefGoogle Scholar
  3. 3.
    Beeler, T., Bickel, B., Noris, G., Marschner, S., Beardsley, P., Sumner, R.W., Gross, M.: Coupled 3D reconstruction of sparse facial hair and skin. ACM Trans. Graph. (ToG) 31(4), 117 (2012)CrossRefGoogle Scholar
  4. 4.
    Brunton, A., Salazar, A., Bolkart, T., Wuhrer, S.: Review of statistical shape spaces for 3D data with comparative analysis for human faces. Comput. Vis. Image Underst. (CVIU) 128, 1–17 (2014)CrossRefGoogle Scholar
  5. 5.
    Chambolle, A., Cremers, D., Pock, T.: A convex approach to minimal partitions. SIAM J. Imaging Sci. 5(4), 1113–1158 (2012)MathSciNetCrossRefzbMATHGoogle Scholar
  6. 6.
    Curless, B., Levoy, M.: A volumetric method for building complex models from range images. In: International Conference on Computer graphics and interactive techniques (SIGGRAPH) (1996)Google Scholar
  7. 7.
    Dame, A., Prisacariu, V.A., Ren, C.Y., Reid, I.: Dense reconstruction using 3D object shape priors. In: Conference on Computer Vision and Pattern Recognition (CVPR) (2013)Google Scholar
  8. 8.
    Echevarria, J.I., Bradley, D., Gutierrez, D., Beeler, T.: Capturing and stylizing hair for 3D fabrication. ACM Trans. Graphics (ToG) 33(4), 125 (2014)CrossRefGoogle Scholar
  9. 9.
    Esedoglu, S., Osher, S.J.: Decomposition of images by the anisotropic Rudin-Osher-Fatemi model. Commun. Pure Appl. Math. 57(12), 1609–1626 (2004)MathSciNetCrossRefzbMATHGoogle Scholar
  10. 10.
    Fanelli, G., Dantone, M., Gall, J., Fossati, A., Van Gool, L.: Random forests for real time 3D face analysis. Int. J. Comput. Vis. (IJCV) 101(3), 437–458 (2013)CrossRefGoogle Scholar
  11. 11.
    Furukawa, Y., Ponce, J.: Accurate, dense, and robust multi-view stereopsis. IEEE Trans. Pattern Anal. Mach. Intell. (TPAMI) 32(8), 1362–1376 (2010)CrossRefGoogle Scholar
  12. 12.
    Garrido, P., Valgaerts, L., Wu, C., Theobalt, C.: Reconstructing detailed dynamic face geometry from monocular video. ACM Trans. Graph. (ToG) 32(6), 158 (2013)CrossRefGoogle Scholar
  13. 13.
    Häne, C., Heng, L., Lee, G.H., Sizov, A., Pollefeys, M.: Real-time direct dense matching on fisheye images using plane-sweeping stereo. In: International Conference on 3D Vision (3DV) (2014)Google Scholar
  14. 14.
    Häne, C., Savinov, N., Pollefeys, M.: Class specific 3D object shape priors using surface normals. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2014)Google Scholar
  15. 15.
    Häne, C., Zach, C., Cohen, A., Angst, R., Pollefeys, M.: Joint 3D scene reconstruction and class segmentation. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2013)Google Scholar
  16. 16.
    Ichim, A.E., Bouaziz, S., Pauly, M.: Dynamic 3D avatar creation from hand-held video input. ACM Trans. Graph. (ToG) 34(4), 45 (2015)CrossRefGoogle Scholar
  17. 17.
    Jourabloo, A., Liu, X.: Pose-invariant 3D face alignment. In: International Conference on Computer Vision (ICCV) (2015)Google Scholar
  18. 18.
    Kazhdan, M., Bolitho, M., Hoppe, H.: Poisson surface reconstruction. In: Eurographics Symposium on Geometry Processing (SGP) (2006)Google Scholar
  19. 19.
    Ladicky, L., Russell, C., Kohli, P., Torr, P.H.: Associative hierarchical random fields. IEEE Trans. Pattern Anal. Mach. Intell. (TPAMI) 36(6), 1056–1077 (2014)CrossRefGoogle Scholar
  20. 20.
    Lempitsky, V., Boykov, Y.: Global optimization for shape fitting (2007)Google Scholar
  21. 21.
    Lorensen, W.E., Cline, H.E.: Marching cubes: a high resolution 3D surface construction algorithm. In: Conference on Computer Graphics and Interactive Techniques (SIGGRAPH) (1987)Google Scholar
  22. 22.
    Luo, L., Li, H., Paris, S., Weise, T., Pauly, M., Rusinkiewicz, S.: Multi-view hair capture using orientation fields. In: Conference on Computer Vision and Pattern Recognition (CVPR) (2012)Google Scholar
  23. 23.
    Paysan, P., Knothe, R., Amberg, B., Romdhani, S., Vetter, T.: A 3D face model for pose and illumination invariant face recognition (2009)Google Scholar
  24. 24.
    Pock, T., Chambolle, A.: Diagonal preconditioning for first order primal-dual algorithms in convex optimization. In: International Conference on Computer Vision (ICCV) (2011)Google Scholar
  25. 25.
    Preparata, F.P., Shamos, M.: Computational Geometry: An Introduction. Springer, New York (1985)CrossRefzbMATHGoogle Scholar
  26. 26.
    Prisacariu, V.A., Segal, A.V., Reid, I.: Simultaneous monocular 2D segmentation, 3D pose recovery and 3D reconstruction. In: Lee, K.M., Matsushita, Y., Rehg, J.M., Hu, Z. (eds.) ACCV 2012, Part I. LNCS, vol. 7724, pp. 593–606. Springer, Heidelberg (2013)Google Scholar
  27. 27.
    Ren, S., Cao, X., Wei, Y., Sun, J.: Face alignment at 3000 FPS via regressing local binary features. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2014)Google Scholar
  28. 28.
    Saragih, J.M., Lucey, S., Cohn, J.F.: Deformable model fitting by regularized landmark mean-shift. Int. J. Comput. Vis. (IJCV) 91(2), 200–215 (2011)MathSciNetCrossRefzbMATHGoogle Scholar
  29. 29.
    Savinov, N., Häne, C., Ladicky, L., Pollefeys, M.: Semantic 3D reconstruction with continuous regularization and ray potentials using a visibility consistency constraint. In: Conference on Computer Vision and Pattern Recognition (CVPR) (2016)Google Scholar
  30. 30.
    Savinov, N., Ladicky, L., Häne, C., Pollefeys, M.: Discrete optimization of ray potentials for semantic 3D reconstruction. In: Conference on Computer Vision and Pattern Recognition (CVPR) (2015)Google Scholar
  31. 31.
    Stühmer, J., Schröder, P., Cremers, D.: Tree shape priors with connectivity constraints using convex relaxation on general graphs. In: International Conference on Computer Vision Proceedings (ICCV) (2013)Google Scholar
  32. 32.
    Tanskanen, P., Kolev, K., Meier, L., Camposeco, F., Saurer, O., Pollefeys, M.: Live metric 3D reconstruction on mobile phones. In: International Conference on Computer Vision (ICCV) (2013)Google Scholar
  33. 33.
    Vedaldi, A., Fulkerson, B.: VLFeat: an open and portable library of computer vision algorithms (2008).
  34. 34.
    Wei, Y., Ofek, E., Quan, L., Shum, H.Y.: Modeling hair from multiple views. ACM Trans. Graphics (TOG) 24(3), 816–820 (2005)CrossRefGoogle Scholar
  35. 35.
    Wu, C.: VisualSFM: a visual structure from motion system (2011).
  36. 36.
    Zach, C.: Fast and high quality fusion of depth maps. In: International Symposium on 3D Data Processing, Visualization and Transmission (3DPVT) (2008)Google Scholar
  37. 37.
    Zach, C., Häne, C., Pollefeys, M.: What is optimized in tight convex relaxations for multi-label problems? In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2012)Google Scholar
  38. 38.
    Zach, C., Häne, C., Pollefeys, M.: What is optimized in convex relaxations for multilabel problems: Connecting discrete and continuously inspired map inference. IEEE Trans. Pattern Anal. Mach. Intell. (TPAMI) 36(1), 157–170 (2014)CrossRefGoogle Scholar

Copyright information

© Springer International Publishing AG 2016

Authors and Affiliations

  • Fabio Maninchedda
    • 1
    Email author
  • Christian Häne
    • 2
  • Bastien Jacquet
    • 3
  • Amaël Delaunoy
    • 1
  • Marc Pollefeys
    • 1
    • 4
  1. 1.ETH ZurichZurichSwitzerland
  2. 2.UC BerkeleyBerkeleyUSA
  3. 3.Kitware SASVilleurbanneFrance
  4. 4.MicrosoftRedmondUSA

Personalised recommendations