Skip to main content
SpringerLink
Log in

Holistic 3D face and head reconstruction with geometric details from a single image

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

3D morphable model (3DMM) performs favorably in 3D face reconstruction from a single image. However, the 3D face created by 3DMM lacks the fine detail of shape and texture. Existing works address this issue by exploiting a neural network that generates a displacement map for finer details. This way enhances the quality of the reconstructed face but increases the complexity because it utilizes a generative model. In addition, previous works reconstruct only the frontal part of the human face without the full head representation due to the use of the simple 3DMM model. They also neglect the facial-region-only constraint in doing texture extraction, which yields incorrect facial details. In this paper, we answer these challenges by proposing a practical framework that combines two major neural-network modules, i.e. DPMMNet and ResHairNet networks. In detail, we initially generate a coarse 3D face shape through the 3DMM fitting and mesh deformation. Then, we propose the DPMMNet, a network that estimates a displacement map from an RGB input image for producing detailed geometric information. Then, we craft the ResHairNet module, a neural network function that removes non-facial regions and fills them with artificial but plausible skin color and texture. Experimental results show that the proposed method reconstructs the 3D face and full head with a higher level of detail while also achieving approximately 12 times faster computation time than the previous method

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

References

  1. Blanz V, Romdhani S, Vetter T (2002) Face identification across different poses and illuminations with a 3D morphable model. In: Proc. of the IEEE international conference on automatic face gesture recognition, pp 202–207

  2. Blanz V, Scherbaum K, Vetter T, Seidel H P (2004) Exchanging faces in images. Comput Graph Forum 23(3):669–676

    Article  Google Scholar 

  3. Blanz V, Vetter T (1999) A morphable model for the synthesis of 3D faces. In: Proc. of SIGGRAPH, pp 187–194

  4. Blanz V, Vetter T (2003) Face recognition based on fitting a 3D morphable model. IEEE Trans Pattern Anal Mach Intell 25(9):1063–1074

    Article  Google Scholar 

  5. Bulat A, Tzimiropoulos G (2017) How far are we from solving the 2D & 3D face alignment problem?(and a dataset of 230,000 3D facial landmarks). In: Proc. of the IEEE international conference on computer vision

  6. Chen BC, Chen CS, Hsu WH (2014) Cross-age reference coding for age-invariant face recognition and retrieval. In: Proc. of european conference on computer vision

  7. Cheng J, Li Y, Wang J, Yu L, Wang S (2019) Exploiting effective facial patches for robust gender recognition. Tsinghua Sci Technol 24(3):333–345

    Article  Google Scholar 

  8. Chu B, Romdhani S, Chen L (2014) 3D-aided face recognition robust to expression and pose variations. In: Proc. of the IEEE conference on computer vision and pattern recognition

  9. Deng Y, Yang J, Xu S, Chen D, Jia Y, Tong X (2019) Accurate 3D face reconstruction with weakly-supervised learning: From single image to image set. In: Proc. of the IEEE conference on computer vision and pattern recognition workshops

  10. Dou P, Shah SK, Kakadiaris IA (2017) End-to-end 3D face reconstruction with deep neural networks. In: Proc. of the IEEE conference on computer vision and pattern recognition

  11. Feng Y, Wu F, Shao X, Wang Y, Zhou X (2018) Joint 3D face reconstruction and dense alignment with position map regression network. In: Proc. of the european conference on computer vision, pp 534–551

  12. Genova K, Cole F, Maschinot A, Sarna A, Vlasic D, Freeman W T (2018) Unsupervised training for 3D morphable model regression. In: Proc. of the IEEE conference on computer vision and pattern recognition, pp 8377–8386

  13. Guo J, Zhu X, Yang Y, Yang F, Lei Z, Li SZ (2020) Towards fast accurate and stable 3d dense face alignment

  14. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proc. of the IEEE conference on computer vision and pattern recognition

  15. Hu G, Yan F, Chan C H, Deng W, Christmas W, Kittler J, Robertson NM (2016) Face recognition using a unified 3D morphable model. In: Proc. of the european conference on computer vision

  16. Ichim AE, Bouaziz S, Pauly M (2015) Dynamic 3D avatar creation from hand-held video input. ACM Trans Graph 34(4):1–14

    Article  Google Scholar 

  17. Isola P, Zhu JY, Zhou T, Efros AA (2017) Image-to-image translation with conditional adversarial networks. In: Proc. of the IEEE conference on computer vision and pattern recognition

  18. Jackson PT, Atapour-Abarghouei A, Bonner S, Breckon T, Obara B (2019) Style augmentation: data augmentation via style randomization Proc. of the IEEE conference on computer vision and pattern recognition workshops

  19. Lumentut JS, Lee J, Park IK (2019) Hair removal on face images using a deep neural network. In: Proceedings of the Korean society of broadcast engineers conference, The Korean Institute of Broadcast and Media Engineers, pp 163–165

  20. Or-El R, Rosman G, Wetzler A, Kimmel R, Bruckstein AM (2015) RGBD-fusion: real-time high precision depth recovery. In: Proc. of the IEEE conference on computer vision and pattern recognition

  21. Romdhani S, Vetter T (2003) Efficient, robust and accurate fitting of a 3D morphable model. In: Proc. of the european conference on computer vision

  22. Roth J, Tong Y, Liu X (2015) Unconstrained 3D face reconstruction. In: Proc. of the IEEE conference on computer vision and pattern recognition

  23. Sela M, Richardson E, Kimmel R (2017) Unrestricted facial geometry reconstruction using image-to-image translation. In: Proc. of the IEEE international conference on computer vision

  24. Tang H, Hu Y, Fu Y, Hasegawa-Johnson M, Huang TS (2008) Real-time conversion from a single 2D face image to a 3D text-driven emotive audio-visual avatar. In: Proc. of the IEEE international conference on multimedia and expo

  25. Tran AT, Hassner T, Masi I, Paz E, Nirkin Y, Medioni GG (2018) Extreme 3D face reconstruction: seeing through occlusions. In: Proc. of the IEEE conference on computer vision and pattern recognition

  26. Tran L, Liu X (2018) Nonlinear 3D face morphable model. In: Proc. of the IEEE conference on computer vision and pattern recognition

  27. Yang C, Lv Z (2020) Gender based face aging with cycle-consistent adversarial networks. Image Vis Comput 103945:100

    Google Scholar 

  28. Yang F, Wang J, Shechtman E, Bourdev L, Metaxas D (2011) Expression flow for 3D-aware face component transfer. ACM Trans Graph 30:60

    Article  Google Scholar 

  29. Yin L, Wei X, Sun Y, Wang J, Rosato M J (2006) A 3D facial expression database for facial behavior research. In: Proc. of the international conference on automatic face and gesture recognition

  30. Zhao J, Pan Z, Duan F, Lv Z, Li J, Zhou Q, Shang X, Sun J, Wang P (2019) A survey on 3d face recognition based on geodesics. J Inf Hiding Multimed Signal Process 10:368–383

    Google Scholar 

  31. Zhao JL, Wu ZK, Pan ZK, Duan FQ, Li JH, Lv ZH, Wang K (2018) 3d face similarity measure by fréchet distances of geodesics. J Comput Sci Technol 33:207–222

    Article  Google Scholar 

  32. Zhu JY, Park T, Isola P, Efros AA (2017) Unpaired image-to-image translation using cycle-consistent adversarial networks. In: Proc. of the IEEE international conference on computer vision

  33. Zhu X, Lei Z, Yan J, Yi D, Li SZ (2015) High-fidelity pose and expression normalization for face recognition in the wild. In: Proc. of the IEEE conference on computer vision and pattern recognition

  34. Zhu X, Liu X, Lei Z, Li S Z (2017) Face alignment in full pose range: a 3d total solution. IEEE Trans Pattern Anal Mach Intell 41(1):78–92

    Article  Google Scholar 

Download references

Acknowledgements

This work was partly supported by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (No. 2022R1A4A1033549, NRF-2019R1A2C1006706). This work was partly supported by Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (2020-0-01389, Artificial Intelligence Convergence Research Center (Inha University), RS-2022-00155915, Artificial Intelligence Convergence Innovation Human Resources Development (Inha University)).

Author information

Authors and Affiliations

  1. Department of Electrical and Computer Engineering, Inha University, Incheon, 22212, Korea

    Jungwoo Lee, Jonathan Samuel Lumentut & In Kyu Park

Authors
  1. Jungwoo Lee
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jonathan Samuel Lumentut
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. In Kyu Park
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to In Kyu Park.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Lee, J., Lumentut, J.S. & Park, I.K. Holistic 3D face and head reconstruction with geometric details from a single image. Multimed Tools Appl 81, 38217–38233 (2022). https://doi.org/10.1007/s11042-022-13590-9

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-022-13590-9

Keywords

Search

Navigation