Abstract
3D morphable model (3DMM) performs favorably in 3D face reconstruction from a single image. However, the 3D face created by 3DMM lacks the fine detail of shape and texture. Existing works address this issue by exploiting a neural network that generates a displacement map for finer details. This way enhances the quality of the reconstructed face but increases the complexity because it utilizes a generative model. In addition, previous works reconstruct only the frontal part of the human face without the full head representation due to the use of the simple 3DMM model. They also neglect the facial-region-only constraint in doing texture extraction, which yields incorrect facial details. In this paper, we answer these challenges by proposing a practical framework that combines two major neural-network modules, i.e. DPMMNet and ResHairNet networks. In detail, we initially generate a coarse 3D face shape through the 3DMM fitting and mesh deformation. Then, we propose the DPMMNet, a network that estimates a displacement map from an RGB input image for producing detailed geometric information. Then, we craft the ResHairNet module, a neural network function that removes non-facial regions and fills them with artificial but plausible skin color and texture. Experimental results show that the proposed method reconstructs the 3D face and full head with a higher level of detail while also achieving approximately 12 times faster computation time than the previous method
Similar content being viewed by others
References
Blanz V, Romdhani S, Vetter T (2002) Face identification across different poses and illuminations with a 3D morphable model. In: Proc. of the IEEE international conference on automatic face gesture recognition, pp 202–207
Blanz V, Scherbaum K, Vetter T, Seidel H P (2004) Exchanging faces in images. Comput Graph Forum 23(3):669–676
Blanz V, Vetter T (1999) A morphable model for the synthesis of 3D faces. In: Proc. of SIGGRAPH, pp 187–194
Blanz V, Vetter T (2003) Face recognition based on fitting a 3D morphable model. IEEE Trans Pattern Anal Mach Intell 25(9):1063–1074
Bulat A, Tzimiropoulos G (2017) How far are we from solving the 2D & 3D face alignment problem?(and a dataset of 230,000 3D facial landmarks). In: Proc. of the IEEE international conference on computer vision
Chen BC, Chen CS, Hsu WH (2014) Cross-age reference coding for age-invariant face recognition and retrieval. In: Proc. of european conference on computer vision
Cheng J, Li Y, Wang J, Yu L, Wang S (2019) Exploiting effective facial patches for robust gender recognition. Tsinghua Sci Technol 24(3):333–345
Chu B, Romdhani S, Chen L (2014) 3D-aided face recognition robust to expression and pose variations. In: Proc. of the IEEE conference on computer vision and pattern recognition
Deng Y, Yang J, Xu S, Chen D, Jia Y, Tong X (2019) Accurate 3D face reconstruction with weakly-supervised learning: From single image to image set. In: Proc. of the IEEE conference on computer vision and pattern recognition workshops
Dou P, Shah SK, Kakadiaris IA (2017) End-to-end 3D face reconstruction with deep neural networks. In: Proc. of the IEEE conference on computer vision and pattern recognition
Feng Y, Wu F, Shao X, Wang Y, Zhou X (2018) Joint 3D face reconstruction and dense alignment with position map regression network. In: Proc. of the european conference on computer vision, pp 534–551
Genova K, Cole F, Maschinot A, Sarna A, Vlasic D, Freeman W T (2018) Unsupervised training for 3D morphable model regression. In: Proc. of the IEEE conference on computer vision and pattern recognition, pp 8377–8386
Guo J, Zhu X, Yang Y, Yang F, Lei Z, Li SZ (2020) Towards fast accurate and stable 3d dense face alignment
He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proc. of the IEEE conference on computer vision and pattern recognition
Hu G, Yan F, Chan C H, Deng W, Christmas W, Kittler J, Robertson NM (2016) Face recognition using a unified 3D morphable model. In: Proc. of the european conference on computer vision
Ichim AE, Bouaziz S, Pauly M (2015) Dynamic 3D avatar creation from hand-held video input. ACM Trans Graph 34(4):1–14
Isola P, Zhu JY, Zhou T, Efros AA (2017) Image-to-image translation with conditional adversarial networks. In: Proc. of the IEEE conference on computer vision and pattern recognition
Jackson PT, Atapour-Abarghouei A, Bonner S, Breckon T, Obara B (2019) Style augmentation: data augmentation via style randomization Proc. of the IEEE conference on computer vision and pattern recognition workshops
Lumentut JS, Lee J, Park IK (2019) Hair removal on face images using a deep neural network. In: Proceedings of the Korean society of broadcast engineers conference, The Korean Institute of Broadcast and Media Engineers, pp 163–165
Or-El R, Rosman G, Wetzler A, Kimmel R, Bruckstein AM (2015) RGBD-fusion: real-time high precision depth recovery. In: Proc. of the IEEE conference on computer vision and pattern recognition
Romdhani S, Vetter T (2003) Efficient, robust and accurate fitting of a 3D morphable model. In: Proc. of the european conference on computer vision
Roth J, Tong Y, Liu X (2015) Unconstrained 3D face reconstruction. In: Proc. of the IEEE conference on computer vision and pattern recognition
Sela M, Richardson E, Kimmel R (2017) Unrestricted facial geometry reconstruction using image-to-image translation. In: Proc. of the IEEE international conference on computer vision
Tang H, Hu Y, Fu Y, Hasegawa-Johnson M, Huang TS (2008) Real-time conversion from a single 2D face image to a 3D text-driven emotive audio-visual avatar. In: Proc. of the IEEE international conference on multimedia and expo
Tran AT, Hassner T, Masi I, Paz E, Nirkin Y, Medioni GG (2018) Extreme 3D face reconstruction: seeing through occlusions. In: Proc. of the IEEE conference on computer vision and pattern recognition
Tran L, Liu X (2018) Nonlinear 3D face morphable model. In: Proc. of the IEEE conference on computer vision and pattern recognition
Yang C, Lv Z (2020) Gender based face aging with cycle-consistent adversarial networks. Image Vis Comput 103945:100
Yang F, Wang J, Shechtman E, Bourdev L, Metaxas D (2011) Expression flow for 3D-aware face component transfer. ACM Trans Graph 30:60
Yin L, Wei X, Sun Y, Wang J, Rosato M J (2006) A 3D facial expression database for facial behavior research. In: Proc. of the international conference on automatic face and gesture recognition
Zhao J, Pan Z, Duan F, Lv Z, Li J, Zhou Q, Shang X, Sun J, Wang P (2019) A survey on 3d face recognition based on geodesics. J Inf Hiding Multimed Signal Process 10:368–383
Zhao JL, Wu ZK, Pan ZK, Duan FQ, Li JH, Lv ZH, Wang K (2018) 3d face similarity measure by fréchet distances of geodesics. J Comput Sci Technol 33:207–222
Zhu JY, Park T, Isola P, Efros AA (2017) Unpaired image-to-image translation using cycle-consistent adversarial networks. In: Proc. of the IEEE international conference on computer vision
Zhu X, Lei Z, Yan J, Yi D, Li SZ (2015) High-fidelity pose and expression normalization for face recognition in the wild. In: Proc. of the IEEE conference on computer vision and pattern recognition
Zhu X, Liu X, Lei Z, Li S Z (2017) Face alignment in full pose range: a 3d total solution. IEEE Trans Pattern Anal Mach Intell 41(1):78–92
Acknowledgements
This work was partly supported by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (No. 2022R1A4A1033549, NRF-2019R1A2C1006706). This work was partly supported by Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (2020-0-01389, Artificial Intelligence Convergence Research Center (Inha University), RS-2022-00155915, Artificial Intelligence Convergence Innovation Human Resources Development (Inha University)).
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher’s note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Lee, J., Lumentut, J.S. & Park, I.K. Holistic 3D face and head reconstruction with geometric details from a single image. Multimed Tools Appl 81, 38217–38233 (2022). https://doi.org/10.1007/s11042-022-13590-9
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11042-022-13590-9