Abstract
Recent years have witnessed significant progress in image-based 3D face reconstruction using deep convolutional neural networks. However, current reconstruction methods often perform improperly in self-occluded regions and can lead to inaccurate correspondences between a 2D input image and a 3D face template, hindering use in real applications. To address these problems, we propose a deep shape reconstruction and texture completion network, SRTC-Net, which jointly reconstructs 3D facial geometry and completes texture with correspondences from a single input face image. In SRTC-Net, we leverage the geometric cues from completed 3D texture to reconstruct detailed structures of 3D shapes. The SRTC-Net pipeline has three stages. The first introduces a correspondence network to identify pixel-wise correspondence between the input 2D image and a 3D template model, and transfers the input 2D image to a U-V texture map. Then we complete the invisible and occluded areas in the U-V texture map using an inpainting network. To get the 3D facial geometries, we predict coarse shape (U-V position maps) from the segmented face from the correspondence network using a shape network, and then refine the 3D coarse shape by regressing the U-V displacement map from the completed U-V texture map in a pixel-to-pixel way. We examine our methods on 3D reconstruction tasks as well as face frontalization and pose invariant face recognition tasks, using both in-the-lab datasets (MICC, MultiPIE) and in-the-wild datasets (CFP). The qualitative and quantitative results demonstrate the effectiveness of our methods on inferring 3D facial geometry and complete texture; they outperform or are comparable to the state-of-the-art.
Article PDF
Similar content being viewed by others
Avoid common mistakes on your manuscript.
References
Ichim, A. E.; Bouaziz, S.; Pauly, M. Dynamic 3D avatar creation from hand-held video input. ACM Transactions on Graphics Vol. 34, No. 4, Article No. 45, 2015.
Thies, J.; Zollhöfer, M.; Stamminger, M.; Theobalt, C.; Nießner, M. Face2Face: Real-time face capture and reenactment of RGB videos. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2387–2395, 2016.
Deng, J.; Cheng, S.; Xue, N.; Zhou, Y.; Zafeiriou, S. UV-GAN: Adversarial facial UV map completion for pose-invariant face recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 7093–7102, 2018.
Zhao, J.; Xiong, L.; Cheng, Y.; Cheng, Y.; Li, J.; Zhou, L.; Xu, Y.; Karlekar, J.; Pranata, S.; Shen, S. et al. 3D-aided deep pose-invariant face recognition. In: Proceedings of the 27th International Joint Conference on Artificial Intelligence, 1184–1190, 2018.
Bessaoudi, M.; Belahcene, M.; Ouamane, A.; Chouchane, A.; Bourennane, S. Multilinear Enhanced Fisher Discriminant Analysis for robust multimodal 2D and 3D face verification. Applied Intelligence Vol. 49, No. 4, 1339–1354, 2019.
Devi, P. R. S.; Baskaran, R. SL2E-AFRE: Personalized 3D face reconstruction using autoencoder with simultaneous subspace learning and landmark estimation. Applied Intelligence Vol. 51, No. 4, 2253–2268, 2021.
Yuan, Z. H.; Lu, T. Incremental 3D reconstruction using Bayesian learning. Applied Intelligence Vol. 39, No. 4, 761–771, 2013.
Alexander, O.; Rogers, M.; Lambeth, W.; Chiang, J. Y.; Ma, W. C.; Wang, C. C.; Debevec, P. The Digital Emily project: Achieving a photorealistic digital actor. IEEE Computer Graphics and Applications Vol. 30, No. 4, 20–31, 2010.
Jackson, A. S.; Bulat, A.; Argyriou, V.; Tzimiropoulos, G. Large pose 3D face reconstruction from a single image via direct volumetric CNN regression. In: Proceedings of the IEEE International Conference on Computer Vision, 1031–1039, 2017.
Feng, Y.; Wu, F.; Shao, X. H.; Wang, Y. F.; Zhou, X. Joint 3D face reconstruction and dense alignment with position map regression network. In: Computer Vision-ECCV 2018. Lecture Notes in Computer Science, Vol. 11218. Ferrari, V.; Hebert, M.; Sminchisescu, C.; Weiss, Y. Eds. Springer Cham, 557–574, 2018.
Deng, Y.; Yang, J. L.; Xu, S. C.; Chen, D.; Jia, Y. D.; Tong, X. Accurate 3D face reconstruction with weakly-supervised learning: From single image to image set. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, 285–295, 2019.
Blanz, V.; Vetter, T. A morphable model for the synthesis of 3D faces. In: Proceedings of the 26th Annual Conference on Computer Graphics and Interactive Techniques, 187–194, 1999.
Tran, A. T.; Hassner, T.; Masi, I.; Medioni, G. Regressing robust and discriminative 3D morphable models with a very deep neural network. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 1493–1502, 2017.
Tran, L.; Yin, X.; Liu, X. M. Disentangled representation learning GAN for pose-invariant face recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 1283–1292, 2017.
Bas, A.; Huber, P.; Smith, W. A. P.; Awais, M.; Kittler, J. 3D morphable models as spatial transformer networks. In: Proceedings of the IEEE International Conference on Computer Vision Workshops, 895–903, 2017.
Paysan, P.; Knothe, R.; Amberg, B.; Romdhani, S.; Vetter, T. A 3D face model for pose and illumination invariant face recognition. In: Proceedings of the 6th IEEE International Conference on Advanced Video and Signal Based Surveillance, 296–301, 2009.
Sela, M. T.; Richardson, E.; Kimmel, R. Unrestricted facial geometry reconstruction using image-to-image translation. In: Proceedings of the IEEE International Conference on Computer, 1585–1594, 2017.
Li, Y.; Ma, L. Q.; Fan, H. Q.; Mitchell, K. Feature-preserving detailed 3D face reconstruction from a single image. In: Proceedings of the 15th ACM SIGGRAPH European Conference on Visual Media Production, Article No. 1, 2018.
Richardson, E.; Sela, M. T.; Or-El, R.; Kimmel, R. Learning detailed face reconstruction from a single image. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 5553–5562, 2017.
Ronneberger, O.; Fischer, P.; Brox, T. U-net: Convolutional networks for biomedical image segmentation. In: Medical Image Computing and Computer-Assisted Intervention. Lecture Notes in Computer Science, Vol. 9351. Navab, N.; Hornegger, J.; Wells, W.; Frangi, A. Eds. Springer Cham, 234–241, 2015.
Shen, Y. J.; Luo, P.; Luo, P.; Yan, J. J.; Wang, X. G.; Tang, X. O. FaceID-GAN: Learning a symmetry three-player GAN for identity-preserving face synthesis. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 821–830, 2018.
Yin, X.; Yu, X.; Sohn, K.; Liu, X. M.; Chandraker, M. Towards large-pose face frontalization in the wild. In: Proceedings of the IEEE International Conference on Computer Vision, 4010–4019, 2017.
Zhou, Y. X.; Deng, J. K.; Kotsia, I.; Zafeiriou, S. Dense 3D face decoding over 2500FPS: Joint texture & shape convolutional mesh decoders. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 1097–1106, 2019.
Huang, R.; Zhang, S.; Li, T.; He, R. Beyond face rotation: Global and local perception GAN for photorealistic and identity preserving frontal view synthesis. In: Proceedings of the IEEE International Conference on Computer Vision, 2458–2467, 2017.
Gecer, B.; Ploumpis, S.; Kotsia, I.; Zafeiriou, S. GANFIT: Generative adversarial network fitting for high fidelity 3D face reconstruction. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 1155–1164, 2019.
Hassner, T.; Harel, S.; Paz, E.; Enbar, R. Effective face frontalization in unconstrained images. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 4295–4304, 2015.
Chang, F. J.; Yu, X.; Nevatia, R.; Chandraker, M. Posevariant 3D facial attribute generation. arXiv preprint arXiv:1907.10202, 2019.
Xue, N. N.; Deng, J. K.; Cheng, S. Y.; Panagakis, Y.; Zafeiriou, S. Side information for face completion: A robust PCA approach. IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 41, No. 10, 2349–2364, 2019.
Goodfellow, I.; Pouget-Abadie, J.; Mirza, M.; Xu, B.; Warde-Farley, D.; Ozair, S.; Courville, A.; Bengio, Y. Generative adversarial nets. In: Proceedings of the 27th International Conference on Neural Information Processing Systems, 2672–2680, 2014.
Asthana, A.; Marks, T. K.; Jones, M. J.; Tieu, K. H.; Rohith, M. Fully automatic pose-invariant face recognition via 3D pose normalization. In: Proceedings of the International Conference on Computer Vision, 937–944, 2011.
Cootes, T. F.; Walker, K.; Taylor, C. J. View-based active appearance models. In: Proceedings of the 4th IEEE International Conference on Automatic Face and Gesture Recognition, 227–232, 2000.
Wei, L.-Y.; Lefebvre, S.; Kwatra, V.; Turk, G. State of the art in example-based texture synthesis. In: Proceedings of the Eurographics 2009, State of the Art Reports (STARs), 93–117, 2009.
Efros, A. A.; Leung, T. K. Texture synthesis by non-parametric sampling. In: Proceedings of the 7th IEEE International Conference on Computer Vision, 1033–1038, 1999.
Li, Y. J.; Liu, S. F.; Yang, J. M.; Yang, M. H. Generative face completion. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 5892–5900, 2017.
Wen, Y. D.; Zhang, K. P.; Li, Z. F.; Qiao, Y. A discriminative feature learning approach for deep face recognition. In: Computer Vision-ECCV 2016. Lecture Notes in Computer Science, Vol. 9911. Leibe, B.; Matas, J.; Sebe, N.; Welling, M. Eds. Springer Cham, 499–515, 2016.
Sengupta, S.; Chen, J. C.; Castillo, C.; Patel, V. M.; Chellappa, R.; Jacobs, D. W. Frontal to profile face verification in the wild. In: Proceedings of the IEEE Winter Conference on Applications of Computer Vision, 1–9, 2016.
Tran, L.; Yin, X.; Liu, X. Representation learning by rotating your faces. IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 41, No. 12, 3007–3021, 2018.
Qian, Y. C.; Deng, W. H.; Hu, J. N. Unsupervised face normalization with extreme pose and expression in the wild. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 9843–9850, 2019.
Guo, J. Z.; Zhu, X. Y.; Yang, Y.; Yang, F.; Lei, Z.; Li, S. Z. Towards fast, accurate and stable 3D dense face alignment. In: Computer Vision-ECCV 2020. Lecture Notes in Computer Science, Vol. 12364. Vedaldi, A.; Bischof, H.; Brox, T.; Frahm, J. M. Eds. Springer Cham, 152–168, 2020.
Sanyal, S.; Bolkart, T.; Feng, H. W.; Black, M. J. Learning to regress 3D face shape and expression from an image without 3D supervision. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 7755–7764, 2019.
Tran, L.; Liu, X. M. Nonlinear 3D face morphable model. arXiv preprint arXiv: 1804.03786, 2018.
Tewari, A.; Zollhöfer, M.; Garrido, P.; Bernard, F.; Kim, H.; Perez, P.; Theobalt, C. Self-supervised multilevel face model learning for monocular reconstruction at over 250 Hz. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2549–2559, 2018.
Wu, S. Z.; Rupprecht, C.; Vedaldi, A. Unsupervised learning of probably symmetric deformable 3D objects from images in the wild. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 1–10, 2020.
Lin, J. K.; Yuan, Y.; Shao, T. J.; Zhou, K. Towards high-fidelity 3D face reconstruction from in-the-wild images using graph convolutional networks. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 5890–5899, 2020.
Lee, G. H.; Lee, S. W. Uncertainty-aware mesh decoder for high fidelity 3D face reconstruction. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 6099–6108, 2020.
Lattas, A.; Moschoglou, S.; Gecer, B.; Ploumpis, S.; Triantafyllou, V.; Ghosh, A.; Zafeiriou, S. AvatarMe: Realistically renderable 3D facial reconstruction “in-the-wild”. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 757–766, 2020.
Zhu, X. Y.; Liu, X. M.; Lei, Z.; Li, S. Z. Face alignment in full pose range: A 3D total solution. IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 41, No. 1, 78–92, 2019.
Zhang, K. P.; Zhang, Z. P.; Li, Z. F.; Qiao, Y. Joint face detection and alignment using multitask cascaded convolutional networks. IEEE Signal Processing Letters Vol. 23, No. 10, 1499–1503, 2016.
Iizuka, S.; Simo-Serra, E.; Ishikawa, H. Globally and locally consistent image completion. ACM Transactions on Graphics Vol. 36, No. 4, Article No. 107, 2017.
Yu, F.; Koltun, V. Multi-scale context aggregation by dilated convolutions. arXiv preprint arXiv:1511.07122, 2015.
Karras, T.; Laine, S.; Aila, T. M. A style-based generator architecture for generative adversarial networks. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 4396–4405, 2019.
Parkhi, O. M.; Vedaldi, A.; Zisserman, A. Deep face recognition. In: Proceedings of the British Machine Vision Conference, Article No. 41, 2015.
Simonyan, K.; Zisserman, A. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014.
Mescheder, L.; Geiger, A.; Nowozin, S. Which training methods for GANs do actually converge? arXiv preprint arXiv:1801.04406, 2018.
Ma, L. M.; Deng, Z. G. Real-time hierarchical facial performance capture. In: Proceedings of the ACM SIGGRAPH Symposium on Interactive 3D Graphics and Games, Article No. 11, 2019.
Tran, L.; Liu, F.; Liu, X. M. Towards high-fidelity nonlinear 3D face morphable model. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 1126–1135, 2019.
Tewari, A.; Zollhoefer, M.; Bernard, F.; Garrido, P.; Kim, H.; Perez, P.; Theobalt, C. High-fidelity monocular face reconstruction based on an unsupervised modelbased face autoencoder. IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 42, No. 2, 357–370, 2018.
Sengupta, S.; Kanazawa, A.; Castillo, C. D.; Jacobs, D. W. SfSNet: Learning shape, reflectance and illuminance of faces ‘in the wild’. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 6296–6305, 2018.
Yin, L. J.; Chen, X. C.; Sun, Y.; Worm, T.; Reale, M. A high-resolution 3D dynamic facial expression database. In: Proceedings of the 8th IEEE International Conference on Automatic Face & Gesture Recognition, 1–6, 2008.
Savran, A.; Alyüz, N.; Dibeklioğlu, H.; Çeliktutan, O.; Gökberk, B.; Sankur, B.; Akarun, L. Bosphorus database for 3D face analysis. In: Biometrics and Identity Management. Lecture Notes in Computer Science, Vol. 5372. Schouten, B.; Juul, N. C.; Drygajlo, A.; Tistarelli, M. Eds. Springer Berlin Heidelberg, 47–56, 2008.
White, J. D.; Ortega-Castrillón, A.; Matthews, H.; Zaidi, A. A.; Ekrami, O.; Snyders, J.; Fan, Y.; Penington, T.; Van Dongen, S.; Shriver, M. D. MeshMonk: Open-source large-scale intensive 3D phenotyping. Scientific Reports Vol. 9, No. 1, 6085, 2019.
Hormann, K.; Lévy, B.; Sheffer, A. Mesh parameterization: Theory and practice. In: Proceedings of the ACM SIGGRAPH 2007 Courses, 1-es, 2007.
Gross, R.; Matthews, I.; Cohn, J.; Kanade, T.; Baker, S. Multi-PIE. Image and Vision Computing Vol. 28, No. 5, 807–813, 2010.
Baltrusaitis, T.; Zadeh, A.; Lim, Y. C.; Morency, L. P. OpenFace 2.0: Facial behavior analysis toolkit. In: Proceedings of the 13th IEEE International Conference on Automatic Face & Gesture Recognition, 59–66, 2018.
Sankaranarayanan, S.; Alavi, A.; Castillo, C. D.; Chellappa, R. Triplet probabilistic embedding for face verification and clustering. arXiv preprint arXiv:1604.05417, 2016.
Zhao, J.; Cheng, Y.; Xu, Y.; Xiong, L.; Li, J. S.; Zhao, F.; Jayashree, K.; Pranata, S.; Shen, S.; Xing, J. L. Towards pose invariant face recognition in the wild. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2207–2216, 2018.
Chen, D.; Cao, X.; Wang, L.; Wen, F.; Sun, J. Bayesian face revisited: A joint formulation. In: Proceedings of the 12th European Conference on Computer Vision, 566–579, 2012.
Cao, K. D.; Rong, Y.; Li, C.; Tang, X. O.; Loy, C. C. Pose-robust face recognition via deep residual equivariant mapping. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 5187–5196, 2018.
Catmull, E. E. A subdivision algorithm for computer display of curved surfaces. Technical Report, CSTD-74-006. The University of Utah, 1974.
Hughes, J. F.; van Dam, A.; Foley, J. D.; McGuire, M.; Feiner, S. K.; Akeley, K.; Sklar, D. F. Computer Graphics: Principles and Practice, 3rd edn. Addison-Wesley, 2014.
Deng, J.; Guo, J.; Xue, N.; Zafeiriou, S. ArcFace: Additive angular margin loss for deep face recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 4690–4699, 2019.
Jackson, A. S.; Bulat, A.; Argyriou, V.; Tzimiropoulos, G. Large pose 3D face reconstruction from a single image via direct volumetric CNN regression. In: Proceedings of the IEEE International Conference on Computer Vision, 1031–1039, 2017.
Guo, Y. D.; Zhang, L.; Hu, Y. X.; He, X. D.; Gao, J. F. MS-celeb-1M: A dataset and benchmark for large-scale face recognition. In: Computer Vision — ECCV 2016. Lecture Notes in Computer Science, Vol. 9907. Leibe, B.; Matas, J.; Sebe, N.; Welling, M. Eds. Springer Cham, 87–102, 2016.
Li, A. N.; Shan, S. G.; Gao, W. Coupled bias-variance tradeoff for cross-pose face recognition. IEEE Transactions on Image Processing Vol. 21, No. 1, 305–315, 2012.
Wu, X.; He, R.; Sun, Z. N.; Tan, T. N. A light CNN for deep face representation with noisy labels. IEEE Transactions on Information Forensics and Security Vol. 13, No. 11, 2884–2896, 2018.
Ding, C. X.; Tao, D. C. Pose-invariant face recognition with homography-based normalization. Pattern Recognition Vol. 66, 144–152, 2017.
Xiong, C.; Zhao, X. W.; Tang, D. H.; Jayashree, K.; Yan, S. C.; Kim, T. K. Conditional convolutional neural network for modality-aware face recognition. In: Proceedings of the IEEE International Conference on Computer Vision, 3667–3675, 2015.
Hu, Y. B.; Wu, X.; Yu, B.; He, R.; Sun, Z. N. Pose-guided photorealistic face rotation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 8398–8406, 2018.
Bagdanov, A. D.; del Bimbo, A.; Masi, I. The Florence 2D/3D hybrid face dataset. In: Proceedings of the Joint ACM Workshop on Human Gesture and Behavior Understanding, 79–80, 2011.
Acknowledgements
This work was supported by the National Natural Science Foundation of China (Nos. U1613211 and U1813218), and Shenzhen Research Program (Nos. JCYJ20170818164704758 and JCYJ20150925163005055). We would like to thank Yu Deng et al. for their Deep 3D Face work in 3D face analysis, whose contribution to this field permitted our further study.
Author information
Authors and Affiliations
Corresponding author
Additional information
Xiaoxing Zeng is now a Ph.D. student at Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences, majoring in computer applied technology. His research interests include computer vision and deep learning.
Zhelun Wu is currently visiting at Shenzhen Institute of Advanced Technology. He received his master degree from Princeton Univeristy in 2018 and B.Eng. degree from Tsinghua Univeristy in 2016. His research interests lie in computer vision and deep learning.
Xiaojiang Peng received his Ph.D. degree from School of Information Science and Technology, Southwest Jiaotong University, China, in 2014. He currently is an associate professor at the Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences, China. He was a postdoctoral researcher at Idiap Institute, Switzerland, from 2016 to 2017, and was a postdoctoral researcher in LEAR Team, INRIA, France, working with Prof. Cordelia Schmid from 2015 to 2016. He serves as a reviewer for IJCV, TMM, TIP, CVPR, ICCV, AAAI, IJCAI, FG, Image and Vision Computing, IEEE Signal Processing Letter, Neurocomputing, etc. His research focus is in the areas of action recognition and detection, face recognition, facial emotion analysis, and deep learning.
Yu Qiao received his Ph.D. degree from the University of Electro-Communications, Japan, in 2006. He was a JSPS Fellow and a Project Assistant Professor with the University of Tokyo, from 2007 to 2010. He is currently a professor with the Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences. He has authored over 140 papers in journals and conferences including PAMI, IJCV, TIP, ICCV, CVPR, ECCV, and AAAI. His research interests include computer vision, deep learning, and intelligent robots. He was a recipient of the Lu Jiaxi Young Researcher Award from the Chinese Academy of Sciences in 2012. He was the first runner-up at the ImageNet Large Scale Visual Recognition Challenge 2015 in scene recognition and the recipient at the ActivityNet Large Scale Activity Recognition Challenge 2016 in video classification.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made.
The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.
To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
Other papers from this open access journal are available free of charge from http://www.springer.com/journal/41095. To submit a manuscript, please go to https://www.editorialmanager.com/cvmj.
About this article
Cite this article
Zeng, X., Wu, Z., Peng, X. et al. Joint 3D facial shape reconstruction and texture completion from a single image. Comp. Visual Media 8, 239–256 (2022). https://doi.org/10.1007/s41095-021-0238-4
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s41095-021-0238-4