Abstract
Cartoon face is a prevalent kind of stylized face, which is widely used in movies, TVs and advertisements. Although plenty of methods have been proposed to generate 2D cartoon faces, it is still challenging to learn personalized 3D cartoon faces directly from 2D real photos. To solve this problem, we contribute the first 3D cartoon face hybrid dataset with both large amounts of low-quality and a small number of high-quality face triplets. Each triplet contains a 2D real face, as well as its corresponding 2D and 3D cartoon faces. To leverage the hybrid dataset, we propose Recon2AGen which first pretrains our network with low-quality triplets in a reconstruction-then-generation manner and then finetunes it with high-quality triplets in an adversarial manner. In this way, we solve the 2D-to-3D ambiguity and the real-to-cartoon transformation by disentangling the task into three progressively learned sub-tasks. And the hybrid dataset is fully explored to achieve generalizable and high accuracy results. Extensive experiments show that our generated 3D cartoon faces are of high quality and can be easily edited and animated, enabling extensive practical applications. Code and dataset will be available at https://github.com/mingsjtu/3DCartoonGenerator.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
References
Alaluf, Y., Patashnik, O., Cohen-Or, D.: ReStyle: a residual-based StyleGAN encoder via iterative refinement. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 6711–6720 (2021)
Blanz, V., Vetter, T.: A morphable model for the synthesis of 3D faces. In: Proceedings of the 26th Annual Conference on Computer Graphics and Interactive Techniques, pp. 187–194 (1999)
Cai, H., Guo, Y., Peng, Z., Zhang, J.: Landmark detection and 3D face reconstruction for caricature using a nonlinear parametric model. Graph. Models 115, 101103 (2021)
Cao, C., Weng, Y., Zhou, S., Tong, Y., Zhou, K.: FaceWarehouse: a 3D facial expression database for visual computing. IEEE Trans. Visual. Comput. Graph. 20(3), 413–425 (2013)
Cao, K., Liao, J., Yuan, L.: CariGANs: unpaired photo-to-caricature translation. arXiv preprint arXiv:1811.00222 (2018)
Deng, J., Cheng, S., Xue, N., Zhou, Y., Zafeiriou, S.: UV-GAN: adversarial facial UV map completion for pose-invariant face recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7093–7102 (2018)
Duda, R.O., Hart, P.E., et al.: Pattern Classification and Scene Analysis, vol. 3. Wiley, New York (1973)
Gecer, B., Ploumpis, S., Kotsia, I., Zafeiriou, S.: GANFIT: generative adversarial network fitting for high fidelity 3D face reconstruction. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019
Gong, J., Hold-Geoffroy, Y., Lu, J.: AutoToon: automatic geometric warping for face cartoon generation. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 360–369 (2020)
Han, X., Gao, C., Yu, Y.: DeepSketch2Face: a deep learning based sketching system for 3D face and caricature modeling. ACM Trans. Graph. (TOG) 36(4), 1–12 (2017)
Han, X., et al.: CaricatureShop: personalized and photorealistic caricature sketching. IEEE Trans. Visual. Comput. Graph. 26(7), 2349–2361 (2018)
Jang, W., Ju, G., Jung, Y., Yang, J., Tong, X., Lee, S.: StyleCariGAN: caricature generation via StyleGAN feature map modulation. ACM Trans. Graph. (TOG) 40(4), 1–16 (2021)
Karkkainen, K., Joo, J.: FairFace: face attribute dataset for balanced race, gender, and age for bias measurement and mitigation. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 1548–1558 (2021)
Karras, T., Laine, S., Aila, T.: A style-based generator architecture for generative adversarial networks. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4401–4410 (2019)
Lewiner, T., Vieira, T., MartÃnez, D., Peixoto, A., Mello, V., Velho, L.: Interactive 3D caricature from harmonic exaggeration. Comput. Graph. 35(3), 586–595 (2011)
Li, T., Bolkart, T., Black, M.J., Li, H., Romero, J.: Learning a model of facial shape and expression from 4D scans. ACM Trans. Graph. 36(6), 194–1 (2017)
Liu, J., et al.: Semi-supervised learning in reconstructed manifold space for 3D caricature generation. In: Computer Graphics Forum, vol. 28, pp. 2104–2116. Wiley Online Library (2009)
Pinkney, J.N., Adler, D.: Resolution dependent GAN interpolation for controllable image synthesis between domains. arXiv preprint arXiv:2010.05334 (2020)
Qiu, Y., et al.: 3DCaricShop: a dataset and a baseline method for single-view 3D caricature face reconstruction. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10236–10245 (2021)
Richardson, E., et al.: Encoding in style: a StyleGAN encoder for image-to-image translation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2287–2296 (2021)
Shi, Y., Deb, D., Jain, A.K.: WarpGAN: automatic caricature generation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10762–10771 (2019)
Song, G., et al.: AgileGAN: stylizing portraits by inversion-consistent transfer learning. ACM Trans. Graph. (TOG) 40(4), 1–13 (2021)
Tov, O., Alaluf, Y., Nitzan, Y., Patashnik, O., Cohen-Or, D.: Designing an encoder for StyleGAN image manipulation. arXiv preprint arXiv:2102.02766 (2021)
Vieira, R.C.C., Vidal, C.A., Cavalcante-Neto, J.B.: Three-dimensional face caricaturing by anthropometric distortions. In: 2013 XXVI Conference on Graphics, Patterns and Images, pp. 163–170. IEEE (2013)
Wu, Q., Zhang, J., Lai, Y.K., Zheng, J., Cai, J.: Alive caricature from 2D to 3D. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7336–7345 (2018)
Yang, H., et al.: FaceScape: a large-scale high quality 3D face dataset and detailed riggable 3D face prediction. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 601–610 (2020)
Zhu, X., Lei, Z., Yan, J., Yi, D., Li, S.Z.: High-fidelity pose and expression normalization for face recognition in the wild. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 787–796 (2015)
Acknowledgements
This work was supported by Beijing Natural Science Foundation (JQ19015), the NSFC (No. 62021002, 61727808), the National Key R &D Program of China (2018YFA0704000), and the Key Research and Development Project of Tibet Autonomous Region (XZ202101ZY0019G). This work was also supported by THUIBCS, Tsinghua University, and BLBCI, Beijing Municipal Education Commission.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Guo, M. et al. (2022). 3D Face Cartoonizer: Generating Personalized 3D Cartoon Faces from 2D Real Photos with a Hybrid Dataset. In: Fang, L., Povey, D., Zhai, G., Mei, T., Wang, R. (eds) Artificial Intelligence. CICAI 2022. Lecture Notes in Computer Science(), vol 13604. Springer, Cham. https://doi.org/10.1007/978-3-031-20497-5_29
Download citation
DOI: https://doi.org/10.1007/978-3-031-20497-5_29
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-20496-8
Online ISBN: 978-3-031-20497-5
eBook Packages: Computer ScienceComputer Science (R0)