Skip to main content
SpringerLink
Log in

Generating High-Resolution 3D Faces Using VQ-VAE-2 with PixelSNAIL Networks

  • Conference paper
  • First Online:
Image Analysis and Processing. ICIAP 2022 Workshops (ICIAP 2022)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13374))

Included in the following conference series:

Abstract

The realistic generation of synthetic 3D faces is an open challenge due to the complexity of the geometry and the lack of large and diverse publicly available datasets. Generative models based on convolutional neural networks (CNNs) have recently demonstrated great ability to produce novel synthetic high-resolution images indistinguishable from the original pictures by an expert human observer. However, applying them to non-grid-like data like 3D meshes presents many challenges. In our work, we overcome the challenges by first reducing the face mesh to a 2D regular image representation and then exploiting one prominent state-of-the-art generative approach. The approach uses a Vector Quantized Variational Autoencoder VQ-VAE-2 to learn a latent discrete representation of the 2D images. Then, the 3D synthesis is achieved by fitting the latent space and sampling it with an autoregressive model, PixelSNAIL. The quantitative and qualitative evaluation demonstrate that synthetic faces generated with our method are statistically closer to the real faces when compared to a classical synthesis approach based on Principal Component Analysis (PCA).

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Liu, S.-L., Liu, Y., Dong, L.-F., Tong, X.: RAS: a data-driven rigidity-aware skinning model for 3D facial animation. In: Computer Graphics Forum, pp. 581–594 (2020)

    Google Scholar 

  2. Carrigan, E., Zell, E., Guiard, C., McDonnell, R.: Expression packing: as-few-as-possible training expressions for blendshape transfer. In: Computer Graphics Forum, pp. 219–233 (2020)

    Google Scholar 

  3. Li, T., Bolkart, T., Black, M.J., Li, H., Romero, J.: Learning a model of facial shape and expression from 4D scans. ACM Trans. Graph. 36, 191–194 (2017)

    Article  Google Scholar 

  4. Valev, H., Gallucci, A., Leufkens, T., Westerink, J., Sas, C.: Applying delaunay triangulation augmentation for deep learning facial expression generation and recognition. In: Del Bimbo, A., et al. (eds.) ICPR 2021. LNCS, vol. 12663, pp. 730–740. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-68796-0_53

    Chapter  Google Scholar 

  5. Taigman, Y., Yang, M., Ranzato, M., Wolf, L.: DeepFace: closing the gap to human-level performance in face verification. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1701–1708 (2014)

    Google Scholar 

  6. Varol, G., et al.: Learning from synthetic humans. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 109–117 (2017)

    Google Scholar 

  7. Blanz, V., Vetter, T.: A morphable model for the synthesis of 3D faces. In: Proceedings 26th Annual Conference on Computer Graphics and Interactive Techniques, pp. 187–194 (1999)

    Google Scholar 

  8. Bronstein, M.M., Bruna, J., LeCun, Y., Szlam, A., Vandergheynst, P.: Geometric deep learning: going beyond euclidean data. IEEE Signal Process. Mag. 34, 18–42 (2017)

    Article  Google Scholar 

  9. Ranjan, A., Bolkart, T., Sanyal, S., Black, M.J.: Generating 3D faces using convolutional mesh autoencoders. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11207, pp. 725–741. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01219-9_43

    Chapter  Google Scholar 

  10. De Haan, P., Weiler, M., Cohen, T., Welling, M.: Gauge equivariant mesh CNNs: anisotropic convolutions on geometric graphs. arXiv Prepr. arXiv2003.05425 (2020)

    Google Scholar 

  11. Zhang, S., Tong, H., Xu, J., Maciejewski, R.: Graph convolutional networks: a comprehensive review. Comput. Soc. Netw. 6(1), 1–23 (2019). https://doi.org/10.1186/s40649-019-0069-y

    Article  Google Scholar 

  12. Razavi, A., van den Oord, A., Vinyals, O.: Generating diverse high-fidelity images with VQ-VAE-2. In: Advances in Neural Information Processing Systems, pp. 14837–14847 (2019)

    Google Scholar 

  13. Van Oord, A., Kalchbrenner, N., Kavukcuoglu, K.: Pixel recurrent neural networks. In: International Conference on Machine Learning, pp. 1747–1756 (2016)

    Google Scholar 

  14. den Oord, A., Kalchbrenner, N., Espeholt, L., Vinyals, O., Graves, A., et al.: Conditional image generation with pixelcnn decoders. In: Advances in Neural Information Processing Systems, pp. 4790–4798 (2016)

    Google Scholar 

  15. Vaswani, A., e al.: Attention is all you need. In: Advances in Neural Information Processing Systems, pp. 5998–6008 (2017)

    Google Scholar 

  16. Chen, X., Mishra, N., Rohaninejad, M., Abbeel, P.: PixelSNAIL: an improved autoregressive generative model. In: 35th International Conference on Machine Learning ICML 2018, vol. 2, pp. 1364–1372 (2018)

    Google Scholar 

  17. Davies, R., Twining, C., Taylor, C.: Statistical Models of Shape: Optimisation and Evaluation. Springer, London (2008). https://doi.org/10.1007/978-1-84800-138-1

  18. Abrevaya, V.F., Boukhayma, A., Wuhrer, S., Boyer, E.: A decoupled 3D facial shape model by adversarial training. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9419–9428 (2019)

    Google Scholar 

  19. Thies, J., Zollhofer, M., Stamminger, M., Theobalt, C., Nießner, M.: Face2face: real-time face capture and reenactment of RGB videos. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2387–2395 (2016)

    Google Scholar 

  20. Vlasic, D., Brand, M., Pfister, H., Popovic, J.: Face transfer with multilinear models. In: ACM SIGGRAPH 2006 Courses, pp. 24–es (2006)

    Google Scholar 

  21. Booth, J., Roussos, A., Zafeiriou, S., Ponniah, A., Dunaway, D.: A 3D morphable model learnt from 10,000 faces. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5543–5552 (2016)

    Google Scholar 

  22. Tuan Tran, A., Hassner, T., Masi, I., Medioni, G.: Regressing robust and discriminative 3D morphable models with a very deep neural network. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5163–5172 (2017)

    Google Scholar 

  23. Gu, X., Gortler, S.J., Hoppe, H.: Geometry images. In: Proceedings of the 29th Annual Conference on Computer Graphics and Interactive Techniques, pp. 355–361 (2002)

    Google Scholar 

  24. Booth, J., Zafeiriou, S.: Optimal UV spaces for facial morphable model construction. In: 2014 IEEE International Conference on Image Processing (ICIP), pp. 4672–4676 (2014)

    Google Scholar 

  25. Goodfellow, I., et al.: Generative adversarial nets. In: Advances in Neural Information Processing Systems, pp. 2672–2680 (2014)

    Google Scholar 

  26. Arjovsky, M., Chintala, S., Bottou, L.: Wasserstein generative adversarial networks. In: International Conference on Machine Learning, pp. 214–223 (2017)

    Google Scholar 

  27. Slossberg, R., Shamai, G., Kimmel, R.: High quality facial surface and texture synthesis via generative adversarial networks. In: Leal-Taixé, L., Roth, S. (eds.) ECCV 2018. LNCS, vol. 11131, pp. 498–513. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-11015-4_36

    Chapter  Google Scholar 

  28. Shamai, G., Slossberg, R., Kimmel, R.: Synthesizing facial photometries and corresponding geometries using generative adversarial networks. ACM Trans. Multimedia Comput. Commun. Appl. 15, 1–24 (2019)

    Article  Google Scholar 

  29. Moschoglou, S., Ploumpis, S., Nicolaou, M.A., Papaioannou, A., Zafeiriou, S.: 3DFaceGAN: adversarial nets for 3D face representation, generation, and translation. Int. J. Comput. Vis. 128, 2534–2551 (2020)

    Article  Google Scholar 

  30. Kingma, D.P., Welling, M.: Auto-encoding variational Bayes. In: 2nd International Conference on Learning Representations ICLR 2014 - Conference Track Proceedings, pp. 1–14 (2014)

    Google Scholar 

  31. Bagautdinov, T., Wu, C., Saragih, J., Fua, P., Sheikh, Y.: Modeling facial geometry using compositional VAEs. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3877–3886 (2018)

    Google Scholar 

  32. Abrevaya, V.F., Wuhrer, S., Boyer, E.: Multilinear autoencoder for 3D face model learning. In: 2018 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 1–9 (2018)

    Google Scholar 

  33. Li, K., Liu, J., Lai, Y.-K., Yang, J.: Generating 3D faces using multi-column graph convolutional networks. In: Computer Graphics Forum, pp. 215–224 (2019)

    Google Scholar 

  34. Tam, G.K.L.L., et al.: Registration of 3D point clouds and meshes: a survey from rigid to Nonrigid. IEEE Trans. Vis. Comput. Graph. 19, 1199–1217 (2013)

    Google Scholar 

  35. van Kaick, O., Zhang, H., Hamarneh, G., Cohen-Or, D.: A survey on shape correspondence. In: Eurographics Symposium on Geometry Processing (2011)

    Google Scholar 

  36. Gallucci, A., Znamenskiy, D., Petkovic, M.: Prediction of 3D body parts from face shape and anthropometric measurements. J. Image Graph. 8, 67–77 (2020)

    Google Scholar 

  37. van den Oord, A., Vinyals, O., et al.: Neural discrete representation learning. In: Advances in Neural Information Processing Systems, pp. 6306–6315 (2017)

    Google Scholar 

  38. Kingma, D.P., Welling, M.: An introduction to variational autoencoders. arXiv Prepr. arXiv1906.02691 (2019)

    Google Scholar 

  39. Russakovsky, O., et al.: ImageNet large scale visual recognition challenge. Int. J. Comput. Vision 115(3), 211–252 (2015). https://doi.org/10.1007/s11263-015-0816-y

    Article  MathSciNet  Google Scholar 

  40. Kabsch, W.: A solution for the best rotation to relate two sets of vectors. Acta Crystallogr. Sect. A Cryst. Phys. Diffr. Theor. Gen. Crystallogr. 32, 922–923 (1976)

    Google Scholar 

  41. Ball, R., Molenbroek, J.F.M.: Measuring Chinese heads and faces. In: Proceedings of the 9th International Congress of Physiological Anthropology, Human Diversity Design for Life, pp. 150–155 (2008)

    Google Scholar 

  42. Robinette, K.M., Daanen, H., Paquet, E.: The CAESAR project: a 3-D surface anthropometry survey. In: Second International Conference on 3-D Digital Imaging and Modeling (Cat. No.PR00062), pp. 380–386 (1999)

    Google Scholar 

  43. Robinette, K.M., Daanen, H.: Lessons learned from CAESAR: a 3-D anthropometric survey, 5 (2003)

    Google Scholar 

  44. Gallucci, A., Pezzotti, N., Znamenskiy, D., Petkovic, M.: A latent space exploration for microscopic skin lesion augmentations with VQ-VAE-2 and PixelSNAIL. In: SPIE Medical Imaging Proceedings (2021)

    Google Scholar 

  45. Paszke, A., et al.: Automatic differentiation in PyTorch (2017)

    Google Scholar 

Download references

Acknowledgments

We thank Philips Research for providing access to the datasets of facial scans and software resources to manage the high-resolution parametric models.

Author information

Authors and Affiliations

  1. Eindhoven University of Technology, Eindhoven, The Netherlands

    Alessio Gallucci, Nicola Pezzotti & Milan Petkovic

  2. Philips Research, Eindhoven, The Netherlands

    Alessio Gallucci, Dmitry Znamenskiy, Nicola Pezzotti & Milan Petkovic

Authors
  1. Alessio Gallucci
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Dmitry Znamenskiy
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Nicola Pezzotti
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Milan Petkovic
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Alessio Gallucci .

Editor information

Editors and Affiliations

  1. National Research Council, Lecce, Italy

    Pier Luigi Mazzeo

  2. Università Politecnica delle Marche, Ancona, Italy

    Emanuele Frontoni

  3. Boston University, Boston, MA, USA

    Stan Sclaroff

  4. National Research Council, Lecce, Italy

    Cosimo Distante

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Gallucci, A., Znamenskiy, D., Pezzotti, N., Petkovic, M. (2022). Generating High-Resolution 3D Faces Using VQ-VAE-2 with PixelSNAIL Networks. In: Mazzeo, P.L., Frontoni, E., Sclaroff, S., Distante, C. (eds) Image Analysis and Processing. ICIAP 2022 Workshops. ICIAP 2022. Lecture Notes in Computer Science, vol 13374. Springer, Cham. https://doi.org/10.1007/978-3-031-13324-4_20

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-13324-4_20

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-13323-7

  • Online ISBN: 978-3-031-13324-4

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation