Skip to main content
SpringerLink
Log in

REALY: Rethinking the Evaluation of 3D Face Reconstruction

  • Conference paper
  • First Online:
Computer Vision – ECCV 2022 (ECCV 2022)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13668))

Included in the following conference series:

Abstract

The evaluation of 3D face reconstruction results typically relies on a rigid shape alignment between the estimated 3D model and the ground-truth scan. We observe that aligning two shapes with different reference points can largely affect the evaluation results. This poses difficulties for precisely diagnosing and improving a 3D face reconstruction method. In this paper, we propose a novel evaluation approach with a new benchmark REALY, consists of 100 globally aligned face scans with accurate facial keypoints, high-quality region masks, and topology-consistent meshes. Our approach performs region-wise shape alignment and leads to more accurate, bidirectional correspondences during computing the shape errors. The fine-grained, region-wise evaluation results provide us detailed understandings about the performance of state-of-the-art 3D face reconstruction methods. For example, our experiments on single-image based reconstruction methods reveal that DECA performs the best on nose regions, while GANFit performs better on cheek regions. Besides, a new and high-quality 3DMM basis, HIFI3D\(^{\pmb {+}\pmb {+}}\), is further derived using the same procedure as we construct REALY to align and retopologize several 3D face datasets. We will release REALY, HIFI3D\(^{\pmb {+}\pmb {+}}\), and our new evaluation pipeline at https://realy3dface.com.

Z. Chai and H. Zhang—Equal Contributions.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 89.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 119.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Amberg, B., Romdhani, S., Vetter, T.: Optimal step nonrigid ICP algorithms for surface registration. In: CVPR (2007)

    Google Scholar 

  2. Bagdanov, A.D., Bimbo, A.D., Masi, I.: The Florence 2d/3d hybrid face dataset. In: J-HGBU@MM (2011)

    Google Scholar 

  3. Bai, Z., Cui, Z., Liu, X., Tan, P.: Riggable 3d face reconstruction via in-network optimization. In: CVPR (2021)

    Google Scholar 

  4. Bao, L., et al.: High-fidelity 3d digital human head creation from RGB-D selfies. TOG (2021)

    Google Scholar 

  5. Besl, P.J., McKay, N.D.: A method for registration of 3d shapes. TPAMI (1992)

    Google Scholar 

  6. Blanz, V., Vetter, T.: A morphable model for the synthesis of 3d faces. In: SIGGRAPH (1999)

    Google Scholar 

  7. Blanz, V., Vetter, T.: Face recognition based on fitting a 3d morphable model. TPAMI (2003)

    Google Scholar 

  8. Booth, J., Roussos, A., Zafeiriou, S., Ponniah, A., Dunaway, D.: A 3d morphable model learnt from 10,000 faces. In: CVPR (2016)

    Google Scholar 

  9. Brunton, A., Salazar, A., Bolkart, T., Wuhrer, S.: Review of statistical shape spaces for 3d data with comparative analysis for human faces. In: CVIU (2014)

    Google Scholar 

  10. Cao, C., Weng, Y., Lin, S., Zhou, K.: 3d shape regression for real-time facial animation. TOG 32, 1 (2013)

    Article  MATH  Google Scholar 

  11. Cao, C., Weng, Y., Zhou, S., Tong, Y., Zhou, K.: Facewarehouse: a 3d facial expression database for visual computing. TVCG 20, 413–425 (2014)

    Google Scholar 

  12. Cao, C., Wu, H., Weng, Y., Shao, T., Zhou, K.: Real-time facial animation with image-based dynamic avatars. TOG 35, 1–13 (2016)

    Google Scholar 

  13. Cao, K., Rong, Y., Li, C., Tang, X., Loy, C.C.: Pose-robust face recognition via deep residual equivariant mapping. In: CVPR (2018)

    Google Scholar 

  14. Chang, F., Tran, A.T., Hassner, T., Masi, I., Nevatia, R., Medioni, G.G.: Faceposenet: making a case for landmark-free face alignment. In: ICCV Workshops (2017)

    Google Scholar 

  15. Chang, F., Tran, A.T., Hassner, T., Masi, I., Nevatia, R., Medioni, G.G.: ExpNet: landmark-free, deep, 3d facial expressions. In: FG (2018)

    Google Scholar 

  16. Chaudhuri, B., Vesdapunt, N., Shapiro, L., Wang, B.: Personalized face modeling for improved face reconstruction and motion retargeting. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12350, pp. 142–160. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58558-7_9

    Chapter  Google Scholar 

  17. Chen, Y., Wu, F., Wang, Z., Song, Y., Ling, Y., Bao, L.: Self-supervised learning of detailed 3d face reconstruction. TIP 29, 8696–8705 (2020)

    MATH  Google Scholar 

  18. Dai, H., Pears, N.E., Smith, W.A.P., Duncan, C.: Statistical modeling of craniofacial shape and texture. IJCV 128, 547–571 (2020)

    Article  Google Scholar 

  19. Deng, Y., Yang, J., Xu, S., Chen, D., Jia, Y., Tong, X.: Accurate 3d face reconstruction with weakly-supervised learning: From single image to image set. In: CVPR Workshops (2019)

    Google Scholar 

  20. Dib, A., Thebault, C., Ahn, J., Gosselin, P., Theobalt, C., Chevallier, L.: Towards high fidelity monocular face reconstruction with rich reflectance using self-supervised learning and ray tracing. In: ICCV (2021)

    Google Scholar 

  21. Egger, B., et al.: 3d morphable face models - past, present, and future. TOG. 39, 1–38 (2020)

    Article  Google Scholar 

  22. Feng, Y., Feng, H., Black, M.J., Bolkart, T.: Learning an animatable detailed 3d face model from in-the-wild images. In: SIGGRAPH (2021)

    Google Scholar 

  23. Feng, Y., Wu, F., Shao, X., Wang, Y., Zhou, X.: Joint 3D face reconstruction and dense alignment with position map regression network. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) Computer Vision – ECCV 2018. LNCS, vol. 11218, pp. 557–574. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01264-9_33

    Chapter  Google Scholar 

  24. Feng, Z., et al.: Evaluation of dense 3d reconstruction from 2d face images in the wild. In: FG (2018)

    Google Scholar 

  25. Gao, Z., Zhang, J., Guo, Y., Ma, C., Zhai, G., Yang, X.: Semi-supervised 3d face representation learning from unconstrained photo collections. In: CVPR Workshops (2020)

    Google Scholar 

  26. Gecer, B., Ploumpis, S., Kotsia, I., Zafeiriou, S.: GANFIT: generative adversarial network fitting for high fidelity 3d face reconstruction. In: CVPR (2019)

    Google Scholar 

  27. Gecer, B., Ploumpis, S., Kotsia, I., Zafeiriou, S.: Fast-GANFIT: generative adversarial network for high fidelity 3d face reconstruction. TPAMI (2021)

    Google Scholar 

  28. Genova, K., Cole, F., Maschinot, A., Sarna, A., Vlasic, D., Freeman, W.T.: Unsupervised training for 3d morphable model regression. In: CVPR (2018)

    Google Scholar 

  29. Guo, J., Zhu, X., Yang, Y., Yang, F., Lei, Z., Li, S.Z.: Towards Fast, Accurate and Stable 3D Dense Face Alignment. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12364, pp. 152–168. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58529-7_10

    Chapter  Google Scholar 

  30. Hu, L., et al.: Avatar digitization from a single image for real-time rendering. TOG 36, 1–4 (2017)

    Article  Google Scholar 

  31. Jiang, D., et al.: Reconstructing recognizable 3d face shapes based on 3d morphable models. CoRR, abs/2104.03515 (2021)

    Google Scholar 

  32. Karras, T., Laine, S., Aila, T.: A style-based generator architecture for generative adversarial networks. In: CVPR (2019)

    Google Scholar 

  33. Lattas, A., et al.: Avatarme: Realistically renderable 3d facial reconstruction “in-the-wild”. In: CVPR (2020)

    Google Scholar 

  34. Lee, G., Lee, S.: Uncertainty-aware mesh decoder for high fidelity 3d face reconstruction. In: CVPR (2020)

    Google Scholar 

  35. Li, T., Bolkart, T., Black, M.J., Li, H., Romero, J.: Learning a model of facial shape and expression from 4d scans. TOG (2017)

    Google Scholar 

  36. Lin, J., Yuan, Y., Shao, T., Zhou, K.: Towards high-fidelity 3d face reconstruction from in-the-wild images using graph convolutional networks. In: CVPR (2020)

    Google Scholar 

  37. Lin, J., Yuan, Y., Zou, Z.: Meingame: create a game character face from a single portrait. In: AAAI (2021)

    Google Scholar 

  38. Liu, F., Zhu, R., Zeng, D., Zhao, Q., Liu, X.: Disentangling features in 3d face shapes for joint face reconstruction and recognition. In: CVPR (2018)

    Google Scholar 

  39. Liu, P., Han, X., Lyu, M.R., King, I., Xu, J.: Learning 3d face reconstruction with a pose guidance network. In: ACCV (2020)

    Google Scholar 

  40. Liu, Z., Luo, P., Wang, X., Tang, X.: Deep learning face attributes in the wild. In: ICCV (2015)

    Google Scholar 

  41. Luo, H., et al.: Normalized avatar synthesis using styleGAN and perceptual refinement. In: CVPR (2021)

    Google Scholar 

  42. Lyu, J., Li, X., Zhu, X., Cheng, C.: Pixel-face: A large-scale, high-resolution benchmark for 3d face reconstruction. arXiv preprint arXiv:2008.12444 (2020)

  43. Ma, S., et al.: Pixel codec avatars. In: CVPR (2021)

    Google Scholar 

  44. Pan, X., Dai, B., Liu, Z., Chen, C.L., Luo, P.: Do 2d GANs know 3d shape? Unsupervised 3d shape reconstruction from 2d image GANs. In: ICLR (2021)

    Google Scholar 

  45. Paysan, P., Knothe, R., Amberg, B., Romdhani, S., Vetter, T.: A 3d face model for pose and illumination invariant face recognition. In: AVSS (2009)

    Google Scholar 

  46. Phillips, P.J., et al.: Overview of the face recognition grand challenge. In: CVPR (2005)

    Google Scholar 

  47. Piao, J., Sun, K., Wang, Q., Lin, K., Li, H.: Inverting generative adversarial renderer for face reconstruction. In: CVPR (2021)

    Google Scholar 

  48. Ploumpis, S., Wang, H., Pears, N.E., Smith, W.A.P., Zafeiriou, S.: Combining 3d morphable models: a large scale face-and-head model. In: CVPR (2019)

    Google Scholar 

  49. R, M.B., Tewari, A., Seidel, H., Elgharib, M., Theobalt, C.: Learning complete 3d morphable face models from images and videos. In: CVPR (2021)

    Google Scholar 

  50. Ramon, E., et al.: H3D-Net: few-shot high-fidelity 3d head reconstruction. In: ICCV (2021)

    Google Scholar 

  51. Richardson, E., Sela, M., Kimmel, R.: 3d face reconstruction by learning from synthetic data. In: 3DV (2016)

    Google Scholar 

  52. Sanyal, S., Bolkart, T., Feng, H., Black, M.J.: Learning to regress 3d face shape and expression from an image without 3d supervision. In: CVPR (2019)

    Google Scholar 

  53. Sela, M., Richardson, E., Kimmel, R.: Unrestricted facial geometry reconstruction using image-to-image translation. In: ICCV (2017)

    Google Scholar 

  54. Shang, J., et al.: Self-supervised monocular 3d face reconstruction by occlusion-aware multi-view geometry consistency. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12360, pp. 53–70. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58555-6_4

    Chapter  Google Scholar 

  55. Smith, W.A.P., Seck, A., Dee, H., Tiddeman, B., Tenenbaum, J.B., Egger, B.: A morphable face albedo model. In: CVPR (2020)

    Google Scholar 

  56. Stratou, G., Ghosh, A., Debevec, P.E., Morency, L.P.: Effect of illumination on automatic expression recognition: a novel 3d relightable facial database. In: FG (2011)

    Google Scholar 

  57. Tewari, A., et al.: MoFA: Model-based deep convolutional face autoencoder for unsupervised monocular reconstruction. In: ICCV (2017)

    Google Scholar 

  58. Thies, J., Zollhöfer, M., Stamminger, M., Theobalt, C., Nießner, M.: Face2face: Real-time face capture and reenactment of RGB videos. In: CVPR (2016)

    Google Scholar 

  59. Tran, A.T., Hassner, T., Masi, I., Medioni, G.G.: Regressing robust and discriminative 3d morphable models with a very deep neural network. In: CVPR (2017)

    Google Scholar 

  60. Tran, L., Liu, F., Liu, X.: Towards high-fidelity nonlinear 3d face morphable model. In: CVPR (2019)

    Google Scholar 

  61. Tran, L., Liu, X.: Nonlinear 3d face morphable model. In: CVPR (2018)

    Google Scholar 

  62. Tran, L., Liu, X.: On learning 3d face morphable model from in-the-wild images. TPAMI 43, 157–171 (2021)

    Google Scholar 

  63. Wen, Y., Liu, W., Raj, B., Singh, R.: Self-supervised 3d face reconstruction via conditional estimation. In: ICCV (2021)

    Google Scholar 

  64. Wright, J., Yang, A.Y., Ganesh, A., Sastry, S.S., Ma, Y.: Robust face recognition via sparse representation. TPAMI (2009)

    Google Scholar 

  65. Wu, F., et al.: MVF-Net: multi-view 3d face morphable model regression. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 959–968 (2019)

    Google Scholar 

  66. Wu, S., Rupprecht, C., Vedaldi, A.: Unsupervised learning of probably symmetric deformable 3d objects from images in the wild. In: CVPR (2020)

    Google Scholar 

  67. Yamaguchi, S., et al.: High-fidelity facial reflectance and geometry inference from an unconstrained image. TOG. 37, 1–4 (2018)

    Article  Google Scholar 

  68. Yang, H., et al.: Facescape: a large-scale high quality 3d face dataset and detailed riggable 3d face prediction. In: CVPR (2020)

    Google Scholar 

  69. Yenamandra, T., et al.: i3DMM: deep implicit 3d morphable model of human heads. In: CVPR (2021)

    Google Scholar 

  70. Yin, L., Wei, X., Sun, Y., Wang, J., Rosato, M.J.: A 3d facial expression database for facial behavior research. In: FG (2006)

    Google Scholar 

  71. Zeng, X., Peng, X., Qiao, Y.: DF2Net: a dense-fine-finer network for detailed 3d face reconstruction. In: ICCV (2019)

    Google Scholar 

  72. Zhang, Z., et al.: Learning to aggregate and personalize 3d face from in-the-wild photo collection. In: CVPR (2021)

    Google Scholar 

  73. Zhu, X., Liu, X., Lei, Z., Li, S.Z.: Face alignment in full pose range: a 3d total solution. TPAMI. 41, 18–92 (2019)

    Article  Google Scholar 

  74. Zhu, X., Ramanan, D.: Face detection, pose estimation, and landmark localization in the wild. In: CVPR (2012)

    Google Scholar 

  75. Zhu, X., et al.: Beyond 3DMM space: towards fine-grained 3d face reconstruction. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12353, pp. 343–358. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58598-3_21

    Chapter  Google Scholar 

  76. Zollhöfer, M., et al.: State of the art on monocular 3d face reconstruction, tracking, and applications. In: CGF (2018)

    Google Scholar 

Download references

Acknowledgment

This work was supported by SZSTC Grant No. JCYJ20190 809172201639 and WDZC20200820200655001, Shenzhen Key Laboratory ZDSY S20210623092001004.

Author information

Authors and Affiliations

  1. Shenzhen International Graduate School, Tsinghua University, Beijing, China

    Zenghao Chai, Zhengzhuo Xu & Chun Yuan

  2. Tencent AI Lab, Bellevue, China

    Haoxian Zhang, Jing Ren, Di Kang, Xuefei Zhe & Linchao Bao

  3. Peng Cheng National Laboratory, Shenzhen, China

    Chun Yuan

Authors
  1. Zenghao Chai
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Haoxian Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jing Ren
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Di Kang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Zhengzhuo Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Xuefei Zhe
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Chun Yuan
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Linchao Bao
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Chun Yuan or Linchao Bao .

Editor information

Editors and Affiliations

  1. Tel Aviv University, Tel Aviv, Israel

    Shai Avidan

  2. University College London, London, UK

    Gabriel Brostow

  3. Google AI, Accra, Ghana

    Moustapha Cissé

  4. University of Catania, Catania, Italy

    Giovanni Maria Farinella

  5. Facebook (United States), Menlo Park, CA, USA

    Tal Hassner

1 Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (pdf 18358 KB)

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Chai, Z. et al. (2022). REALY: Rethinking the Evaluation of 3D Face Reconstruction. In: Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T. (eds) Computer Vision – ECCV 2022. ECCV 2022. Lecture Notes in Computer Science, vol 13668. Springer, Cham. https://doi.org/10.1007/978-3-031-20074-8_5

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-20074-8_5

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-20073-1

  • Online ISBN: 978-3-031-20074-8

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation