Skip to main content

Practical and Scalable Desktop-Based High-Quality Facial Capture

  • Conference paper
  • First Online:
Computer Vision – ECCV 2022 (ECCV 2022)

Abstract

We present a novel desktop-based system for high-quality facial capture including geometry and facial appearance. The proposed acquisition system is highly practical and scalable, consisting purely of commodity components. The setup consists of a set of displays for controlled illumination for reflectance capture, in conjunction with multiview acquisition of facial geometry. We additionally present a novel set of modulated binary illumination patterns for efficient acquisition of reflectance and photometric normals using our setup, with diffuse-specular separation. We demonstrate high-quality results with two different variants of the capture setup – one entirely consisting of portable mobile devices targeting static facial capture, and the other consisting of desktop LCD displays targeting both static and dynamic facial capture.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Subscribe and save

Springer+ Basic
EUR 32.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or Ebook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 89.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 119.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Beeler, T., Bickel, B., Beardsley, P., Sumner, B., Gross, M.: High-quality single-shot capture of facial geometry. ACM Trans. Graph. (TOG) 29(3), 40:1–40:9 (2010)

    Google Scholar 

  2. Beeler, T., et al.: High-quality passive facial performance capture using anchor frames. ACM Trans. Graph. (ACM) 30, 75:1–75:10 (2011)

    Google Scholar 

  3. Bradley, D., Heidrich, W., Popa, T., Sheffer, A.: High resolution passive facial performance capture. ACM Trans. Graph. (TOG) 29(4), 41 (2010)

    Article  Google Scholar 

  4. Debevec, P., Hawkins, T., Tchou, C., Duiker, H.P., Sarokin, W., Sagar, M.: Acquiring the reflectance field of a human face. In: Proceedings of the 27th Annual Conference on Computer Graphics and Interactive Techniques, pp. 145–156 (2000)

    Google Scholar 

  5. Deng, J., Guo, J., Zhou, Y., Yu, J., Kotsia, I., Zafeiriou, S.: RetinaFace: single-stage dense face localisation in the wild. arXiv preprint arXiv:1905.00641 (2019)

  6. Fyffe, G., Debevec, P.: Single-shot reflectance measurement from polarized color gradient illumination. In: International Conference on Computational Photography (ICCP). IEEE (2015)

    Google Scholar 

  7. Fyffe, G., Graham, P., Tunwattanapong, B., Ghosh, A., Debevec, P.: Near-instant capture of high-resolution facial geometry and reflectance. In: Computer Graphics Forum (2016)

    Google Scholar 

  8. Fyffe, G., Hawkins, T., Watts, C., Ma, W.C., Debevec, P.: Comprehensive facial performance capture. In: Computer Graphics Forum (CGF), vol. 30, no. 2 (2011)

    Google Scholar 

  9. Gecer, B., et al.: Synthesizing coupled 3D face modalities by trunk-branch generative adversarial networks. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12374, pp. 415–433. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58526-6_25

    Chapter  Google Scholar 

  10. Ghosh, A., Fyffe, G., Tunwattanapong, B., Busch, J., Yu, X., Debevec, P.: Multiview face capture using polarized spherical gradient illumination. ACM TOG 30(6), 1–10 (2011)

    Article  Google Scholar 

  11. Ghosh, A., Hawkins, T., Peers, P., Frederiksen, S., Debevec, P.: Practical modeling and acquisition of layered facial reflectance. ACM TOG 27(5), 1–10 (2008)

    Article  Google Scholar 

  12. Gotardo, P., Riviere, J., Bradley, D., Ghosh, A., Beeler, T.: Practical dynamic facial appearance modeling and acquisition. ACM Trans. Graph. 37(6), 1–13 (2018)

    Article  Google Scholar 

  13. Guo, K., et al.: The relightables: volumetric performance capture of humans with realistic relighting. ACM Trans. Graph. (TOG) 38(6), 1–19 (2019)

    Google Scholar 

  14. Hernandez, C., Vogiatzis, G., Brostow, G.J., Stenger, B., Cipolla, R.: Non-rigid photometric stereo with colored lights. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (2007)

    Google Scholar 

  15. Jiang, J., Liu, D., Gu, J., Süsstrunk, S.: What is the space of spectral sensitivity functions for digital color cameras? In: 2013 IEEE Workshop on Applications of Computer Vision (WACV), pp. 168–179. IEEE (2013)

    Google Scholar 

  16. Kampouris, C., Zafeiriou, S., Ghosh, A.: Diffuse-specular separation using binary spherical gradient illumination. In: EGSR (EI &I), pp. 1–10 (2018)

    Google Scholar 

  17. Klaudiny, M., Hilton, A.: High-detail 3D capture and non-sequential alignment of facial performance. In: Proceedings of 3DIMPVT (2012)

    Google Scholar 

  18. Klehm, O., et al.: Recent advances in facial appearance capture. In: Computer Graphics Forum (CGF), vol. 34, no. 2, pp. 709–733 (2015)

    Google Scholar 

  19. Lattas, A., et al.: AvatarMe: realistically renderable 3D facial reconstruction “in-the-wild”. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 760–769 (2020)

    Google Scholar 

  20. Lattas, A., Moschoglou, S., Ploumpis, S., Gecer, B., Ghosh, A., Zafeiriou, S.P.: AvatarMe++: facial shape and BRDF inference with rendering-aware GANs. TPAMI (2021). https://ieeexplore.ieee.org/abstract/document/9606538

  21. Ma, W.C., Hawkins, T., Peers, P., Chabert, C.F., Weiss, M., Debevec, P.: Rapid acquisition of specular and diffuse normal maps from polarized spherical gradient illumination. In: Proceedings of the EGSR (2007)

    Google Scholar 

  22. Riviere, J., Gotardo, P., Bradley, D., Ghosh, A., Beeler, T.: Single-shot high-quality facial geometry and skin appearance capture. ACM Trans. Graph. 39(4), 1–12 (2020)

    Article  Google Scholar 

  23. Schonberger, J.L., Frahm, J.: Structure-from-motion revisited. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Los Alamitos, CA, USA, pp. 4104–4113. IEEE Computer Society, June 2016

    Google Scholar 

  24. Schönberger, J.L., Frahm, J.M.: Structure-from-motion revisited. In: Conference on Computer Vision and Pattern Recognition (CVPR) (2016)

    Google Scholar 

  25. Schönberger, J.L., Zheng, E., Frahm, J.-M., Pollefeys, M.: Pixelwise view selection for unstructured multi-view stereo. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9907, pp. 501–518. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46487-9_31

    Chapter  Google Scholar 

  26. Sengupta, S., Curless, B., Kemelmacher-Shlizerman, I., Seitz, S.M.: A light stage on every desk. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2420–2429 (2021)

    Google Scholar 

  27. Smith, W.A., Seck, A., Dee, H., Tiddeman, B., Tenenbaum, J.B., Egger, B.: A morphable face albedo model. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5011–5020 (2020)

    Google Scholar 

  28. Weyrich, T., Lawrence, J., Lensch, H., Rusinkiewicz, S., Zickler, T.: Principles of appearance acquisition and representation. Found. Trends Comput. Graph. Vis. 4(2), 75–191 (2008)

    Article  MATH  Google Scholar 

  29. Weyrich, T., et al.: Analysis of human faces using a measurement-based skin reflectance model. ACM Trans. Graph. (TOG) 25(3), 1013–1024 (2006)

    Article  Google Scholar 

  30. Wilson, C.A., Ghosh, A., Peers, P., Chiang, J.Y., Busch, J., Debevec, P.: Temporal upsampling of performance geometry using photometric alignment. ACM Trans. Graph. (TOG) 29(2), 17 (2010)

    Article  Google Scholar 

  31. Yuille, A.L., Snow, D., Epstein, R., Belhumeur, P.N.: Determining generative models of objects under varying illumination: shape and albedo from multiple images using SVD and integrability. Int. J. Comput. Vis. 35(3), 203–222 (1999). https://doi.org/10.1023/A:1008180726317

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Abhijeet Ghosh .

Editor information

Editors and Affiliations

1 Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (mp4 13839 KB)

Supplementary material 2 (pdf 5784 KB)

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Lattas, A. et al. (2022). Practical and Scalable Desktop-Based High-Quality Facial Capture. In: Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T. (eds) Computer Vision – ECCV 2022. ECCV 2022. Lecture Notes in Computer Science, vol 13666. Springer, Cham. https://doi.org/10.1007/978-3-031-20068-7_30

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-20068-7_30

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-20067-0

  • Online ISBN: 978-3-031-20068-7

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics