Skip to main content

Relighting4D: Neural Relightable Human from Videos

  • Conference paper
  • First Online:
Computer Vision – ECCV 2022 (ECCV 2022)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13674))

Included in the following conference series:

Abstract

Human relighting is a highly desirable yet challenging task. Existing works either require expensive one-light-at-a-time (OLAT) captured data using light stage or cannot freely change the viewpoints of the rendered body. In this work, we propose a principled framework, Relighting4D, that enables free-viewpoints relighting from only human videos under unknown illuminations. Our key insight is that the space-time varying geometry and reflectance of the human body can be decomposed as a set of neural fields of normal, occlusion, diffuse, and specular maps. These neural fields are further integrated into reflectance-aware physically based rendering, where each vertex in the neural field absorbs and reflects the light from the environment. The whole framework can be learned from videos in a self-supervised manner, with physically informed priors designed for regularization. Extensive experiments on both real and synthetic datasets demonstrate that our framework is capable of relighting dynamic human actors with free-viewpoints. Codes are available at https://github.com/FrozenBurning/Relighting4D.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Subscribe and save

Springer+ Basic
EUR 32.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or Ebook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 89.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 119.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Notes

  1. 1.

    For simplicity, we also use \(R(\cdot )\) to denote BRDF when necessary.

  2. 2.

    https://polyhaven.com/.

References

  1. Alldieck, T., Magnor, M., Xu, W., Theobalt, C., Pons-Moll, G.: Video based reconstruction of 3D people models. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8387–8397. IEEE (2018). https://doi.org/10.1109/CVPR.2018.00875, https://ieeexplore.ieee.org/document/8578973/

  2. Alldrin, N.G., Mallick, S.P., Kriegman, D.J.: Resolving the generalized bas-relief ambiguity by entropy minimization. In: 2007 IEEE Conference on Computer Vision and Pattern Recognition.,pp. 1–7 (2007). https://doi.org/10.1109/CVPR.2007.383208

  3. Barron, J.T., Malik, J.: Shape, albedo, and illumination from a single image of an unknown object. In: 2012 IEEE Conference on Computer Vision and Pattern Recognition, pp. 334–341. IEEE (2012). https://doi.org/10.1109/CVPR.2012.6247693, https://ieeexplore.ieee.org/document/6247693/

  4. Bi, S., et al.: Deep relightable appearance models for animatable faces. ACM Trans. Graph. 40(4), 1–15 (2021). https://doi.org/10.1145/3476576.3476647, https://dl.acm.org/doi/10.1145/3476576.3476647

  5. Bi, S., et al.: Neural reflectance fields for appearance acquisition. arXiv:2008.03824 [cs] (2020)

  6. Boss, M., Braun, R., Jampani, V., Barron, J.T., Liu, C., Lensch, H.P.A.: NeRD: neural reflectance decomposition from image collections. arXiv:2012.03918 [cs] (2021)

  7. Community, B.O.: Blender - a 3D modelling and rendering package. Blender Foundation (2018). http://www.blender.org

  8. Debevec, P., Hawkins, T., Tchou, C., Duiker, H.P., Sarokin, W.: Acquiring the Reflectance Field of a Human Face. In: SIGGRAPH (2000)

    Google Scholar 

  9. Finlayson, G.D., Drew, M.S., Lu, C.: Entropy minimization for shadow removal. Int. J. Comput. Vis. 85(1), 35–57 (2009). https://doi.org/10.1007/s11263-009-0243-z

  10. Genova, K., Cole, F., Sud, A., Sarna, A., Funkhouser, T.: Local deep implicit functions for 3D Shape. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4856–4865. IEEE (2020). https://doi.org/10.1109/CVPR42600.2020.00491, https://ieeexplore.ieee.org/document/9157823/

  11. Guo, K., et al.: The relightables: volumetric performance capture of humans with realistic relighting. ACM Trans. Graph. 38(6), 1–19 (2019) https://doi.org/10.1145/3355089.3356571, https://dl.acm.org/doi/10.1145/3355089.3356571

  12. Joo, H., Simon, T., Sheikh, Y.: Total capture: a 3D deformation model for tracking faces, hands, and bodies. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8320–8329 (2018). https://doi.org/10.1109/CVPR.2018.00868

  13. Kanamori, Y., Endo, Y.: Relighting humans: occlusion-aware inverse rendering for full-body human images. ACM Trans. Graph. 37(6), 1–11 (2019). https://doi.org/10.1145/3272127.3275104, https://arxiv.org/abs/1908.02714

  14. Kwon, Y., Kim, D., Ceylan, D., Fuchs, H.: Neural Human Performer: Learning Generalizable Radiance Fields for Human Performance Rendering. arXiv:2109.07448 [cs] (2021)

  15. Lagunas, M., et al.: Single-image Full-body human relighting. arXiv:2107.07259 [cs] (2021). https://doi.org/10.2312/sr.20211300

  16. LeGendre, C., et al.: DeepLight: learning illumination for unconstrained mobile mixed reality. CoRR abs/1904.01175 (2019), http://arxiv.org/abs/1904.01175

  17. LeGendre, C., et al.: Learning illumination from diverse portraits. In: SIGGRAPH Asia 2020 Technical Communications. SA 2020, Association for Computing Machinery (2020). https://doi.org/10.1145/3410700.3425432

  18. Li, J., Feng, Z., She, Q., Ding, H., Wang, C., Lee, G.H.: Mine: towards continuous depth MPI with nerf for novel view synthesis. In: ICCV (2021)

    Google Scholar 

  19. Li, Z., Niklaus, S., Snavely, N., Wang, O.: Neural scene flow fields for space-time view synthesis of dynamic scenes. arXiv:2011.13084 [cs] (2021)

  20. Liu, L., Habermann, M., Rudnev, V., Sarkar, K., Gu, J., Theobalt, C.: Neural actor: neural free-view synthesis of human actors with pose control. arXiv:2106.02019 [cs] (2021)

  21. Liu, Y., Neophytou, A., Sengupta, S., Sommerlade, E.: Relighting images in the wild with a self-supervised siamese auto-encoder. In: 2021 IEEE Winter Conference on Applications of Computer Vision (WaACV), pp. 32–40. IEEE (2021). https://doi.org/10.1109/WACV48630.2021.00008, https://ieeexplore.ieee.org/document/9423347/

  22. Liu, Y., Li, Y., You, S., Lu, F.: Unsupervised learning for intrinsic image decomposition from a single image. arXiv:1911.09930 [cs] (2020)

  23. Loper, M., Mahmood, N., Romero, J., Pons-Moll, G., Black, M.J.: SMPL: a skinned multi-person linear model. ACM Trans. Graphics (Proc. SIGGRAPH Asia) 34(6), 248:1–248:16 (2015)

    Google Scholar 

  24. Martin-Brualla, R., Radwan, N., Sajjadi, M.S.M., Barron, J.T., Dosovitskiy, A., Duckworth, D.: NeRF in the Wild: Neural radiance fields for unconstrained photo collections. In: CVPR (2021)

    Google Scholar 

  25. Meka, A., et al.: Deep reflectance fields: high-quality facial reflectance field inference from color gradient illumination. ACM Trans. Graph. 38(4), 1–12 (2019). https://doi.org/10.1145/3306346.3323027, https://dl.acm.org/doi/10.1145/3306346.3323027

  26. Meka, A., et al.: Deep relightable textures: volumetric performance capture with neural rendering. ACM Trans. Graph. 39(6), 1–21 (2020)

    Article  Google Scholar 

  27. Meka, A., Shafiei, M., Zollhoefer, M., Richardt, C., Theobalt, C.: Real-time global illumination decomposition of videos. ACM Trans. Graph. 40(3), 1–16 (2021). https://doi.org/10.1145/3374753, http://arxiv.org/abs/1908.01961

  28. Mildenhall, B., Srinivasan, P.P., Tancik, M., Barron, J.T., Ramamoorthi, R., Ng, R.: NeRF: representing Scenes as neural radiance fields for view synthesis. arXiv:2003.08934 [cs] (2020)

  29. Oechsle, M., Peng, S., Geiger, A.: UNISURF: unifying neural implicit surfaces and radiance fields for multi-view reconstruction. arXiv:2104.10078 [cs] (2021)

  30. Ost, J., Mannan, F., Thuerey, N., Knodt, J., Heide, F.: Neural Scene Graphs for Dynamic Scenes. arXiv:2011.10379 [cs] (2021)

  31. Pandey, R., et al.: Total relighting: learning to relight portraits for background replacement. ACM Trans. Graph. 40(4), 1–21 (2021). https://doi.org/10.1145/3476576.3476588, https://dl.acm.org/doi/10.1145/3476576.3476588

  32. Park, K., et al.: Nerfies: deformable neural radiance fields. arXiv:2011.12948 [cs] (2021)

  33. Pavlakos, G., et al.: Expressive body capture: 3D hands, face, and body from a single image. In: Proceedings IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10975–10985 (2019)

    Google Scholar 

  34. Peng, S., et al.: Neural body: implicit neural representations with structured latent codes for novel view synthesis of dynamic humans. arXiv:2012.15838 [cs] (2021)

  35. Pumarola, A., Corona, E., Pons-Moll, G., Moreno-Noguer, F.: D-NeRF: neural radiance fields for dynamic scenes. arXiv:2011.13961 [cs] (2020)

  36. Raj, A., Tanke, J., Hays, J., Vo, M., Stoll, C., Lassner, C.: ANR: articulated neural rendering for virtual avatars. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3722–3731 (2021)

    Google Scholar 

  37. Sengupta, S., Gu, J., Kim, K., Liu, G., Jacobs, D., Kautz, J.: Neural Inverse rendering of an indoor scene from a single image. In: 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 8597–8606. IEEE (2019). https://doi.org/10.1109/ICCV.2019.00869, https://ieeexplore.ieee.org/document/9008823/

  38. Shen, L., Yeo, C.: Intrinsic images decomposition using a local and global sparse representation of reflectance. In: CVPR 2011, pp. 697–704 (2011). https://doi.org/10.1109/CVPR.2011.5995738

  39. Shu, Z., Yumer, E., Hadap, S., Sunkavalli, K., Shechtman, E., Samaras, D.: Neural face editing with intrinsic image disentangling. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5444–5453. IEEE (2017). https://doi.org/10.1109/CVPR.2017.578, https://ieeexplore.ieee.org/document/8100061/

  40. Sitzmann, V., Martel, J., Bergman, A., Lindell, D., Wetzstein, G.: Implicit neural representations with periodic activation functions. In: Advances in Neural Information Processing Systems, vol. 33, pp. 7462–7473. Curran Associates, Inc. (2020). https://proceedings.neurips.cc/paper/2020/hash/53c04118df112c13a8c34b38343b9c10-Abstract.html

  41. Sitzmann, V., Zollhöfer, M., Wetzstein, G.: Scene representation networks: continuous 3D-structure-aware neural scene representations. arXiv:1906.01618 [cs] (2020)

  42. Srinivasan, P.P., Deng, B., Zhang, X., Tancik, M., Mildenhall, B., Barron, J.T.: NeRV: neural reflectance and visibility fields for relighting and view synthesis. arXiv:2012.03927 [cs] (2020)

  43. Sun, G., et al.: Neural free-viewpoint performance rendering under complex human-object interactions. arXiv:2108.00362 [cs] (2021)

  44. Sun, T., et al.: Single Image Portrait Relighting. ACM Trans. Graph. 38(4), 1–12 (2019). |DOIurl10.1145/3306346.3323008, https://arxiv.org/abs/1905.00824

  45. Sun, T., Lin, K., Bi, S., Xu, Z., Ramamoorthi, R.: Nelf: Neural light-transport field for portrait view synthesis and relighting. CoRR abs/2107.12351 (2021). https://arxiv.org/abs/2107.12351

  46. Suo, X., et al.: NeuralHumanFVV: real-time neural volumetric human performance rendering using RGB cameras. arXiv:2103.07700 [cs] (2021)

  47. Tajima, D., Kanamori, Y., Endo, Y.: Relighting humans in the wild: Monocular full-body human relighting with domain adaptation (2021)

    Google Scholar 

  48. Tancik, M., et al.: Fourier features let networks learn high frequency functions in low dimensional domains. arXiv:2006.10739 [cs] (2020)

  49. Tucker, R., Snavely, N.: Single-view view synthesis with multiplane images. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 548–557. IEEE (2020). https://doi.org/10.1109/CVPR42600.2020.00063, https://ieeexplore.ieee.org/document/9156372/

  50. Walter, B., Marschner, S.R., Li, H., Torrance, K.E.: Microfacet models for refraction through rough surfaces. In: Proceedings of the 18th Eurographics Conference on Rendering Techniques, pp. 195–206. EGSR 2007, Eurographics Association (2007). https://doi.org/10.2312/EGWR/EGSR07/195-206

  51. Wang, Z., Yu, X., Lu, M., Wang, Q., Qian, C., Xu, F.: Single image portrait relighting via explicit multiple reflectance channel modeling. ACM Trans. Graph. 39(6), 1–13 (2020). https://doi.org/10.1145/3414685.3417824, https://dl.acm.org/doi/10.1145/3414685.3417824

  52. Wang, Z., Bovik, A., Sheikh, H., Simoncelli, E.: Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13(4), 600–612 (2004). https://doi.org/10.1109/TIP.2003.819861

    Article  Google Scholar 

  53. Wang, Z., Philion, J., Fidler, S., Kautz, J.: Learning indoor inverse rendering with 3D spatially-varying lighting. In: Proceedings of International Conference on Computer Vision (ICCV) (2021)

    Google Scholar 

  54. Xian, W., Huang, J.B., Kopf, J., Kim, C.: Space-time neural irradiance fields for free-viewpoint video. arXiv:2011.12950 [cs] (2021)

  55. Xu, D., Principe, J.: Learning from examples with quadratic mutual information. In: Neural Networks for Signal Processing VIII. Proceedings of the 1998 IEEE Signal Processing Society Workshop (Cat. No.98TH8378), pp. 155–164 (1998). https://doi.org/10.1109/NNSP.1998.710645

  56. Yu, A., Ye, V., Tancik, M., Kanazawa, A.: pixelNeRF: neural radiance fields from one or few images (2020)

    Google Scholar 

  57. Yu, Y., Smith, W.A.P.: InverseRenderNet: Learning single image inverse rendering. arXiv:1811.12328 [cs] (2018)

  58. Zhang, J., et al.: Editable free-viewpoint video using a layered neural representation. ACM Trans. Graph. 40(4), 1–18 (2021). https://doi.org/10.1145/3450626.3459756, https://arxiv.org/abs/2104.14786

  59. Zhang, K., Luan, F., Wang, Q., Bala, K., Snavely, N.: PhySG: inverse rendering with spherical gaussians for physics-based material editing and relighting. arXiv:2104.00674 [cs] (2021)

  60. Zhang, L., Zhang, Q., Wu, M., Yu, J., Xu, L.: Neural video portrait relighting in real-time via consistency modeling. arXiv:2104.00484 [cs] (2021)

  61. Zhang, R., Isola, P., Efros, A.A., Shechtman, E., Wang, O.: The unreasonable effectiveness of deep features as a perceptual metric. In: CVPR (2018)

    Google Scholar 

  62. Zhang, X., et al.: Neural light transport for relighting and view synthesis. ACM Trans. Graph. 40(1), 1–17 (2021). https://dl.acm.org/doi/10.1145/3446328

  63. Zhang, X., Srinivasan, P.P., Deng, B., Debevec, P., Freeman, W.T., Barron, J.T.: Nerfactor. ACM Trans. Graph. 40(6), 1–18 (2021). https://doi.org/10.1145/3478513.3480496, https://dx.doi.org/10.1145/3478513.3480496

  64. Zhou, H., Hadap, S., Sunkavalli, K., Jacobs, D.: Deep single-image portrait relighting. In: 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 7193–7201. IEEE (2019). https://doi.org/10.1109/ICCV.2019.00729, https://ieeexplore.ieee.org/document/9010718/

Download references

Acknowledgements

This work is supported by the National Research Foundation, Singapore under its AI Singapore Programme (AISG Award No: AISG2-PhD-2021-08-019), NTU NAP, MOE AcRF Tier 2 (T2EP20221-0033), and under the RIE2020 Industry Alignment Fund - Industry Collaboration Projects (IAF-ICP) Funding Initiative, as well as cash and in-kind contribution from the industry partner(s).

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Ziwei Liu .

Editor information

Editors and Affiliations

1 Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (pdf 11694 KB)

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Chen, Z., Liu, Z. (2022). Relighting4D: Neural Relightable Human from Videos. In: Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T. (eds) Computer Vision – ECCV 2022. ECCV 2022. Lecture Notes in Computer Science, vol 13674. Springer, Cham. https://doi.org/10.1007/978-3-031-19781-9_35

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-19781-9_35

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-19780-2

  • Online ISBN: 978-3-031-19781-9

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics