Skip to main content
SpringerLink
Log in

Neural Radiance Transfer Fields for Relightable Novel-View Synthesis with Global Illumination

  • Conference paper
  • First Online:
Computer Vision – ECCV 2022 (ECCV 2022)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13677))

Included in the following conference series:

Abstract

Given a set of images of a scene, the re-rendering of this scene from novel views and lighting conditions is an important and challenging problem in Computer Vision and Graphics. On the one hand, most existing works in Computer Vision usually impose many assumptions regarding the image formation process, e.g. direct illumination and predefined materials, to make scene parameter estimation tractable. On the other hand, mature Computer Graphics tools allow modeling of complex photo-realistic light transport given all the scene parameters. Combining these approaches, we propose a method for scene relighting under novel views by learning a neural precomputed radiance transfer function, which implicitly handles global illumination effects using novel environment maps. Our method can be solely supervised on a set of real images of the scene under a single unknown lighting condition. To disambiguate the task during training, we tightly integrate a differentiable path tracer in the training process and propose a combination of a synthesized OLAT and a real image loss. Results show that the recovered disentanglement of scene parameters improves significantly over the current state of the art and, thus, also our re-rendering results are more realistic and accurate.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 89.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 119.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Azinović, D., Li, T.M., Kaplanyan, A., Nießner, M.: Inverse path tracing for joint material and lighting estimation. In: Proceedings of the Computer Vision and Pattern Recognition (CVPR). IEEE (2019)

    Google Scholar 

  2. Bangaru, S., Michel, J., Mu, K., Bernstein, G., Li, T.M., Ragan-Kelley, J.: Systematically differentiating parametric discontinuities. ACM Trans. Graph. 40(107), 107:1–107:17 (2021)

    Google Scholar 

  3. Boss, M., Braun, R., Jampani, V., Barron, J.T., Liu, C., Lensch, H.P.: NeRD: neural reflectance decomposition from image collections. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 12684–12694 (2021)

    Google Scholar 

  4. Boss, M., Jampani, V., Braun, R., Liu, C., Barron, J., Lensch, H.: Neural-PIL: neural pre-integrated lighting for reflectance decomposition. Adv. Neural. Inf. Process. Syst. 34, 10691–10704 (2021)

    Google Scholar 

  5. Chen, Z., et al.: A neural rendering framework for free-viewpoint relighting. In: CVPR (2020)

    Google Scholar 

  6. Community, B.O.: Blender - a 3D modelling and rendering package (2018). https://www.blender.org

  7. Debevec, P., Hawkins, T., Tchou, C., Duiker, H.P., Sarokin, W., Sagar, M.: Acquiring the reflectance field of a human face. In: Proceedings of the 27th Annual Conference on Computer Graphics and Interactive Techniques, pp. 145–156 (2000)

    Google Scholar 

  8. Goel, P., Cohen, L., Guesman, J., Thamizharasan, V., Tompkin, J., Ritchie, D.: Shape from tracing: towards reconstructing 3D object geometry and svbrdf material from images via differentiable path tracing. In: 2020 International Conference on 3D Vision (3DV), pp. 1186–1195. IEEE (2020)

    Google Scholar 

  9. Hao, X., Baby, T., Varshney, A.: Interactive subsurface scattering for translucent meshes. In: Proceedings of the 2003 Symposium on Interactive 3D Graphics, pp. 75–82 (2003)

    Google Scholar 

  10. Jensen, R., Dahl, A., Vogiatzis, G., Tola, E., Aanæs, H.: Large scale multi-view stereopsis evaluation. In: 2014 IEEE Conference on Computer Vision and Pattern Recognition, pp. 406–413. IEEE (2014)

    Google Scholar 

  11. Kajiya, J.T.: The rendering equation. In: Proceedings of the 13th Annual Conference on Computer Graphics and Interactive Techniques, pp. 143–150 (1986)

    Google Scholar 

  12. Kautz, J., Sloan, P.P., Lehtinen, J.: Precomputed radiance transfer: theory and practice. In: ACM SIGGRAPH 2005 Courses, pp. 1–es (2005)

    Google Scholar 

  13. Kautz, J., Snyder, J., Sloan, P.P.J.: Fast arbitrary BRDF shading for low-frequency lighting using spherical harmonics. Rendering Tech. 2(291–296), 1 (2002)

    Google Scholar 

  14. Laffont, P.Y., Bousseau, A., Drettakis, G.: Rich intrinsic image decomposition of outdoor scenes from multiple views. IEEE Trans. Visual. Comput. Graph. 19(2), 210–224 (2012)

    Article  Google Scholar 

  15. Land, E.H., McCann, J.J.: Lightness and retinex theory. Josa 61(1), 1–11 (1971)

    Article  Google Scholar 

  16. Lehtinen, J.: A framework for precomputed and captured light transport. ACM Trans. Graph. (TOG) 26(4), 13-es (2007)

    Article  Google Scholar 

  17. Lehtinen, J., et al.: Noise2noise: learning image restoration without clean data. arXiv preprint arXiv:1803.04189 (2018)

  18. Li, T.M., Aittala, M., Durand, F., Lehtinen, J.: Differentiable monte carlo ray tracing through edge sampling. ACM Trans. Graph. (TOG) 37(6), 1–11 (2018)

    Article  Google Scholar 

  19. Lombardi, S., Nishino, K.: Reflectance and illumination recovery in the wild. IEEE Trans. Pattern Anal. Mach. Intell. 38(1), 129–141 (2015)

    Article  Google Scholar 

  20. Lorensen, W.E., Cline, H.E.: Marching cubes: a high resolution 3D surface construction algorithm. ACM siggraph comput. graph. 21(4), 163–169 (1987)

    Article  Google Scholar 

  21. Lyu, L., Habermann, M., Liu, L., Tewari, A., Theobalt, C., et al.: Efficient and differentiable shadow computation for inverse problems. In: ICCV, pp. 13107–13116 (2021)

    Google Scholar 

  22. Marschner, S.R.: Inverse Rendering for Computer Graphics. Cornell University, Ithaca (1998)

    Google Scholar 

  23. Mildenhall, B., Srinivasan, P.P., Tancik, M., Barron, J.T., Ramamoorthi, R., Ng, R.: NeRF: representing scenes as neural radiance fields for view synthesis. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12346, pp. 405–421. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58452-8_24

    Chapter  Google Scholar 

  24. Müller, T., Evans, A., Schied, C., Keller, A.: Instant neural graphics primitives with a multiresolution hash encoding. arXiv:2201.05989 (2022)

  25. Munkberg, J., et al.: Extracting triangular 3d models, materials, and lighting from images. arXiv preprint arXiv:2111.12503 (2021)

  26. Ng, R., Ramamoorthi, R., Hanrahan, P.: All-frequency shadows using non-linear wavelet lighting approximation. In: ACM SIGGRAPH 2003 Papers, pp. 376–381 (2003)

    Google Scholar 

  27. Nimier-David, M., Dong, Z., Jakob, W., Kaplanyan, A.: Material and lighting reconstruction for complex indoor scenes with texture-space differentiable rendering (2021)

    Google Scholar 

  28. Nimier-David, M., Vicini, D., Zeltner, T., Jakob, W.: Mitsuba 2: a retargetable forward and inverse renderer. ACM Trans. Graph. (TOG) 38(6), 1–17 (2019)

    Article  Google Scholar 

  29. Pandey, R., et al.: Total relighting: learning to relight portraits for background replacement, vol. 40 (2021). https://doi.org/10.1145/3450626.3459872

  30. Philip, J., Gharbi, M., Zhou, T., Efros, A.A., Drettakis, G.: Multi-view relighting using a geometry-aware network. ACM Trans. Graph. 38(4), 78–1 (2019)

    Article  Google Scholar 

  31. Philip, J., Morgenthaler, S., Gharbi, M., Drettakis, G.: Free-viewpoint indoor neural relighting from multi-view stereo. ACM Trans. Graph. (TOG) 40(5), 1–18 (2021)

    Article  Google Scholar 

  32. Rainer, G., Bousseau, A., Ritschel, T., Drettakis, G.: Neural precomputed radiance transfer. In: Computer Graphics Forum (Proceedings of Eurographics), vol. 41, no. 2 (2022). https://www-sop.inria.fr/reves/Basilic/2022/RBRD22

  33. Ramamoorthi, R., Hanrahan, P.: A signal-processing framework for inverse rendering. In: Proceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques, pp. 117–128 (2001)

    Google Scholar 

  34. Ramamoorthi, R., Hanrahan, P.: A signal-processing framework for reflection. ACM Trans. Graph. (TOG) 23(4), 1004–1042 (2004)

    Article  Google Scholar 

  35. Ritschel, T., Dachsbacher, C., Grosch, T., Kautz, J.: The state of the art in interactive global illumination. In: Computer Graphics Forum, vol. 31, pp. 160–188. Wiley Online Library (2012)

    Google Scholar 

  36. Sloan, P.P., Kautz, J., Snyder, J.: Precomputed radiance transfer for real-time rendering in dynamic, low-frequency lighting environments. In: Proceedings of the SIGGRAPH, pp. 527–536 (2002)

    Google Scholar 

  37. Sloan, P.P., Luna, B., Snyder, J.: Local, deformable precomputed radiance transfer. ACM Trans. Graph. (TOG) 24(3), 1216–1224 (2005)

    Article  Google Scholar 

  38. Srinivasan, P.P., Deng, B., Zhang, X., Tancik, M., Mildenhall, B., Barron, J.T.: NeRV: neural reflectance and visibility fields for relighting and view synthesis. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7495–7504 (2021)

    Google Scholar 

  39. Sun, T., et al.: Light stage super-resolution: continuous high-frequency relighting. ACM Trans. Graph. (TOG) 39(6), 1–12 (2020)

    Article  Google Scholar 

  40. Tewari, A., et al.: Advances in neural rendering. arXiv preprint arXiv:2111.05849 (2021)

  41. Thul, D., Tsiminaki, V., Ladickỳ, L., Pollefeys, M.: Precomputed radiance transfer for reflectance and lighting estimation. In: 2020 International Conference on 3D Vision (3DV), pp. 1147–1156. IEEE (2020)

    Google Scholar 

  42. Tsai, Y.T., Shih, Z.C.: All-frequency precomputed radiance transfer using spherical radial basis functions and clustered tensor approximation. ACM Trans. graph. (TOG) 25(3), 967–976 (2006)

    Article  Google Scholar 

  43. Wang, J., Ramamoorthi, R.: Analytic spherical harmonic coefficients for polygonal area lights. ACM Trans. Graph. (Proc. SIGGRAPH 2018) 37(4), 1–11 (2018)

    Google Scholar 

  44. Wang, P., Liu, L., Liu, Y., Theobalt, C., Komura, T., Wang, W.: NeuS: learning neural implicit surfaces by volume rendering for multi-view reconstruction. arXiv preprint arXiv:2106.10689 (2021)

  45. Wang, Z., Bovik, A., Sheikh, H., Simoncelli, E.: Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13(4), 600–612 (2004). https://doi.org/10.1109/TIP.2003.819861

    Article  Google Scholar 

  46. Wu, L., Cai, G., Zhao, S., Ramamoorthi, R.: Analytic spherical harmonic gradients for real-time rendering with many polygonal area lights. ACM Trans. Graph. (TOG) 39(4), 134–1 (2020)

    Google Scholar 

  47. Wu, S., Rupprecht, C., Vedaldi, A.: Unsupervised learning of probably symmetric deformable 3D objects from images in the wild. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1–10 (2020)

    Google Scholar 

  48. Xie, Y., et al.: Neural fields in visual computing and beyond. arXiv preprint arXiv:2111.11426 (2021)

  49. Xu, K., Sun, W.L., Dong, Z., Zhao, D.Y., Wu, R.D., Hu, S.M.: Anisotropic spherical gaussians. ACM Trans. Graph. (TOG) 32(6), 1–11 (2013)

    Google Scholar 

  50. Yu, A., Fridovich-Keil, S., Tancik, M., Chen, Q., Recht, B., Kanazawa, A.: Plenoxels: radiance fields without neural networks. arXiv preprint arXiv:2112.05131 (2021)

  51. Zhang, K., Luan, F., Wang, Q., Bala, K., Snavely, N.: PhySG: inverse rendering with spherical gaussians for physics-based material editing and relighting. In: CVPR (2021)

    Google Scholar 

  52. Zhang, R., Isola, P., Efros, A.A., Shechtman, E., Wang, O.: The unreasonable effectiveness of deep features as a perceptual metric. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 586–595. IEEE Computer Society, Los Alamitos (2018). https://doi.org/10.1109/CVPR.2018.00068, https://doi.ieeecomputersociety.org/10.1109/CVPR.2018.00068

  53. Zhang, X., et al.: Neural light transport for relighting and view synthesis. ACM Trans. Graph. (TOG) 40(1), 1–17 (2021)

    Article  Google Scholar 

  54. Zhang, X., Srinivasan, P.P., Deng, B., Debevec, P., Freeman, W.T., Barron, J.T.: NeRFactor: neural factorization of shape and reflectance under an unknown illumination. ACM Trans. Graph. (TOG) 40(6), 1–18 (2021)

    Article  Google Scholar 

Download references

Acknowledgement

We would like to thank Xiuming Zhang for his help with the NeRFactor comparisons. Authors from MPII were supported by the ERC Consolidator Grant 4DRepLy (770784).

Author information

Authors and Affiliations

  1. Max Planck Institute for Informatics, Saarland Informatics Campus, Saarbrücken, Germany

    Linjie Lyu, Thomas Leimkühler, Marc Habermann & Christian Theobalt

  2. MIT, Cambridge, USA

    Ayush Tewari

Authors
  1. Linjie Lyu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ayush Tewari
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Thomas Leimkühler
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Marc Habermann
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Christian Theobalt
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Linjie Lyu .

Editor information

Editors and Affiliations

  1. Tel Aviv University, Tel Aviv, Israel

    Shai Avidan

  2. University College London, London, UK

    Gabriel Brostow

  3. Google AI, Accra, Ghana

    Moustapha Cissé

  4. University of Catania, Catania, Italy

    Giovanni Maria Farinella

  5. Facebook (United States), Menlo Park, CA, USA

    Tal Hassner

1 Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 2 (pdf 4101 KB)

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Lyu, L., Tewari, A., Leimkühler, T., Habermann, M., Theobalt, C. (2022). Neural Radiance Transfer Fields for Relightable Novel-View Synthesis with Global Illumination. In: Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T. (eds) Computer Vision – ECCV 2022. ECCV 2022. Lecture Notes in Computer Science, vol 13677. Springer, Cham. https://doi.org/10.1007/978-3-031-19790-1_10

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-19790-1_10

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-19789-5

  • Online ISBN: 978-3-031-19790-1

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation