Skip to main content

NeRF for Outdoor Scene Relighting

  • Conference paper
  • First Online:
Computer Vision – ECCV 2022 (ECCV 2022)

Abstract

Photorealistic editing of outdoor scenes from photographs requires a profound understanding of the image formation process and an accurate estimation of the scene geometry, reflectance and illumination. A delicate manipulation of the lighting can then be performed while keeping the scene albedo and geometry unaltered. We present NeRF-OSR, i.e., the first approach for outdoor scene relighting based on neural radiance fields. In contrast to the prior art, our method allows simultaneous editing of illumination and camera viewpoint using only a collection of outdoor photos shot in uncontrolled settings. Moreover, it enables direct control over the scene illumination, as defined through a spherical harmonics model. For evaluation, we collect a new benchmark dataset of several outdoor sites photographed from multiple viewpoints and at different times. For each time, a \(360^\circ \) environment map is captured together with a colour-calibration chequerboard to allow accurate numerical evaluations on real data against ground truth. Comparisons against SoTA show that NeRF-OSR enables controllable lighting and viewpoint editing at higher quality and with realistic self-shadowing reproduction. (see the project web page https://4dqv.mpi-inf.mpg.de/NeRF-OSR/)

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Subscribe and save

Springer+ Basic
EUR 32.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 89.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 119.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Barron, J.T., Malik, J.: Shape, illumination, and reflectance from shading. IEEE Trans. Pattern Anal. Mach. Intell. (TPAMI) 37(8), 1670–1687 (2015)

    Article  Google Scholar 

  2. Basri, R., Jacobs, D.W.: Lambertian reflectance and linear subspaces. IEEE TPAMI 25(2), 218–233 (2003)

    Article  Google Scholar 

  3. Boss, M., Braun, R., Jampani, V., Barron, J.T., Liu, C., Lensch, H.P.: Nerd: neural reflectance decomposition from image collections. In: International Conference on Computer Vision (ICCV) (2021)

    Google Scholar 

  4. Chen, X., et al.: Hallucinated neural radiance fields in the wild. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12943–12952 (2022)

    Google Scholar 

  5. Duchêne, S., et al.: Multiview intrinsic images of outdoors scenes with an application to relighting. ACM Trans. Graph. 34(5) (2015)

    Google Scholar 

  6. Finlayson, G.D., Mackiewicz, M., Hurlbert, A.: Color correction using root-polynomial regression. IEEE Trans. Image Process. 24(5), 1460–1470 (2015)

    Article  MathSciNet  Google Scholar 

  7. Garon, M., Sunkavalli, K., Hadap, S., Carr, N., Lalonde, J.F.: Fast spatially-varying indoor lighting estimation. In: Computer Vision and Pattern Recognition (CVPR) (2019)

    Google Scholar 

  8. Guo, K., et al.: The relightables: volumetric performance capture of humans with realistic relighting. ACM Trans. Graph. 38(6), 1–19 (2019)

    Google Scholar 

  9. Jin, Y., et al.: Image matching across wide baselines: from paper to practice. Int. J. Comput. Vis. 1–31 (2020). https://doi.org/10.1007/s11263-020-01385-0

  10. Karsch, K., Hedau, V., Forsyth, D., Hoiem, D.: Rendering synthetic objects into legacy photographs. ACM Trans. Graph. 30(6), 1–12 (2011)

    Article  Google Scholar 

  11. Laffont, P.Y., Bousseau, A., Paris, S., Durand, F., Drettakis, G.: Coherent intrinsic images from photo collections. ACM Trans. Graph. 31(6) (2012)

    Google Scholar 

  12. Lalonde, J.F., Efros, A.A., Narasimhan, S.G.: Webcam clip art: appearance and illuminant transfer from time-lapse sequences. ACM Trans. Graph. 28(5), 1–10 (2009)

    Article  Google Scholar 

  13. Li, Y., Snavely, N., Huttenlocher, D., Fua, P.: Worldwide pose estimation using 3d point clouds. In: European Conference on Computer Vision (ECCV), pp. 15–29 (2012)

    Google Scholar 

  14. Li, Z., Snavely, N.: Megadepth: learning single-view depth prediction from internet photos. In: Computer Vision and Pattern Recognition (CVPR) (2018)

    Google Scholar 

  15. Li, Z., Xian, W., Davis, A., Snavely, N.: Crowdsampling the plenoptic function. In: European Conference on Computer Vision (ECCV) (2020)

    Google Scholar 

  16. Lorensen, W.E., Cline, H.E.: Marching cubes: a high resolution 3d surface construction algorithm. ACM siggraph Comput. Graph. 21(4), 163–169 (1987)

    Article  Google Scholar 

  17. Luan, F., Paris, S., Shechtman, E., Bala, K.: Deep photo style transfer. In: Computer Vision and Pattern Recognition (CVPR), pp. 6997–7005 (2017)

    Google Scholar 

  18. Mallikarjun, B.R., et al.: Photoapp: photorealistic appearance editing of head portraits. ACM Trans. Graph. 40(4), 44 (2021)

    Google Scholar 

  19. Martin-Brualla, R., Radwan, N., Sajjadi, M.S.M., Barron, J.T., Dosovitskiy, A., Duckworth, D.: NeRF in the wild: neural radiance fields for unconstrained photo collections. In: Computer Vision and Pattern Recognition (CVPR) (2021)

    Google Scholar 

  20. Meka, A., et al.: Lime: live intrinsic material estimation. In: Computer Vision and Pattern Recognition (CVPR) (2018)

    Google Scholar 

  21. Meka, A., et al.: Deep relightable textures - volumetric performance capture with neural rendering. In: ACM Trans. Graph. (Proceedings SIGGRAPH Asia), 39(6), 1–21 (2020)

    Google Scholar 

  22. Meshry, M., Goldman, D.B., Khamis, S., Hoppe, H., Pandey, R., Snavely, N., Martin-Brualla, R.: Neural re-rendering in the wild. In: Computer Vision and Pattern Recognition (CVPR), pp. 6871–6880 (2019)

    Google Scholar 

  23. Mildenhall, B., Srinivasan, P.P., Tancik, M., Barron, J.T., Ramamoorthi, R., Ng, R.: Nerf: representing scenes as neural radiance fields for view synthesis. In: European Conference on Computer Vision (ECCV) (2020)

    Google Scholar 

  24. Nam, S., Ma, C., Chai, M., Brendel, W., Xu, N., Kim, S.: End-to-end time-lapse video synthesis from a single outdoor image. In: Computer Vision and Pattern Recognition (CVPR), pp. 1409–1418 (2019)

    Google Scholar 

  25. Oechsle, M., Peng, S., Geiger, A.: Unisurf: unifying neural implicit surfaces and radiance fields for multi-view reconstruction. In: International Conference on Computer Vision (ICCV) (2021)

    Google Scholar 

  26. Park, K., et al.: Nerfies: deformable neural radiance fields. In: International Conference on Computer Vision (ICCV) (2021)

    Google Scholar 

  27. Philip, J., Gharbi, M., Zhou, T., Efros, A.A., Drettakis, G.: Multi-view relighting using a geometry-aware network. ACM Trans. Graph. 38(4), 1–78 (2019)

    Article  Google Scholar 

  28. Schönberger, J.L., Zheng, E., Frahm, J.M., Pollefeys, M.: Pixelwise view selection for unstructured multi-view stereo. In: European Conference on Computer Vision (ECCV)m pp. 501–518 (2016)

    Google Scholar 

  29. Schönberger, J.L., Frahm, J.M.: Structure-from-motion revisited. In: Computer Vision and Pattern Recognition (CVPR), pp. 4104–4113 (2016)

    Google Scholar 

  30. Sengupta, S., Gu, J., Kim, K., Liu, G., Jacobs, D.W., Kautz, J.: Neural inverse rendering of an indoor scene from a single image. In: International Conference on Computer Vision (ICCV) (2019)

    Google Scholar 

  31. Shih, Y., Paris, S., Durand, F., Freeman, W.T.: Data-driven hallucination of different times of day from a single outdoor photo. ACM Trans. Graph. 32(6), 1–11 (2013)

    Article  Google Scholar 

  32. Snavely, N., Seitz, S.M., Szeliski, R.: Photo tourism: Exploring photo collections in 3d. ACM Trans. Graph. 25(3), 835–846 (2006)

    Article  Google Scholar 

  33. Srinivasan, P.P., Deng, B., Zhang, X., Tancik, M., Mildenhall, B., Barron, J.T.: Nerv: neural reflectance and visibility fields for relighting and view synthesis. In: Computer Vision and Pattern Recognition (CVPR) (2021)

    Google Scholar 

  34. Sun, T., et al.: Single image portrait relighting. ACM Trans. Graph. 38(4), 1–79 (2019)

    Article  Google Scholar 

  35. Sun, T., Lin, K.E., Bi, S., Xu, Z., Ramamoorthi, R.: Nelf: neural light-transport field for portrait view synthesis and relighting. In: Eurographics Symposium on Rendering (2021)

    Google Scholar 

  36. Sunkavalli, K., Matusik, W., Pfister, H., Rusinkiewicz, S.: Factored time-lapse video. ACM Trans. Graph. 26(3), 101-es (2007)

    Google Scholar 

  37. Tancik, M., et al.: Block-NeRF: scalable large scene neural view synthesis. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8248–8258 (2022)

    Google Scholar 

  38. Tao, A., Sapra, K., Catanzaro, B.: Hierarchical multi-scale attention for semantic segmentation. arXiv preprint arXiv:2005.10821 (2020)

  39. Tewari, A., et al.: Advances in neural rendering. arXiv e-prints (2021)

    Google Scholar 

  40. Tretschk, E., Tewari, A., Golyanik, V., Zollhöfer, M., Lassner, C., Theobalt, C.: Non-rigid neural radiance fields: Reconstruction and novel view synthesis of a dynamic scene from monocular video. In: International Conference on Computer Vision (ICCV) (2021)

    Google Scholar 

  41. Van der Walt, S., et al.: scikit-image: image processing in python. PeerJ 2, e453 (2014)

    Article  Google Scholar 

  42. Wang, P., Liu, L., Liu, Y., Theobalt, C., Komura, T., Wang, W.: Neus: learning neural implicit surfaces by volume rendering for multi-view reconstruction. In: Neural Information Processing Systems (NeurIPS) (2021)

    Google Scholar 

  43. Williams, L.: Casting curved shadows on curved surfaces. In: Proceedings of the 5th Annual Conference on Computer Graphics and Interactive Techniques, pp. 270–274 (1978)

    Google Scholar 

  44. Xing, G., Zhou, X., Peng, Q., Liu, Y., Qin, X.: Lighting simulation of augmented outdoor scene based on a legacy photograph. In: Computer Graphics Forum (2013)

    Google Scholar 

  45. Xu, Z., Sunkavalli, K., Hadap, S., Ramamoorthi, R.: Deep image-based relighting from optimal sparse samples. ACM Trans. Graph. 37(4), 126 (2018)

    Article  Google Scholar 

  46. Yu, Y., Meka, A., Elgharib, M., Seidel, H.P., Theobalt, C., Smith, W.: Self-supervised outdoor scene relighting. In: European Conference on Computer Vision (ECCV) (2020)

    Google Scholar 

  47. Yu, Y., Smith, W.A.: Inverserendernet: learning single image inverse rendering. In: Computer Vision and Pattern Recognition (CVPR) (2019)

    Google Scholar 

  48. Zhang, K., Riegler, G., Snavely, N., Koltun, V.: Nerf++: analyzing and improving neural radiance fields. arXiv:2010.07492 (2020)

  49. Zhang, X., Srinivasan, P.P., Deng, B., Debevec, P., Freeman, W.T., Barron, J.T.: Nerfactor: neural factorization of shape and reflectance under an unknown illumination. ACM Trans. Graph 40(6), 1–18 (2021)

    Article  Google Scholar 

Download references

Acknowledgements

We thank Christen Millerdurai for the help with the dataset recording. This work was supported by the ERC Consolidator Grant 4DRepLy (770784).

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Viktor Rudnev .

Editor information

Editors and Affiliations

1 Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (pdf 7720 KB)

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Rudnev, V., Elgharib, M., Smith, W., Liu, L., Golyanik, V., Theobalt, C. (2022). NeRF for Outdoor Scene Relighting. In: Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T. (eds) Computer Vision – ECCV 2022. ECCV 2022. Lecture Notes in Computer Science, vol 13676. Springer, Cham. https://doi.org/10.1007/978-3-031-19787-1_35

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-19787-1_35

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-19786-4

  • Online ISBN: 978-3-031-19787-1

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics