Advertisement

Single-Shot Neural Relighting and SVBRDF Estimation

Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 12364)

Abstract

We present a novel physically-motivated deep network for joint shape and material estimation, as well as relighting under novel illumination conditions, using a single image captured by a mobile phone camera. Our physically-based modeling leverages a deep cascaded architecture trained on a large-scale synthetic dataset that consists of complex shapes with microfacet SVBRDF. In contrast to prior works that train rendering layers subsequent to inverse rendering, we propose deep feature sharing and joint training that transfer insights across both tasks, to achieve significant improvements in both reconstruction and relighting. We demonstrate in extensive qualitative and quantitative experiments that our network generalizes very well to real images, achieving high-quality shape and material estimation, as well as image-based relighting. Code, models and data will be publicly released.

Keywords

Single-image relighting SVBRDF estimation Physically-based networks 

Notes

Acknowledgments

This work was supported by NSF CAREER 1751365, along with generous gifts from a Google Research Award and Adobe Research. This work was done during Shen Sang’s graduate studies at UC San Diego.

Supplementary material

504475_1_En_6_MOESM1_ESM.pdf (41.5 mb)
Supplementary material 1 (pdf 42496 KB)

Supplementary material 2 (mp4 2404 KB)

References

  1. 1.
    Aittala, M., Weyrich, T., Lehtinen, J., et al.: Two-shot SVBRDF capture for stationary materials. ACM Trans. Graph. 34(4), 110-1 (2015)CrossRefGoogle Scholar
  2. 2.
    Bansal, A., Russell, B., Gupta, A.: Marr revisited: 2D–3D alignment via surface normal prediction. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5965–5974 (2016)Google Scholar
  3. 3.
    Barron, J.T., Malik, J.: Shape, illumination, and reflectance from shading. IEEE Trans. Pattern Anal. Mach. Intell. 37(8), 1670–1687 (2014)CrossRefGoogle Scholar
  4. 4.
    Bell, S., Upchurch, P., Snavely, N., Bala, K.: Material recognition in the wild with the materials in context database. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3479–3487 (2015)Google Scholar
  5. 5.
    Cai, Z., Vasconcelos, N.: Cascade R-CNN: delving into high quality object detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6154–6162 (2018)Google Scholar
  6. 6.
    Debevec, P., Hawkins, T., Tchou, C., Duiker, H.P., Sarokin, W., Sagar, M.: Acquiring the reflectance field of a human face. In: Proceedings of the 27th Annual Conference on Computer Graphics and Interactive Techniques, pp. 145–156. ACM Press/Addison-Wesley Publishing Co. (2000)Google Scholar
  7. 7.
    Deschaintre, V., Aittala, M., Durand, F., Drettakis, G., Bousseau, A.: Single-image SVBRDF capture with a rendering-aware deep network. ACM Trans. Graph. (TOG) 37(4), 128 (2018)CrossRefGoogle Scholar
  8. 8.
    Eigen, D., Fergus, R.: Predicting depth, surface normals and semantic labels with a common multi-scale convolutional architecture. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2650–2658 (2015)Google Scholar
  9. 9.
    Gardner, M.A., et al.: Learning to predict indoor illumination from a single image. arXiv preprint arXiv:1704.00090 (2017)
  10. 10.
    Georgoulis, S., Rematas, K., Ritschel, T., Fritz, M., Tuytelaars, T., Van Gool, L.: What is around the camera? In: Proceedings of the IEEE International Conference on Computer Vision, pp. 5170–5178 (2017)Google Scholar
  11. 11.
    Hold-Geoffroy, Y., Sunkavalli, K., Hadap, S., Gambaretto, E., Lalonde, J.F.: Deep outdoor illumination estimation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7312–7321 (2017)Google Scholar
  12. 12.
    Hui, Z., Sankaranarayanan, A.C.: Shape and spatially-varying reflectance estimation from virtual exemplars. IEEE Trans. Pattern Anal. Mach. Intell. 39(10), 2060–2073 (2016)CrossRefGoogle Scholar
  13. 13.
    Isola, P., Zhu, J.Y., Zhou, T., Efros, A.A.: Image-to-image translation with conditional adversarial networks. arXiv (2016)Google Scholar
  14. 14.
    Johnson, M.K., Adelson, E.H.: Shape estimation in natural illumination. In: CVPR 2011, pp. 2553–2560. IEEE (2011)Google Scholar
  15. 15.
    Karis, B., Games, E.: Real shading in unreal engine 4. In: Proceedings of the Physically Based Shading Theory Practice, vol. 4 (2013)Google Scholar
  16. 16.
    Li, H., Lin, Z., Shen, X., Brandt, J., Hua, G.: A convolutional neural network cascade for face detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5325–5334 (2015)Google Scholar
  17. 17.
    Li, X., Dong, Y., Peers, P., Tong, X.: Modeling surface appearance from a single photograph using self-augmented convolutional neural networks. ACM Trans. Graph. (TOG) 36(4), 45 (2017)Google Scholar
  18. 18.
    Li, Z., Sunkavalli, K., Chandraker, M.: Materials for masses: SVBRDF acquisition with a single mobile phone image. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11207, pp. 74–90. Springer, Cham (2018).  https://doi.org/10.1007/978-3-030-01219-9_5CrossRefGoogle Scholar
  19. 19.
    Li, Z., Xu, Z., Ramamoorthi, R., Sunkavalli, K., Chandraker, M.: Learning to reconstruct shape and spatially-varying reflectance from a single image. In: SIGGRAPH Asia 2018 Technical Papers, p. 269. ACM (2018)Google Scholar
  20. 20.
    Matusik, W., Loper, M., Pfister, H.: Progressively-refined reflectance functions from natural illumination. In: Rendering Techniques, pp. 299–308 (2004)Google Scholar
  21. 21.
    Meka, A., et al.: Lime: live intrinsic material estimation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6315–6324 (2018)Google Scholar
  22. 22.
    Newell, A., Yang, K., Deng, J.: Stacked hourglass networks for human pose estimation. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9912, pp. 483–499. Springer, Cham (2016).  https://doi.org/10.1007/978-3-319-46484-8_29CrossRefGoogle Scholar
  23. 23.
    Oxholm, G., Nishino, K.: Shape and reflectance estimation in the wild. IEEE Trans. Pattern Anal. Mach. Intell. 38(2), 376–389 (2015)CrossRefGoogle Scholar
  24. 24.
    Paszke, A., et al.: Automatic differentiation in PyTorch (2017) Google Scholar
  25. 25.
    Peers, P., Dutré, P.: Inferring reflectance functions from wavelet noise. In: Proceedings of the Sixteenth Eurographics conference on Rendering Techniques, pp. 173–182. Eurographics Association (2005)Google Scholar
  26. 26.
    Peers, P., et al.: Compressive light transport sensing. ACM Trans. Graph. (TOG) 28(1), 3 (2009)CrossRefGoogle Scholar
  27. 27.
    Rematas, K., Ritschel, T., Fritz, M., Gavves, E., Tuytelaars, T.: Deep reflectance maps. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4508–4516 (2016)Google Scholar
  28. 28.
    Ren, P., Dong, Y., Lin, S., Tong, X., Guo, B.: Image based relighting using neural networks. ACM Trans. Graph. (TOG) 34(4), 111 (2015)CrossRefGoogle Scholar
  29. 29.
    Riviere, J., Peers, P., Ghosh, A.: Mobile surface reflectometry. In: Computer Graphics Forum, vol. 35, pp. 191–202. Wiley Online Library (2016)Google Scholar
  30. 30.
    Ronneberger, O., Fischer, P., Brox, T.: U-Net: convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9351, pp. 234–241. Springer, Cham (2015).  https://doi.org/10.1007/978-3-319-24574-4_28CrossRefGoogle Scholar
  31. 31.
    Shi, B., Wu, Z., Mo, Z., Duan, D., Yeung, S.K., Tan, P.: A benchmark dataset and evaluation for non-Lambertian and uncalibrated photometric stereo. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3707–3716 (2016)Google Scholar
  32. 32.
    Shi, J., Dong, Y., Su, H., Yu, S.X.: Learning non-Lambertian object intrinsics across ShapeNet categories. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1685–1694 (2017)Google Scholar
  33. 33.
    Sun, T., et al.: Single image portrait relighting. ACM Trans. Graph. 38(4), 79-1 (2019)CrossRefGoogle Scholar
  34. 34.
    Wei, S.E., Ramakrishna, V., Kanade, T., Sheikh, Y.: Convolutional pose machines. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4724–4732 (2016)Google Scholar
  35. 35.
    Xu, Z., Sunkavalli, K., Hadap, S., Ramamoorthi, R.: Deep image-based relighting from optimal sparse samples. ACM Trans. Graph. (TOG) 37(4), 126 (2018)CrossRefGoogle Scholar
  36. 36.
    Zhou, H., Hadap, S., Sunkavalli, K., Jacobs, D.W.: Deep single-image portrait relighting. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 7194–7202 (2019)Google Scholar
  37. 37.
    Zhu, J.Y., Park, T., Isola, P., Efros, A.A.: Unpaired image-to-image translation using cycle-consistent adversarial networks. In: 2017 IEEE International Conference on Computer Vision (ICCV) (2017)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  1. 1.University of California, San DiegoLa JollaUSA
  2. 2.ByteDance ResearchMountain ViewUSA

Personalised recommendations