Skip to main content
SpringerLink
Log in

PS-NeRF: Neural Inverse Rendering for Multi-view Photometric Stereo

  • Conference paper
  • First Online:
Computer Vision – ECCV 2022 (ECCV 2022)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13661))

Included in the following conference series:

Abstract

Traditional multi-view photometric stereo (MVPS) methods are often composed of multiple disjoint stages, resulting in noticeable accumulated errors. In this paper, we present a neural inverse rendering method for MVPS based on implicit representation. Given multi-view images of a non-Lambertian object illuminated by multiple unknown directional lights, our method jointly estimates the geometry, materials, and lights. Our method first employs multi-light images to estimate per-view surface normal maps, which are used to regularize the normals derived from the neural radiance field. It then jointly optimizes the surface normals, spatially-varying BRDFs, and lights based on a shadow-aware differentiable rendering layer. After optimization, the reconstructed object can be used for novel-view rendering, relighting, and material editing. Experiments on both synthetic and real datasets demonstrate that our method achieves far more accurate shape reconstruction than existing MVPS and neural rendering methods. Our code and model can be found at https://ywq.github.io/psnerf.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 89.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 119.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    i.e., multiple images are captured for each view, where each image is illuminated by a single unknown directional light.

  2. 2.

    We rescale the meshes into the range of \([-1,1]\) for all the objects, and show Chamfer distance in the unit of mm).

References

  1. Agarwal, S., et al.: Building Rome in a day. Communications of the ACM (2011)

    Google Scholar 

  2. Barron, J.T., Mildenhall, B., Tancik, M., Hedman, P., Martin-Brualla, R., Srinivasan, P.P.: Mip-NeRF: a multiscale representation for anti-aliasing neural radiance fields. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) (2021)

    Google Scholar 

  3. Bi, S., et al.: Neural reflectance fields for appearance acquisition. arXiv preprint arXiv:2008.03824 (2020)

  4. Boss, M., Braun, R., Jampani, V., Barron, J.T., Liu, C., Lensch, H.: NeRD: Neural reflectance decomposition from image collections. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 12684–12694 (2021)

    Google Scholar 

  5. Boss, M., Jampani, V., Braun, R., Liu, C., Barron, J., Lensch, H.: Neural-PIL: Neural pre-integrated lighting for reflectance decomposition. In: Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), vol. 34 (2021)

    Google Scholar 

  6. Chen, G., Han, K., Shi, B., Matsushita, Y., Wong, K.Y.K.: Self-calibrating deep photometric stereo networks. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8739–8747 (2019)

    Google Scholar 

  7. Chen, G., Han, K., Shi, B., Matsushita, Y., Wong, K.Y.K.: Deep photometric stereo for non-Lambertian surfaces. IEEE Trans. Pattern Anal. Mach. Intell. (T-PAMI) 44(1), 129–142 (2020)

    Google Scholar 

  8. Chen, G., Han, K., Wong, K.Y.K.: PS-FCN: A flexible learning framework for photometric stereo. In: Proceedings of the European Conference on Computer Vision (ECCV). pp. 3–18 (2018)

    Google Scholar 

  9. Cheng, Z., Li, H., Asano, Y., Zheng, Y., Sato, I.: Multi-view 3d reconstruction of a texture-less smooth surface of unknown generic reflectance. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 16226–16235 (2021)

    Google Scholar 

  10. Chung, H.S., Jia, J.: Efficient photometric stereo on glossy surfaces with wide specular lobes. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2008)

    Google Scholar 

  11. Esteban, C.H., Vogiatzis, G., Cipolla, R.: Multiview photometric stereo. IEEE Transactions on Pattern Analysis and Machine Intelligence (T-PAMI) (2008)

    Google Scholar 

  12. Furukawa, Y., Ponce, J.: Accurate, dense, and robust multiview stereopsis. IEEE Trans. Pattern Anal. Mach. Intell. (T-PAMI) 32(8), 1362–1376 (2009)

    Google Scholar 

  13. Garbin, S.J., Kowalski, M., Johnson, M., Shotton, J., Valentin, J.: FastNeRF: High-fidelity neural rendering at 200fps. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 14346–14355 (2021)

    Google Scholar 

  14. Hayakawa, H.: Photometric stereo under a light source with arbitrary motion. JOSA A (1994)

    Google Scholar 

  15. Hertzmann, A., Seitz, S.M.: Example-based photometric stereo: Shape reconstruction with general, varying BRDFs. IEEE Trans. Pattern Anal. Mach. Intell. (T-PAMI) (2005)

    Google Scholar 

  16. Hui, Z., Sankaranarayanan, A.C.: Shape and spatially-varying reflectance estimation from virtual exemplars. IEEE Trans. Pattern Anal. Mach. Intell. (T-PAMI) (2017)

    Google Scholar 

  17. Ikehata, S.: CNN-PS: CNN-based photometric stereo for general non-convex surfaces. In: Proceedings of the European Conference on Computer Vision (ECCV) (2018)

    Google Scholar 

  18. Ikehata, S., Aizawa, K.: Photometric stereo using constrained bivariate regression for general isotropic surfaces. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2014)

    Google Scholar 

  19. Jakob, W.: Mitsuba renderer (2010)

    Google Scholar 

  20. Kajiya, J.T.: The rendering equation. In: Proceedings of the 13th Annual Conference on Computer Graphics and Interactive Techniques, pp. 143–150 (1986)

    Google Scholar 

  21. Kaya, B., Kumar, S., Oliveira, C., Ferrari, V., Van Gool, L.: Uncertainty-aware deep multi-view photometric stereo. arXiv preprint arXiv:2010.07492 (2020)

  22. Kaya, B., Kumar, S., Oliveira, C., Ferrari, V., Van Gool, L.: Uncalibrated neural inverse rendering for photometric stereo of general surfaces. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3804–3814 (2021)

    Google Scholar 

  23. Kaya, B., Kumar, S., Sarno, F., Ferrari, V., Van Gool, L.: Neural radiance fields approach to deep multi-view photometric stereo. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pp. 1965–1977 (2022)

    Google Scholar 

  24. Li, J., Li, H.: Neural reflectance for shape recovery with shadow handling. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2022)

    Google Scholar 

  25. Li, J., Robles-Kelly, A., You, S., Matsushita, Y.: Learning to minify photometric stereo. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2019)

    Google Scholar 

  26. Li, M., Zhou, Z., Wu, Z., Shi, B., Diao, C., Tan, P.: Multi-view photometric stereo: a robust solution and benchmark dataset for spatially varying isotropic materials. IEEE Trans. Image Process. (TIP) (2020)

    Google Scholar 

  27. Li, Z., Sunkavalli, K., Chandraker, M.: Materials for masses: Svbrdf acquisition with a single mobile phone image. In: Proceedings of the European Conference on Computer Vision (ECCV). pp. 72–87 (2018)

    Google Scholar 

  28. Li, Z., Xu, Z., Ramamoorthi, R., Sunkavalli, K., Chandraker, M.: Learning to reconstruct shape and spatially-varying reflectance from a single image. ACM Trans. Graph. (TOG) 37(6), 1–11 (2018)

    Article  Google Scholar 

  29. Lim, J., Ho, J., Yang, M.H., Kriegman, D.: Passive photometric stereo from motion. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 1635–1642 (2005)

    Google Scholar 

  30. Liu, L., Gu, J., Lin, K.Z., Chua, T.S., Theobalt, C.: Neural sparse voxel fields. In: Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), vol. 33, pp. 15651–15663 (2020)

    Google Scholar 

  31. Logothetis, F., Budvytis, I., Mecca, R., Cipolla, R.: Px-net: simple and efficient pixel-wise training of photometric stereo networks. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 12757–12766 (2021)

    Google Scholar 

  32. Logothetis, F., Mecca, R., Cipolla, R.: A differential volumetric approach to multi-view photometric stereo. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 1052–1061 (2019)

    Google Scholar 

  33. Lu, F., Chen, X., Sato, I., Sato, Y.: SymPS: BRDF symmetry guided photometric stereo for shape and light source estimation. IEEE Trans. Pattern Anal. Mach. Intell. (T-PAMI) 40, 221–234 (2018)

    Google Scholar 

  34. Martin-Brualla, R., Radwan, N., Sajjadi, M.S., Barron, J.T., Dosovitskiy, A., Duckworth, D.: NeRF in the wild: Neural radiance fields for unconstrained photo collections. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7210–7219 (2021)

    Google Scholar 

  35. Mescheder, L., Oechsle, M., Niemeyer, M., Nowozin, S., Geiger, A.: Occupancy networks: learning 3d reconstruction in function space. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4460–4470 (2019)

    Google Scholar 

  36. Mildenhall, B., Srinivasan, P.P., Tancik, M., Barron, J.T., Ramamoorthi, R., Ng, R.: NeRF: representing scenes as neural radiance fields for view synthesis. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 405–421 (2020)

    Google Scholar 

  37. Mukaigawa, Y., Ishii, Y., Shakunaga, T.: Analysis of photometric factors based on photometric linearization. JOSA A (2007)

    Google Scholar 

  38. Nam, G., Lee, J.H., Gutierrez, D., Kim, M.H.: Practical svbrdf acquisition of 3d objects with unstructured flash photography. ACM Trans. Graph. (TOG) 37(6), 1–12 (2018)

    Article  Google Scholar 

  39. Niemeyer, M., Mescheder, L., Oechsle, M., Geiger, A.: Differentiable volumetric rendering: learning implicit 3d representations without 3d supervision. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3504–3515 (2020)

    Google Scholar 

  40. Oechsle, M., Peng, S., Geiger, A.: UNISURF: Unifying neural implicit surfaces and radiance fields for multi-view reconstruction. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 5589–5599 (2021)

    Google Scholar 

  41. Park, J., Sinha, S.N., Matsushita, Y., Tai, Y.W., Kweon, I.S.: Robust multiview photometric stereo using planar mesh parameterization. IEEE Trans. Pattern Anal. Mach. Intell. (T-PAMI) 39, 1591–1604 (2016)

    Google Scholar 

  42. Reiser, C., Peng, S., Liao, Y., Geiger, A.: KiloNeRF: Speeding up neural radiance fields with thousands of tiny mlps. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). pp. 14335–14345 (2021)

    Google Scholar 

  43. Rusinkiewicz, S.M.: A new change of variables for efficient BRDF representation. In: Drettakis, G., Max, N. (eds.) EGSR 1998. E, pp. 11–22. Springer, Vienna (1998). https://doi.org/10.1007/978-3-7091-6453-2_2

    Chapter  Google Scholar 

  44. Santo, H., Samejima, M., Sugano, Y., Shi, B., Matsushita, Y.: Deep photometric stereo network. In: Proceedings of the IEEE International Conference on Computer Vision Workshops (ICCVW) (2017)

    Google Scholar 

  45. Seitz, S.M., Curless, B., Diebel, J., Scharstein, D., Szeliski, R.: A comparison and evaluation of multi-view stereo reconstruction algorithms. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2006)

    Google Scholar 

  46. Shi, B., Mo, Z., Wu, Z., Duan, D., Yeung, S.K., Tan, P.: A benchmark dataset and evaluation for non-Lambertian and uncalibrated photometric stereo. IEEE Trans. Pattern Anal. Mach. Intell. (T-PAMI) (2019)

    Google Scholar 

  47. Sitzmann, V., Zollhöfer, M., Wetzstein, G.: Scene representation networks: continuous 3d-structure-aware neural scene representations. In: Proceedings of the Advances in Neural Information Processing Systems (NeurIPS) 32 (2019)

    Google Scholar 

  48. Srinivasan, P.P., Deng, B., Zhang, X., Tancik, M., Mildenhall, B., Barron, J.T.: NeRV: Neural reflectance and visibility fields for relighting and view synthesis. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7495–7504 (2021)

    Google Scholar 

  49. Taniai, T., Maehara, T.: Neural inverse rendering for general reflectance photometric stereo. In: Proceedings of the ACM International Conference on Machine Learning (ICML) (2018)

    Google Scholar 

  50. Tewari, A., et al.: Advances in neural rendering. arXiv preprint arXiv:2111.05849 (2021)

  51. Tozza, S., Mecca, R., Duocastella, M., Del Bue, A.: Direct differential photometric stereo shape recovery of diffuse and specular surfaces. Journal of Mathematical Imaging and Vision (2016)

    Google Scholar 

  52. Wang, P., Liu, L., Liu, Y., Theobalt, C., Komura, T., Wang, W.: NeuS: learning neural implicit surfaces by volume rendering for multi-view reconstruction. In: Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), vol. 34 (2021)

    Google Scholar 

  53. Wang, X., Jian, Z., Ren, M.: Non-Lambertian photometric stereo network based on inverse reflectance model with collocated light. IEEE Trans. Image Process. (TIP) 29, 6032–6042 (2020)

    Article  Google Scholar 

  54. Wang, Z., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P.: Image quality assessment: From error visibility to structural similarity. IEEE Trans. Image Process. (TIP) (2004)

    Google Scholar 

  55. Wood, D.N., et al.: Surface light fields for 3d photography. In: Proceedings of the ACM SIGGRAPH Conference and Exhibition on Computer Graphics and Interactive Techniques in Asia (SIGGRAPH Aisa), pp. 287–296 (2000)

    Google Scholar 

  56. Woodham, R.J.: Photometric method for determining surface orientation from multiple images. Optical Engineering (1980)

    Google Scholar 

  57. Wu, C., Liu, Y., Dai, Q., Wilburn, B.: Fusing multiview and photometric stereo for 3d reconstruction under uncalibrated illumination. IEEE Transactions on Visualization and Computer Graphics (TVCG) 17(8), 1082–1095 (2010)

    Google Scholar 

  58. Wu, L., Ganesh, A., Shi, B., Matsushita, Y., Wang, Y., Ma, Y.: Robust photometric stereo via low-rank matrix completion and recovery. In: Proceedings of the Asian Conference on Computer Vision (ACCV) (2010)

    Google Scholar 

  59. Wu, T.P., Tang, C.K.: Photometric stereo via expectation maximization. IEEE Transactions on Pattern Analysis and Machine Intelligence (T-PAMI) (2010)

    Google Scholar 

  60. Yariv, L., Kasten, Y., Moran, D., Galun, M., Atzmon, M., Basri, R., Lipman, Y.: Multiview neural surface reconstruction by disentangling geometry and appearance. In: Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), vol. 33, pp. 2492–2502 (2020)

    Google Scholar 

  61. Zhang, K., Luan, F., Wang, Q., Bala, K., Snavely, N.: PhySG: Inverse rendering with spherical gaussians for physics-based material editing and relighting. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5453–5462 (2021)

    Google Scholar 

  62. Zhang, K., Riegler, G., Snavely, N., Koltun, V.: NeRF++: Analyzing and improving neural radiance fields. arXiv preprint arXiv:2010.07492 (2020)

  63. Zhang, R., Isola, P., Efros, A.A., Shechtman, E., Wang, O.: The unreasonable effectiveness of deep features as a perceptual metric. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 586–595 (2018)

    Google Scholar 

  64. Zhang, X., Srinivasan, P.P., Deng, B., Debevec, P., Freeman, W.T., Barron, J.T.: NeRFactor: Neural factorization of shape and reflectance under an unknown illumination. ACM Trans. Graph. (TOG) 40(6) (2021)

    Google Scholar 

  65. Zheng, Q., Jia, Y., Shi, B., Jiang, X., Duan, L.Y., Kot, A.C.: SPLINE-Net: Sparse photometric stereo through lighting interpolation and normal estimation networks. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) (2019)

    Google Scholar 

  66. Zhou, Z., Wu, Z., Tan, P.: Multi-view photometric stereo with spatially varying isotropic materials. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1482–1489 (2013)

    Google Scholar 

Download references

Acknowledgements

This work was partially supported by the National Key R &D Program of China (No. 2018YFB1800800), the Basic Research Project No. HZQB- KCZYZ-2021067 of Hetao Shenzhen-HK S &T Cooperation Zone, NSFC-62202409, and Hong Kong RGC GRF grant (project# 17203119).

Author information

Authors and Affiliations

  1. The University of Hong Kong, Pok Fu Lam, Hong Kong

    Wenqi Yang & Kwan-Yee K. Wong

  2. FNii and SSE, CUHK-Shenzhen, Shenzhen, China

    Guanying Chen

  3. Nanyang Technological University, Singapore, Singapore

    Chaofeng Chen

  4. MIT-IBM Watson AI Lab, Cambridge, USA

    Zhenfang Chen

Authors
  1. Wenqi Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Guanying Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Chaofeng Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Zhenfang Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Kwan-Yee K. Wong
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Guanying Chen .

Editor information

Editors and Affiliations

  1. Tel Aviv University, Tel Aviv, Israel

    Shai Avidan

  2. University College London, London, UK

    Gabriel Brostow

  3. Google AI, Accra, Ghana

    Moustapha Cissé

  4. University of Catania, Catania, Italy

    Giovanni Maria Farinella

  5. Facebook (United States), Menlo Park, CA, USA

    Tal Hassner

1 Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (pdf 10424 KB)

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Yang, W., Chen, G., Chen, C., Chen, Z., Wong, KY.K. (2022). PS-NeRF: Neural Inverse Rendering for Multi-view Photometric Stereo. In: Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T. (eds) Computer Vision – ECCV 2022. ECCV 2022. Lecture Notes in Computer Science, vol 13661. Springer, Cham. https://doi.org/10.1007/978-3-031-19769-7_16

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-19769-7_16

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-19768-0

  • Online ISBN: 978-3-031-19769-7

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation