Skip to main content
SpringerLink
Log in

Deep Shape from Polarization

  • Conference paper
  • First Online:
Computer Vision – ECCV 2020 (ECCV 2020)

Part of the book series: Lecture Notes in Computer Science ((LNIP,volume 12369))

Included in the following conference series:

Abstract

This paper makes a first attempt to bring the Shape from Polarization (SfP) problem to the realm of deep learning. The previous state-of-the-art methods for SfP have been purely physics-based. We see value in these principled models, and blend these physical models as priors into a neural network architecture. This proposed approach achieves results that exceed the previous state-of-the-art on a challenging dataset we introduce. This dataset consists of polarization images taken over a range of object textures, paints, and lighting conditions. We report that our proposed method achieves the lowest test error on each tested condition in our dataset, showing the value of blending data-driven and physics-driven approaches.

A. Gilbert and F. Wang—Equal contribution.

Project page: https://visual.ee.ucla.edu/deepsfp.htm.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    For a detailed discussion of other sources of noise please refer to Schechner [47].

  2. 2.

    The dataset is available at: https://visual.ee.ucla.edu/deepsfp.htm.

  3. 3.

    https://github.com/waps101/depth-from-polarisation.

  4. 4.

    MAE is the most commonly reported measure for surface normal reconstruction, but in many cases it is a deceptive metric. We find that a few outliers in high-frequency regions can skew the MAE for entire reconstructions. Accordingly, we emphasize the qualitative comparisons of the proposed method to its physics-based counterparts.

References

  1. Atkinson, G.A.: Polarisation photometric stereo. Comput. Vis. Image Understand. 160, 158–167 (2017)

    Article  Google Scholar 

  2. Atkinson, G.A., Ernst, J.D.: High-sensitivity analysis of polarization by surface reflection. Mach. Vis. Appl. 29, 1171–1189 (2018)

    Article  Google Scholar 

  3. Atkinson, G.A., Hancock, E.R.: Multi-view surface reconstruction using polarization. In: ICCV (2005)

    Google Scholar 

  4. Atkinson, G.A., Hancock, E.R.: Recovery of surface orientation from diffuse polarization. In: IEEE TIP (2006)

    Google Scholar 

  5. Baek, S.H., Jeon, D.S., Tong, X., Kim, M.H.: Simultaneous acquisition of polarimetric SVBRDF and normals. In: ACM SIGGRAPH (TOG) (2018)

    Google Scholar 

  6. Berger, K., Voorhies, R., Matthies, L.H.: Depth from stereo polarization in specular scenes for urban robotics. In: ICRA (2017)

    Google Scholar 

  7. Chen, G., Han, K., Wong, K.-Y.K.: PS-FCN: a flexible learning framework for photometric stereo. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11213, pp. 3–19. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01240-3_1

    Chapter  Google Scholar 

  8. Chen, L., Zheng, Y., Subpa-asa, A., Sato, I.: Polarimetric three-view geometry. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11220, pp. 21–37. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01270-0_2

    Chapter  Google Scholar 

  9. Cui, Z., Gu, J., Shi, B., Tan, P., Kautz, J.: Polarimetric multi-view stereo. In: CVPR (2017)

    Google Scholar 

  10. Deschaintre, V., Aittala, M., Durand, F., Drettakis, G., Bousseau, A.: Single-image SVBRDF capture with a rendering-aware deep network. In: ACM SIGGRAPH (TOG) (2018)

    Google Scholar 

  11. Drbohlav, O., Sara, R.: Unambiguous determination of shape from photometric stereo with unknown light sources. In: ICCV (2001)

    Google Scholar 

  12. Ghosh, A., Chen, T., Peers, P., Wilson, C.A., Debevec, P.: Circularly polarized spherical illumination reflectometry. In: ACM SIGGRAPH (TOG) (2010)

    Google Scholar 

  13. Ghosh, A., Fyffe, G., Tunwattanapong, B., Busch, J., Yu, X., Debevec, P.: Multiview face capture using polarized spherical gradient illumination. In: ACM SIGGRAPH (TOG) (2011)

    Google Scholar 

  14. Guarnera, G.C., Peers, P., Debevec, P., Ghosh, A.: Estimating surface normals from spherical stokes reflectance fields. In: Fusiello, A., Murino, V., Cucchiara, R. (eds.) ECCV 2012. LNCS, vol. 7584, pp. 340–349. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-33868-7_34

    Chapter  Google Scholar 

  15. Huang, G., Sun, Y., Liu, Z., Sedra, D., Weinberger, K.: Deep networks with stochastic depth. CoRR (2016)

    Google Scholar 

  16. Huynh, C.P., Robles-Kelly, A., Hancock, E.R.: Shape and refractive index recovery from single-view polarisation images. In: CVPR (2010)

    Google Scholar 

  17. Huynh, C.P., Robles-Kelly, A., Hancock, E.R.: Shape and refractive index from single-view spectro-polarimetric images. IJCV 101, 64–94 (2013). https://doi.org/10.1007/s11263-012-0546-3

    Article  MathSciNet  Google Scholar 

  18. Ikehata, S.: CNN-PS: CNN-based photometric stereo for general non-convex surfaces. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11219, pp. 3–19. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01267-0_1

    Chapter  Google Scholar 

  19. Ioffe, S., Szegedy, C.: Batch normalization: accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167 (2015)

  20. Jakob, W.: Mitsuba renderer (2010). http://www.mitsuba-renderer.org

  21. Kadambi, A., Taamazyan, V., Shi, B., Raskar, R.: Polarized 3D: high-quality depth sensing with polarization cues. In: ICCV (2015)

    Google Scholar 

  22. Kadambi, A., Taamazyan, V., Shi, B., Raskar, R.: Depth sensing using geometrically constrained polarization normals. IJCV 125, 34–51 (2017). https://doi.org/10.1007/s11263-017-1025-7

    Article  MathSciNet  Google Scholar 

  23. Karpatne, A., Watkins, W., Read, J., Kumar, V.: Physics-guided neural networks (PGNN): an application in lake temperature modeling. CoRR (2017)

    Google Scholar 

  24. Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)

  25. Li, X., Dong, Y., Peers, P., Tong, X.: Modeling surface appearance from a single photograph using self-augmented convolutional neural networks. In: ACM SIGGRAPH (TOG) (2017)

    Google Scholar 

  26. Li, Z., Sunkavalli, K., Chandraker, M.: Materials for masses: SVBRDF acquisition with a single mobile phone image. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11207, pp. 74–90. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01219-9_5

    Chapter  Google Scholar 

  27. Li, Z., Xu, Z., Ramamoorthi, R., Sunkavalli, K., Chandraker, M.: Learning to reconstruct shape and spatially-varying reflectance from a single image. In: ACM SIGGRAPH Asia (TOG) (2018)

    Google Scholar 

  28. Lindell, D.B., O’Toole, M., Wetzstein, G.: Single-photon 3D imaging with deep sensor fusion. In: ACM SIGGRAPH (TOG) (2018)

    Google Scholar 

  29. Lucid Vision Phoenix polarization camera (2018). https://thinklucid.com/product/phoenix-5-0-mp-polarized-model/

  30. Lyu, Y., Cui, Z., Li, S., Pollefeys, M., Shi, B.: Reflection separation using a pair of unpolarized and polarized images. In: Advances in Neural Information Processing Systems, pp. 14559–14569 (2019)

    Google Scholar 

  31. Ma, W.C., Hawkins, T., Peers, P., Chabert, C.F., Weiss, M., Debevec, P.: Rapid acquisition of specular and diffuse normal maps from polarized spherical gradient illumination. In: Eurographics Conference on Rendering Techniques (2007)

    Google Scholar 

  32. Maeda, T., Kadambi, A., Schechner, Y.Y., Raskar, R.: Dynamic heterodyne interferometry. In: ICCP (2018)

    Google Scholar 

  33. Mahmoud, A.H., El-Melegy, M.T., Farag, A.A.: Direct method for shape recovery from polarization and shading. In: ICIP (2012)

    Google Scholar 

  34. Marco, J., et al.: Deeptof: off-the-shelf real-time correction of multipath interference in time-of-flight imaging. In: ACM SIGGRAPH (TOG) (2017)

    Google Scholar 

  35. Miyazaki, D., Kagesawa, M., Ikeuchi, K.: Transparent surface modeling from a pair of polarization images. In: PAMI (2004)

    Google Scholar 

  36. Miyazaki, D., Shigetomi, T., Baba, M., Furukawa, R., Hiura, S., Asada, N.: Surface normal estimation of black specular objects from multiview polarization images. Int. Soc. Opt. Photon. Opt. Eng. (2016)

    Google Scholar 

  37. Miyazaki, D., Tan, R.T., Hara, K., Ikeuchi, K.: Polarization-based inverse rendering from a single view. In: ICCV (2003)

    Google Scholar 

  38. Mo, Z., Shi, B., Lu, F., Yeung, S.K., Matsushita, Y.: Uncalibrated photometric stereo under natural illumination. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2936–2945 (2018)

    Google Scholar 

  39. Ngo, T.T., Nagahara, H., Taniguchi, R.: Shape and light directions from shading and polarization. In: CVPR (2015)

    Google Scholar 

  40. Park, T., Liu, M.Y., Wang, T.C., Zhu, J.Y.: Semantic image synthesis with spatially-adaptive normalization. In: CVPR (2019)

    Google Scholar 

  41. Paszke, A., et al.: Automatic differentiation in pytorch. In: NIPS-W (2017)

    Google Scholar 

  42. PolarM polarization camera (2017). http://www.4dtechnology.com/products/polarimeters/polarcam/

  43. Riviere, J., Reshetouski, I., Filipi, L., Ghosh, A.: Polarization imaging reflectometry in the wild. In: ACM SIGGRAPH (TOG) (2017)

    Google Scholar 

  44. Ronneberger, O., Fischer, P., Brox, T.: U-Net: convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9351, pp. 234–241. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-24574-4_28

    Chapter  Google Scholar 

  45. Santo, H., Samejima, M., Sugano, Y., Shi, B., Matsushita, Y.: Deep photometric stereo network. In: ICCV Workshops (2017)

    Google Scholar 

  46. Satat, G., Tancik, M., Gupta, O., Heshmat, B., Raskar, R.: Object classification through scattering media with deep learning on time resolved measurement. OSA Opt. Exp. 25, 17466–17479 (2017)

    Article  Google Scholar 

  47. Schechner, Y.Y.: Self-calibrating imaging polarimetry. In: ICCP (2015)

    Google Scholar 

  48. Sengupta, S., Kanazawa, A., Castillo, C.D., Jacobs, D.W.: SfSnet: learning shape, reflectance and illuminance of faces in the wild. In: CVPR (2018)

    Google Scholar 

  49. Shi, B., Mo, Z., Wu, Z., Duan, D., Yeung, S.K., Tan, P.: A benchmark dataset and evaluation for non-Lambertian and uncalibrated photometric stereo. In: PAMI (2019)

    Google Scholar 

  50. SHINING 3D scanner (2018). https://www.einscan.com/einscan-se-sp

  51. Smith, W.A.P., Ramamoorthi, R., Tozza, S.: Linear depth estimation from an uncalibrated, monocular polarisation image. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9912, pp. 109–125. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46484-8_7

    Chapter  Google Scholar 

  52. Smith, W.A.P., Ramamoorthi, R., Tozza, S.: Height-from-polarisation with unknown lighting or albedo. In: PAMI (2018)

    Google Scholar 

  53. Srivastava, R.K., Greff, K., Schmidhuber, J.: Highway networks. CoRR (2015)

    Google Scholar 

  54. Su, S., Heide, F., Wetzstein, G., Heidrich, W.: Deep end-to-end time-of-flight imaging. In: CVPR (2018)

    Google Scholar 

  55. Tancik, M., Satat, G., Raskar, R.: Flash photography for data-driven hidden scene recovery. arXiv preprint arXiv:1810.11710 (2018)

  56. Tancik, M., Swedish, T., Satat, G., Raskar, R.: Data-driven non-line-of-sight imaging with a traditional camera. In: OSA Imaging and Applied Optics (2018)

    Google Scholar 

  57. Taniai, T., Maehara, T.: Neural inverse rendering for general reflectance photometric stereo. In: ICML (2018)

    Google Scholar 

  58. Teo, D., Shi, B., Zheng, Y., Yeung, S.K.: Self-calibrating polarising radiometric calibration. In: CVPR (2018)

    Google Scholar 

  59. Tozza, S., Smith, W.A.P., Zhu, D., Ramamoorthi, R., Hancock, E.R.: Linear differential constraints for photo-polarimetric height estimation. In: ICCV (2017)

    Google Scholar 

  60. Wolff, L.B.: Polarization vision: a new sensory approach to image understanding. Image Vis. Comput. 15, 81–93 (1997)

    Article  Google Scholar 

  61. Xiong, Y., Chakrabarti, A., Basri, R., Gortler, S.J., Jacobs, D.W., Zickler, T.: From shading to local shape. IEEE Trans. Pattern Anal. Mach. Intell. 37(1), 67–79 (2014)

    Article  Google Scholar 

  62. Yang, L., Tan, F., Li, A., Cui, Z., Furukawa, Y., Tan, P.: Polarimetric dense monocular SLAM. In: CVPR (2018)

    Google Scholar 

  63. Ye, W., Li, X., Dong, Y., Peers, P., Tong, X.: Single image surface appearance modeling with self-augmented CNNs and inexact supervision. Wiley Online Library Computer Graphics Forum (2018)

    Google Scholar 

  64. Zhu, D., Smith, W.A.P.: Depth from a polarisation + RGB stereo pair. In: CVPR (2019)

    Google Scholar 

Download references

Acknowledgements

The work of UCLA authors was supported by a Sony Imaging Young Faculty Award, Google Faculty Award, and the NSF CRII Research Initiation Award (IIS 1849941). The work of Peking University authors was supported by National Natural Science Foundation of China (61872012, 41842048, 41571432), National Key R&D Program of China (2019YFF0302902, 2017YFB0503004), Beijing Academy of Artificial Intelligence (BAAI), and Education Department Project of Guizhou Province.

Author information

Authors and Affiliations

  1. University of California, Los Angeles, USA

    Yunhao Ba, Alex Gilbert, Franklin Wang, Yiqin Wang & Achuta Kadambi

  2. Peking University, Beijing, China

    Jinfa Yang, Rui Chen, Lei Yan & Boxin Shi

Authors
  1. Yunhao Ba
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Alex Gilbert
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Franklin Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jinfa Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Rui Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Yiqin Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Lei Yan
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Boxin Shi
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Achuta Kadambi
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Boxin Shi or Achuta Kadambi .

Editor information

Editors and Affiliations

  1. University of Oxford, Oxford, UK

    Andrea Vedaldi

  2. Graz University of Technology, Graz, Austria

    Horst Bischof

  3. University of Freiburg, Freiburg im Breisgau, Germany

    Thomas Brox

  4. University of North Carolina at Chapel Hill, Chapel Hill, NC, USA

    Jan-Michael Frahm

1 Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (pdf 8165 KB)

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Ba, Y. et al. (2020). Deep Shape from Polarization. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, JM. (eds) Computer Vision – ECCV 2020. ECCV 2020. Lecture Notes in Computer Science(), vol 12369. Springer, Cham. https://doi.org/10.1007/978-3-030-58586-0_33

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-58586-0_33

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-58585-3

  • Online ISBN: 978-3-030-58586-0

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation