Skip to main content
SpringerLink
Log in

Polarimetric Multi-view Inverse Rendering

  • Conference paper
  • First Online:
Computer Vision – ECCV 2020 (ECCV 2020)

Part of the book series: Lecture Notes in Computer Science ((LNIP,volume 12369))

Included in the following conference series:

Abstract

A polarization camera has great potential for 3D reconstruction since the angle of polarization (AoP) of reflected light is related to an object’s surface normal. In this paper, we propose a novel 3D reconstruction method called Polarimetric Multi-View Inverse Rendering (Polarimetric MVIR) that effectively exploits geometric, photometric, and polarimetric cues extracted from input multi-view color polarization images. We first estimate camera poses and an initial 3D model by geometric reconstruction with a standard structure-from-motion and multi-view stereo pipeline. We then refine the initial model by optimizing photometric rendering errors and polarimetric errors using multi-view RGB and AoP images, where we propose a novel polarimetric cost function that enables us to effectively constrain each estimated surface vertex’s normal while considering four possible ambiguous azimuth angles revealed from the AoP measurement. Experimental results using both synthetic and real data demonstrate that our Polarimetric MVIR can reconstruct a detailed 3D shape without assuming a specific polarized reflection depending on the material.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. https://www.jai.com/products/go-5100mp-pge

  2. https://thinklucid.com/product/phoenix-5-0-mp-polarized-model/

  3. https://github.com/cdcseacave/openMVS

  4. http://graphics.stanford.edu/data/3Dscanrep/

  5. https://www.blender.org/

  6. Aanæs, H., Jensen, R.R., Vogiatzis, G., Tola, E., Dahl, A.B.: Large-scale data for multiple-view stereopsis. Int. J. Comput. Vis. 120(2), 153–168 (2016). https://doi.org/10.1007/s11263-016-0902-9

    Article  MathSciNet  Google Scholar 

  7. Agarwal, S., Mierle, K., Others: Ceres solver. http://ceres-solver.org

  8. Agarwal, S., Snavely, N., Simon, I., Seitz, S.M., Szeliski, R.: Building Rome in a day. In: Proceedings of IEEE International Conference on Computer Vision (ICCV), pp. 72–79 (2009)

    Google Scholar 

  9. Atkinson, G.A.: Polarisation photometric stereo. Comput. Vis. Image Underst. 160, 158–167 (2017)

    Article  Google Scholar 

  10. Atkinson, G.A., Hancock, E.R.: Recovery of surface orientation from diffuse polarization. IEEE Trans. Image Process. 15(6), 1653–1664 (2006)

    Article  Google Scholar 

  11. Atkinson, G.A., Hancock, E.R.: Shape estimation using polarization and shading from two views. IEEE Trans. Pattern Anal. Mach. Intell. 29(11), 2001–2017 (2007)

    Article  Google Scholar 

  12. Baek, S.H., Jeon, D.S., Tong, X., Kim, M.H.: Simultaneous acquisition of polarimetric SVBRDF and normals. ACM Trans. Graph. 37(6), 268 (2018)

    Google Scholar 

  13. Barron, J.T., Malik, J.: Shape, illumination, and reflectance from shading. IEEE Trans. Pattern Anal. Mach. Intell. 37(8), 1670–1687 (2014)

    Article  Google Scholar 

  14. Bay, H., Ess, A., Tuytelaars, T., Van Gool, L.: Speeded-up robust features (SURF). Comput. Vis. Image Underst. 110(3), 346–359 (2008)

    Article  Google Scholar 

  15. Biehler, J., Fane, B.: 3D Printing with Autodesk: Create and Print 3D Objects with 123D. AutoCAD and Inventor. Pearson Education, London (2014)

    Google Scholar 

  16. Cao, S., Snavely, N.: Graph-based discriminative learning for location recognition. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 700–707 (2013)

    Google Scholar 

  17. Chen, L., Zheng, Y., Subpa-Asa, A., Sato, I.: Polarimetric three-view geometry. In: Proceedings of European Conference on Computer Vision (ECCV), pp. 20–36 (2018)

    Google Scholar 

  18. Cui, Z., Gu, J., Shi, B., Tan, P., Kautz, J.: Polarimetric multi-view stereo. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1558–1567 (2017)

    Google Scholar 

  19. Cui, Z., Larsson, V., Pollefeys, M.: Polarimetric relative pose estimation. In: Proceedings of IEEE International Conference on Computer Vision (ICCV), pp. 2671–2680 (2019)

    Google Scholar 

  20. Dai, A., Nießner, M.: 3DMV: Joint 3D-multi-view prediction for 3D semantic scene segmentation. In: Proceedings of European Conference on Computer Vision (ECCV), pp. 452–468 (2018)

    Google Scholar 

  21. Furukawa, Y., Curless, B., Seitz, S.M., Szeliski, R.: Towards internet-scale multi-view stereo. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1434–1441 (2010)

    Google Scholar 

  22. Furukawa, Y., Ponce, J.: Accurate, dense, and robust multiview stereopsis. IEEE Trans. Pattern Anal. Mach. Intell. 32(8), 1362–1376 (2009)

    Article  Google Scholar 

  23. Galliani, S., Lasinger, K., Schindler, K.: Massively parallel multiview stereopsis by surface normal diffusion. In: Proceedings of IEEE International Conference on Computer Vision (ICCV), pp. 873–881 (2015)

    Google Scholar 

  24. Ghosh, A., Fyffe, G., Tunwattanapong, B., Busch, J., Yu, X., Debevec, P.: Multiview face capture using polarized spherical gradient illumination. ACM Trans. Graph. 30(6), 129 (2011)

    Article  Google Scholar 

  25. Haefner, B., Ye, Z., Gao, M., Wu, T., Quéau, Y., Cremers, D.: Variational uncalibrated photometric stereo under general lighting. In: Proceedings of IEEE International Conference on Computer Vision (ICCV), pp. 8539–8548 (2019)

    Google Scholar 

  26. Huynh, C.P., Robles-Kelly, A., Hancock, E.R.: Shape and refractive index from single-view spectro-polarimetric images. Int. J. Comput. Vis. 101(1), 64–94 (2013). https://doi.org/10.1007/s11263-012-0546-3

    Article  MathSciNet  Google Scholar 

  27. Ikehata, S., Wipf, D., Matsushita, Y., Aizawa, K.: Photometric stereo using sparse Bayesian regression for general diffuse surfaces. IEEE Trans. Pattern Anal. Mach. Intell. 36(9), 1816–1831 (2014)

    Article  Google Scholar 

  28. Jancosek, M., Pajdla, T.: Multi-view reconstruction preserving weakly-supported surfaces. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3121–3128 (2011)

    Google Scholar 

  29. Kadambi, A., Taamazyan, V., Shi, B., Raskar, R.: Polarized 3D: high-quality depth sensing with polarization cues. In: Proceedings of IEEE International Conference on Computer Vision (ICCV), pp. 3370–3378 (2015)

    Google Scholar 

  30. Kazhdan, M., Hoppe, H.: Screened Poisson surface reconstruction. ACM Trans. Graph. 32(3), 1–13 (2013)

    Article  Google Scholar 

  31. Kiku, D., Monno, Y., Tanaka, M., Okutomi, M.: Beyond color difference: residual interpolation for color image demosaicking. IEEE Trans. Image Process. 25(3), 1288–1300 (2016)

    MathSciNet  MATH  Google Scholar 

  32. Kim, K., Torii, A., Okutomi, M.: Multi-view inverse rendering under arbitrary illumination and albedo. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9907, pp. 750–767. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46487-9_46

    Chapter  Google Scholar 

  33. Kobbelt, L.: \(\sqrt{3}\)-subdivision. In: Proceedings of Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH), pp. 103–112 (2000)

    Google Scholar 

  34. Ley, A., Hänsch, R., Hellwich, O.: SyB3R: a realistic synthetic benchmark for 3D reconstruction from images. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9911, pp. 236–251. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46478-7_15

    Chapter  Google Scholar 

  35. Li, M., Zhou, Z., Wu, Z., Shi, B., Diao, C., Tan, P.: Multi-view photometric stereo: a robust solution and benchmark dataset for spatially varying isotropic materials. IEEE Trans. Image Process. 29, 4159–4173 (2020)

    Article  Google Scholar 

  36. Lowe, D.G.: Distinctive image features from scale-invariant keypoints. Int. J. Comput. Vis. 60(2), 91–110 (2004). https://doi.org/10.1023/B:VISI.0000029664.99615.94

    Article  Google Scholar 

  37. Mahmoud, A.H., El-Melegy, M.T., Farag, A.A.: Direct method for shape recovery from polarization and shading. In: Proceedings of IEEE International Conference on Image Processing (ICIP), pp. 1769–1772 (2012)

    Google Scholar 

  38. Maruyama, Y., et al.: 3.2-MP back-illuminated polarization image sensor with four-directional air-gap wire grid and 2.5-\(\mu \)m pixels. IEEE Trans. Electron Devices 65(6), 2544–2551 (2018)

    Article  Google Scholar 

  39. Maurer, D., Ju, Y.C., Breuß, M., Bruhn, A.: Combining shape from shading and stereo: a variational approach for the joint estimation of depth, illumination and albedo. In: Proceedings of British Machine Vision Conference (BMVC), p. 76 (2016)

    Google Scholar 

  40. Mihoubi, S., Lapray, P.J., Bigué, L.: Survey of demosaicking methods for polarization filter array images. Sensors 18(11), 3688 (2018)

    Article  Google Scholar 

  41. Miyazaki, D., Furuhashi, R., Hiura, S.: Shape estimation of concave specular object from multiview polarization. J. Electron. Imaging 29(4), 041006 (2020)

    Article  Google Scholar 

  42. Miyazaki, D., Shigetomi, T., Baba, M., Furukawa, R., Hiura, S., Asada, N.: Surface normal estimation of black specular objects from multiview polarization images. Opt. Eng. 56(4), 041303 (2016)

    Article  Google Scholar 

  43. Miyazaki, D., Tan, R.T., Hara, K., Ikeuchi, K.: Polarization-based inverse rendering from a single view. In: Proceedings of IEEE International Conference on Computer Vision (ICCV), vol. 2, pp. 982–987 (2003)

    Google Scholar 

  44. Morel, O., Meriaudeau, F., Stolz, C., Gorria, P.: Polarization imaging applied to 3D reconstruction of specular metallic surfaces. In: Proceedings of SPIE-IS&T Electronic Imaging (EI), vol. 5679, pp. 178–186 (2005)

    Google Scholar 

  45. Morimatsu, M., Monno, Y., Tanaka, M., Okutomi, M.: Monochrome and color polarization demosaicking using edge-aware residual interpolation. In: Proceedings of IEEE International Conference on Image Processing (ICIP). To appear (2020)

    Google Scholar 

  46. Ngo Thanh, T., Nagahara, H., Taniguchi, R.I.: Shape and light directions from shading and polarization. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2310–2318 (2015)

    Google Scholar 

  47. Park, J., Sinha, S.N., Matsushita, Y., Tai, Y.W., Kweon, I.S.: Robust multiview photometric stereo using planar mesh parameterization. IEEE Trans. Pattern Anal. Mach. Intell. 39(8), 1591–1604 (2017)

    Article  Google Scholar 

  48. Rahmann, S., Canterakis, N.: Reconstruction of specular surfaces using polarization imaging. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), vol. 1, pp. 149–155 (2001)

    Google Scholar 

  49. Ramamoorthi, R., Hanrahan, P.: An efficient representation for irradiance environment maps. In: Proceedings of Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH), pp. 497–500 (2001)

    Google Scholar 

  50. Schonberger, J.L., Frahm, J.M.: Structure-from-motion revisited. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4104–4113 (2016)

    Google Scholar 

  51. Schönberger, J.L., Zheng, E., Frahm, J.-M., Pollefeys, M.: Pixelwise view selection for unstructured multi-view stereo. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9907, pp. 501–518. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46487-9_31

    Chapter  Google Scholar 

  52. Smith, W., Ramamoorthi, R., Tozza, S.: Height-from-polarisation with unknown lighting or albedo. IEEE Trans. Pattern Anal. Mach. Intell. 41(12), 2875–2888 (2019)

    Article  Google Scholar 

  53. Stokes, G.G.: On the composition and resolution of streams of polarized light from different sources. Trans. Cambridge Philos. Soc. 9, 399 (1851)

    Google Scholar 

  54. Su, H., Maji, S., Kalogerakis, E., Learned-Miller, E.: Multi-view convolutional neural networks for 3D shape recognition. In: Proceedings of IEEE International Conference on Computer Vision (ICCV), pp. 945–953 (2015)

    Google Scholar 

  55. Tozza, S., Smith, W.A., Zhu, D., Ramamoorthi, R., Hancock, E.R.: Linear differential constraints for photo-polarimetric height estimation. In: Proceedings of IEEE International Conference on Computer Vision (ICCV), pp. 2279–2287 (2017)

    Google Scholar 

  56. Wu, C., Liu, Y., Dai, Q., Wilburn, B.: Fusing multiview and photometric stereo for 3D reconstruction under uncalibrated illumination. IEEE Trans. Vis. Comput. Graph. 17(8), 1082–1095 (2010)

    Google Scholar 

  57. Wu, C., Wilburn, B., Matsushita, Y., Theobalt, C.: High-quality shape from multi-view stereo and shading under general illumination. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 969–976 (2011)

    Google Scholar 

  58. Wu, L., Ganesh, A., Shi, B., Matsushita, Y., Wang, Y., Ma, Y.: Robust photometric stereo via low-rank matrix completion and recovery. In: Proceedings of Asian Conference on Computer Vision (ACCV), pp. 703–717 (2010)

    Google Scholar 

  59. Xiong, Y., Chakrabarti, A., Basri, R., Gortler, S.J., Jacobs, D.W., Zickler, T.: From shading to local shape. IEEE Trans. Pattern Anal. Mach. Intell. 37(1), 67–79 (2014)

    Article  Google Scholar 

  60. Yang, L., Tan, F., Li, A., Cui, Z., Furukawa, Y., Tan, P.: Polarimetric dense monocular SLAM. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3857–3866 (2018)

    Google Scholar 

  61. Zhang, R., Tsai, P.S., Cryer, J.E., Shah, M.: Shape-from-shading: a survey. IEEE Trans. Pattern Anal. Mach. Intell. 21(8), 690–706 (1999)

    Article  Google Scholar 

  62. Zhu, D., Smith, W.A.: Depth from a Polarisation + RGB stereo pair. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7586–7595 (2019)

    Google Scholar 

Download references

Acknowledgment

This work was partly supported by JSPS KAKENHI Grant Number 17H00744. The authors would like to thank Dr. Zhaopeng Cui for sharing the data of Polarimetric MVS.

Author information

Authors and Affiliations

  1. Tokyo Institute of Technology, Tokyo, Japan

    Jinyu Zhao, Yusuke Monno & Masatoshi Okutomi

Authors
  1. Jinyu Zhao
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yusuke Monno
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Masatoshi Okutomi
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jinyu Zhao .

Editor information

Editors and Affiliations

  1. University of Oxford, Oxford, UK

    Andrea Vedaldi

  2. Graz University of Technology, Graz, Austria

    Horst Bischof

  3. University of Freiburg, Freiburg im Breisgau, Germany

    Thomas Brox

  4. University of North Carolina at Chapel Hill, Chapel Hill, NC, USA

    Jan-Michael Frahm

1 Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (mp4 79178 KB)

Supplementary material 2 (pdf 14715 KB)

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Zhao, J., Monno, Y., Okutomi, M. (2020). Polarimetric Multi-view Inverse Rendering. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, JM. (eds) Computer Vision – ECCV 2020. ECCV 2020. Lecture Notes in Computer Science(), vol 12369. Springer, Cham. https://doi.org/10.1007/978-3-030-58586-0_6

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-58586-0_6

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-58585-3

  • Online ISBN: 978-3-030-58586-0

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation