Multi-view Inverse Rendering Under Arbitrary Illumination and Albedo

  • Kichang Kim
  • Akihiko Torii
  • Masatoshi Okutomi
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9907)


3D shape reconstruction with multi-view stereo (MVS) relies on a robust evaluation of photo consistencies across images. The robustness is ensured by isolating surface albedo and scene illumination from the shape recovery, i.e. shading and colour variation are regarded as a nuisance in MVS. This yields a gap in the qualities between the recovered shape and the images used. We present a method to address it by jointly estimating detailed shape, illumination and albedo using the initial shape robustly recovered by MVS. This is achieved by solving the multi-view inverse rendering problem using the geometric and photometric smoothness terms and the normalized spherical harmonics illumination model. Our method allows spatially-varying albedo and per image illumination without any prerequisites such as training data or image segmentation. We demonstrate that our method can clearly improve the 3D shape and recover illumination and albedo on real world scenes.


Multi-view stereo Shape from shading Inverse rendering 



This work was partly supported by JSPS KAKENHI Grant Number 25240025 and 15H05313.


  1. 1.
    Biehler, J., Fane, B.: 3D Printing with Autodesk: Create and Print 3D Objects with 123D, AutoCAD and Inventor, 1st edn. Que Publishing Company, Seattle (2014)Google Scholar
  2. 2.
    Anguelov, D., Dulong, C., Filip, D., Frueh, C., Lafon, S., Lyon, R., Ogale, A., Vincent, L., Weaver, J.: Google street view: capturing the world at street level. Computer 6, 32–38 (2010)CrossRefGoogle Scholar
  3. 3.
    Moulon, P., Monasse, P., Marlet, R., et al.: Openmvg. an open multiple view geometry library.
  4. 4.
    Snavely, N., Seitz, S., Szeliski, R.: Modeling the world from internet photo collections. IJCV 80(2), 189–210 (2008)CrossRefGoogle Scholar
  5. 5.
    Wu, C., Agarwal, S., Curless, B., Seitz, S.M.: Multicore bundle adjustment. In: CVPR, pp. 3057–3064. IEEE (2011)Google Scholar
  6. 6.
    Fuhrmann, S., Langguth, F., Goesele, M.: MVE - a multi-view reconstruction environment. In: EG GCH (2014)Google Scholar
  7. 7.
    Furukawa, Y., Ponce, J.: Accurate, dense, and robust multiview stereopsis. PAMI 32(8), 1362–1376 (2010)CrossRefGoogle Scholar
  8. 8.
    Jancosek, M., Pajdla, T.: Multi-view reconstruction preserving weakly-supported surfaces. In: CVPR, pp. 3121–3128. IEEE (2011)Google Scholar
  9. 9.
    Furukawa, Y., Hernndez, C.: Multi-view stereo: a tutorial. Found. Trends Comput. Graph. Vis. 9(1–2), 1–148 (2013)Google Scholar
  10. 10.
    Tola, E., Lepetit, V., Fua, P.: Daisy: an efficient dense descriptor applied to wide-baseline stereo. PAMI 32(5), 815–830 (2010)CrossRefGoogle Scholar
  11. 11.
    Horn, B.K.P.: Obtaining shape from shading information. In: Shape from Shading, pp. 123–171. MIT Press, Cambridge (1989)Google Scholar
  12. 12.
    Barron, J.T., Malik, J.: Shape, albedo, and illumination from a single image of an unknown object. In: CVPR, pp. 334–341. IEEE (2012)Google Scholar
  13. 13.
    Ikehata, S., Wipf, D., Matsushita, Y., Aizawa, K.: Photometric stereo using sparse bayesian regression for general diffuse surfaces. PAMI 36(9), 1816–1831 (2014)CrossRefGoogle Scholar
  14. 14.
    Woodham, R.J.: Photometric method for determining surface orientation from multiple images. Opt. Eng. 19(1), 121–171 (1980)CrossRefGoogle Scholar
  15. 15.
    Blake, A., Zisserman, A., Knowles, G.: Surface descriptions from stereo and shading. Image Vis. Comput. 3(4), 183–191 (1985)CrossRefGoogle Scholar
  16. 16.
    Jin, H., Cremers, D., Wang, D., Prados, E., Yezzi, A., Soatto, S.: 3-d reconstruction of shaded objects from multiple images under unknown illumination. IJCV 76(3), 245–256 (2008)CrossRefGoogle Scholar
  17. 17.
    Park, J., Sinha, S.N., Matsushita, Y., Tai, Y.W., Kweon, I.S.: Multiview photometric stereo using planar mesh parameterization. In: ICCV, pp. 1161–1168. IEEE (2013)Google Scholar
  18. 18.
    Samaras, D., Metaxas, D., Fua, P., Leclerc, Y.G.: Variable albedo surface reconstruction from stereo and shape from shading. In: CVPR, vol. 1, pp. 480–487. IEEE (2000)Google Scholar
  19. 19.
    Wu, C., Liu, Y., Dai, Q., Wilburn, B.: Fusing multiview and photometric stereo for 3d reconstruction under uncalibrated illumination. IEEE Trans. Visual Comput. Graphics 17(8), 1082–1095 (2011)CrossRefGoogle Scholar
  20. 20.
    Wu, C., Wilburn, B., Matsushita, Y., Theobalt, C.: High-quality shape from multi-view stereo and shading under general illumination. In: CVPR, pp. 969–976. IEEE (2011)Google Scholar
  21. 21.
    Shan, Q., Adams, R., Curless, B., Furukawa, Y., Seitz, S.M.: The visual turing test for scene reconstruction. In: 3DV, pp. 25–32. IEEE (2013)Google Scholar
  22. 22.
    Zollhöfer, M., Dai, A., Innman, M., Wu, C., Stamminger, M., Theobalt, C., Nießner, M.: Shading-based refinement on volumetric signed distance functions. ACM Trans. Graph. 34(4), 96:1–96:14 (2015)CrossRefzbMATHGoogle Scholar
  23. 23.
    Okutomi, M., Kanade, T.: A multiple-baseline stereo. PAMI 15(4), 353–363 (1993)CrossRefGoogle Scholar
  24. 24.
    Goesele, M., Curless, B., Seitz, S.M.: Multi-view stereo revisited. In: CVPR, vol. 2, pp. 2402–2409. IEEE (2006)Google Scholar
  25. 25.
    Kolmogorov, V., Zabih, R.: Multi-camera scene reconstruction via graph cuts. In: Heyden, A., Sparr, G., Nielsen, M., Johansen, P. (eds.) ECCV 2002, Part III. LNCS, vol. 2352, pp. 82–96. Springer, Heidelberg (2002)CrossRefGoogle Scholar
  26. 26.
    Campbell, N.D.F., Vogiatzis, G., Hernández, C., Cipolla, R.: Using multiple hypotheses to improve depth-maps for multi-view stereo. In: Forsyth, D., Torr, P., Zisserman, A. (eds.) ECCV 2008, Part I. LNCS, vol. 5302, pp. 766–779. Springer, Heidelberg (2008)CrossRefGoogle Scholar
  27. 27.
    Fuhrmann, S., Goesele, M.: Floating scale surface reconstruction. ACM Trans. Graph. 33(4), 46:1–46:11 (2014)CrossRefGoogle Scholar
  28. 28.
    Kutulakos, K.N., Seitz, S.M.: A theory of shape by space carving. IJCV 38(3), 199–218 (2000)CrossRefzbMATHGoogle Scholar
  29. 29.
    Labatut, P., Pons, J.P., Keriven, R.: Efficient multi-view reconstruction of large-scale scenes using interest points, delaunay triangulation and graph cuts. In: ICCV, pp. 1–8. IEEE (2007)Google Scholar
  30. 30.
    Vogiatzis, G., Hernández, C., Torr, P.H., Cipolla, R.: Multiview stereo via volumetric graph-cuts and occlusion robust photo-consistency. PAMI 29(12), 2241–2246 (2007)CrossRefGoogle Scholar
  31. 31.
    Bleyer, M., Rhemann, C., Rother, C.: Patchmatch stereo-stereo matching with slanted support windows. BMVC 11, 1–11 (2011)Google Scholar
  32. 32.
    Bao, S.Y., Chandraker, M., Lin, Y., Savarese, S.: Dense object reconstruction with semantic priors. In: CVPR, pp. 1264–1271. IEEE (2013)Google Scholar
  33. 33.
    Kazhdan, M., Bolitho, M., Hoppe, H.: Poisson surface reconstruction. In: Proceedings of the Fourth Eurographics Symposium on Geometry Processing, vol. 7 (2006)Google Scholar
  34. 34.
    Vu, H.H., Labatut, P., Pons, J.P., Keriven, R.: High accuracy and visibility-consistent dense multiview stereo. PAMI 34(5), 889–901 (2012)CrossRefGoogle Scholar
  35. 35.
    Ikeuchi, K., Horn, B.K.: Numerical shape from shading and occluding boundaries. Artif. Intell. 17(13), 141–184 (1981)CrossRefGoogle Scholar
  36. 36.
    Zhang, R., Tsai, P.S., Cryer, J.E., Shah, M.: Shape-from-shading: a survey. PAMI 21(8), 690–706 (1999)CrossRefzbMATHGoogle Scholar
  37. 37.
    Forsyth, D.A.: Variable-source shading analysis. IJCV 91(3), 280–302 (2011)MathSciNetCrossRefzbMATHGoogle Scholar
  38. 38.
    Johnson, M.K., Adelson, E.H.: Shape estimation in natural illumination. In: CVPR, pp. 2553–2560. IEEE (2011)Google Scholar
  39. 39.
    Basri, R., Jacobs, D.: Photometric stereo with general, unknown lighting. In: CVPR, vol. 2, pp. 374–381. IEEE (2001)Google Scholar
  40. 40.
    Hernandez, C., Vogiatzis, G., Cipolla, R.: Multiview photometric stereo. PAMI 30(3), 548–554 (2008)CrossRefGoogle Scholar
  41. 41.
    Shi, B., Inose, K., Matsushita, Y., Tan, P., Yeung, S.K., Ikeuchi, K.: Photometric stereo using internet images. In: 3DV, vol. 1, pp. 361–368. IEEE (2014)Google Scholar
  42. 42.
    Barron, J.T., Malik, J.: Intrinsic scene properties from a single rgb-d image. In: CVPR, pp. 17–24. IEEE (2013)Google Scholar
  43. 43.
    Han, Y., Lee, J.Y., Kweon, I.S.: High quality shape from a single rgb-d image under uncalibrated natural illumination. In: ICCV, pp. 1617–1624. IEEE (2013)Google Scholar
  44. 44.
    Wu, C., Zollhfer, M., Niener, M., Stamminger, M., Izadi, S., Theobalt, C.: Real-time shading-based refinement for consumer depth cameras. In: SIGGRAPH Asia (2014)Google Scholar
  45. 45.
    Yu, L.F., Yeung, S.K., Tai, Y.W., Lin, S.: Shading-based shape refinement of rgb-d images. In: CVPR, pp. 1415–1422. IEEE (2013)Google Scholar
  46. 46.
    Wu, C.: VisualSFM: a visual structure from motion system (2011).
  47. 47.
    Sweeney, C.: Theia Multiview Geometry Library: Tutorial & Reference. University of California, Santa BarbaraGoogle Scholar
  48. 48.
    Meagher, D.: Geometric modeling using octree encoding. Comput. Graph. Image Process. 19(2), 129–147 (1982)CrossRefGoogle Scholar
  49. 49.
    Kobbelt, L.: \(\sqrt{3}\)-subdivision. In: SIGGRAPH, pp. 103–112. ACM (2000)Google Scholar
  50. 50.
    Ramamoorthi, R., Hanrahan, P.: An efficient representation for irradiance environment maps. In: SIGGRAPH, pp. 497–500. ACM (2001)Google Scholar
  51. 51.
    Bayer, B.E.: Color imaging array (20 1976) US Patent 3,971,065Google Scholar
  52. 52.
    Kang, H.R.: Computational Color Technology. Spie Press, Bellingham (2006)CrossRefGoogle Scholar
  53. 53.
    Eberly, D.: Least Squares Fitting of Data. Magic Software, Chapel Hill (2000)Google Scholar
  54. 54.
    Horn, B.K.P.: Determining lightness from an image. Comput. Graph. Image Process. 3(4), 277–299 (1974)CrossRefGoogle Scholar
  55. 55.
    Jeon, J., Cho, S., Tong, X., Lee, S.: Intrinsic image decomposition using structure-texture separation and surface normals. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014, Part VII. LNCS, vol. 8695, pp. 218–233. Springer, Heidelberg (2014)Google Scholar
  56. 56.
    Laffont, P.Y., Bousseau, A., Paris, S., Durand, F., Drettakis, G.: Coherent intrinsic images from photo collections. ACM Trans. Graphics 31(6), 202:1–202:11 (2012)CrossRefGoogle Scholar
  57. 57.
    Agarwal, S., Mierle, K., et al.: Ceres solver.
  58. 58.
    Bradski, G., et al.: The openCV library. Doct. Dobbs J. 25(11), 120–126 (2000)Google Scholar
  59. 59.
    Sanderson, C.: Armadillo: an open source C++ linear algebra library for fast prototyping and computationally intensive experiments. Technical report, NICTA (2010)Google Scholar
  60. 60.
    Project webpage (software, dataset).
  61. 61.
    Blender - A 3D Modelling and Rendering Package. Blender Foundation, Blender Institute, Amsterdam (2015)Google Scholar
  62. 62.
    Havlena, M., Torii, A., Pajdla, T.: Efficient structure from motion by graph optimization. In: Daniilidis, K., Maragos, P., Paragios, N. (eds.) ECCV 2010, Part II. LNCS, vol. 6312, pp. 100–113. Springer, Heidelberg (2010)CrossRefGoogle Scholar
  63. 63.
    Li, Y., Snavely, N., Huttenlocher, D.P.: Location recognition using prioritized feature matching. In: Daniilidis, K., Maragos, P., Paragios, N. (eds.) ECCV 2010, Part II. LNCS, vol. 6312, pp. 791–804. Springer, Heidelberg (2010)CrossRefGoogle Scholar
  64. 64.
    Jégou, H., Douze, M., Schmid, C., Pérez, P.: Aggregating local descriptors into a compact image representation. In: CVPR, pp. 3304–3311. IEEE (2010)Google Scholar
  65. 65.
    Wilson, K., Snavely, N.: Robust global translations with 1DSfM. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014, Part III. LNCS, vol. 8691, pp. 61–75. Springer, Heidelberg (2014)Google Scholar

Copyright information

© Springer International Publishing AG 2016

Authors and Affiliations

  1. 1.Tokyo Institute of TechnologyTokyoJapan

Personalised recommendations