What Is Learned in Deep Uncalibrated Photometric Stereo?

Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 12359)


This paper targets at discovering what a deep uncalibrated photometric stereo network learns to resolve the problem’s inherent ambiguity, and designing an effective network architecture based on the new insight to improve the performance. The recently proposed deep uncalibrated photometric stereo method achieved promising results in estimating directional lightings. However, what specifically inside the network contributes to its success remains a mystery. In this paper, we analyze the features learned by this method and find that they strikingly resemble attached shadows, shadings, and specular highlights, which are known to provide useful clues in resolving the generalized bas-relief (GBR) ambiguity. Based on this insight, we propose a guided calibration network, named GCNet, that explicitly leverages object shape and shading information for improved lighting estimation. Experiments on synthetic and real datasets show that GCNet achieves improved results in lighting estimation for photometric stereo, which echoes the findings of our analysis. We further demonstrate that GCNet can be directly integrated with existing calibrated methods to achieve improved results on surface normal estimation. Our code and model can be found at


Uncalibrated photometric stereo Generalized bas-relief ambiguity Deep neural network 



Michael Waechter was supported through a JSPS Postdoctoral Fellowship (JP17F17350). Boxin Shi is supported by the National Natural Science Foundation of China under Grant No. 61872012, National Key R&D Program of China (2019YFF0302902), and Beijing Academy of Artificial Intelligence (BAAI). Kwan-Yee K. Wong is supported by the Research Grant Council of Hong Kong (SAR), China, under the project HKU 17203119. Yasuyuki Matsushita is supported by JSPS KAKENHI Grant Number JP19H01123.

Supplementary material

504468_1_En_44_MOESM1_ESM.pdf (21.7 mb)
Supplementary material 1 (pdf 22238 KB)


  1. 1.
    Ackermann, J., Fuhrmann, S., Goesele, M.: Geometric point light source calibration. In: Vision, Modeling & Visualization (2013)Google Scholar
  2. 2.
    Ackermann, J., Goesele, M.: A survey of photometric stereo techniques. Foundations and Trends in Computer Graphics and Vision (2015)Google Scholar
  3. 3.
    Agarwal, S., et al.: Building Rome in a day. Commun. ACM 54, 105–112 (2011)Google Scholar
  4. 4.
    Alldrin, N.G., Mallick, S.P., Kriegman, D.J.: Resolving the generalized bas-relief ambiguity by entropy minimization. In: CVPR (2007)Google Scholar
  5. 5.
    Basri, R., Jacobs, D., Kemelmacher, I.: Photometric stereo with general, unknown lighting. IJCV 5, 105–113 (2007)Google Scholar
  6. 6.
    Belhumeur, P.N., Kriegman, D.J., Yuille, A.L.: The bas-relief ambiguity. IJCV 35, 33–44 (1999)Google Scholar
  7. 7.
    Chandraker, M.K., Kahl, F., Kriegman, D.J.: Reflections on the generalized bas-relief ambiguity. In: CVPR (2005)Google Scholar
  8. 8.
    Chen, G., Han, K., Shi, B., Matsushita, Y., Wong, K.Y.K.: Self-calibrating deep photometric stereo networks. In: CVPR (2019)Google Scholar
  9. 9.
    Chen, G., Han, K., Shi, B., Matsushita, Y., Wong, K.Y.K.: Deep photometric stereo for non-Lambertian surfaces. TPAMI (2020)Google Scholar
  10. 10.
    Chen, G., Han, K., Wong, K.-Y.K.: PS-FCN: a flexible learning framework for photometric stereo. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11213, pp. 3–19. Springer, Cham (2018). Scholar
  11. 11.
    Cho, D., Matsushita, Y., Tai, Y.W., Kweon, I.S.: Semi-calibrated photometric stereo. TPAMI (2018)Google Scholar
  12. 12.
    Drbohlav, O., Chantler, M.: Can two specular pixels calibrate photometric stereo? In: ICCV (2005)Google Scholar
  13. 13.
    Drbohlav, O., Šára, R.: Specularities reduce ambiguity of uncalibrated photometric stereo. In: Heyden, A., Sparr, G., Nielsen, M., Johansen, P. (eds.) ECCV 2002. LNCS, vol. 2351, pp. 46–60. Springer, Heidelberg (2002). Scholar
  14. 14.
    Einarsson, P., et al.: Relighting human locomotion with flowed reflectance fields. In: EGSR (2006)Google Scholar
  15. 15.
    Epstein, R., Yuille, A.L., Belhumeur, P.N.: Learning object representations from lighting variations. In: International Workshop on Object Representation in Computer Vision (1996)Google Scholar
  16. 16.
    Georghiades, A.S.: Incorporating the torrance and sparrow model of reflectance in uncalibrated photometric stereo. In: ICCV (2003)Google Scholar
  17. 17.
    Goldman, D.B., Curless, B., Hertzmann, A., Seitz, S.M.: Shape and spatially-varying BRDFs from photometric stereo. TPAMI (2010)Google Scholar
  18. 18.
    Haefner, B., Ye, Z., Gao, M., Wu, T., Quéau, Y., Cremers, D.: Variational uncalibrated photometric stereo under general lighting. In: ICCV (2019)Google Scholar
  19. 19.
    Hayakawa, H.: Photometric stereo under a light source with arbitrary motion. JOSA A 11, 3079–3089 (1994)Google Scholar
  20. 20.
    Herbort, S., Wöhler, C.: An introduction to image-based 3D surface reconstruction and a survey of photometric stereo methods. 3D Research (2011)Google Scholar
  21. 21.
    Hertzmann, A., Seitz, S.M.: Example-based photometric stereo: shape reconstruction with general, varying BRDFs. TPAMI 27, 1254–1264 (2005)Google Scholar
  22. 22.
    Ikehata, S.: CNN-PS: CNN-based photometric stereo for general non-convex surfaces. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11219, pp. 3–19. Springer, Cham (2018). Scholar
  23. 23.
    Ikehata, S., Aizawa, K.: Photometric stereo using constrained bivariate regression for general isotropic surfaces. In: CVPR (2014)Google Scholar
  24. 24.
    Jakob, W.: Mitsuba renderer (2010)Google Scholar
  25. 25.
    Kingma, D., Ba, J.: ADAM: a method for stochastic optimization. In: ICLR (2015)Google Scholar
  26. 26.
    Kriegman, D.J., Belhumeur, P.N.: What shadows reveal about object structure. JOSA A 18, 1804–1813 (2001)Google Scholar
  27. 27.
    Li, J., Robles-Kelly, A., You, S., Matsushita, Y.: Learning to minify photometric stereo. In: CVPR (2019)Google Scholar
  28. 28.
    Lu, F., Chen, X., Sato, I., Sato, Y.: SymPS: BRDF symmetry guided photometric stereo for shape and light source estimation. TPAMI 40, 221–234 (2018)Google Scholar
  29. 29.
    Lu, F., Matsushita, Y., Sato, I., Okabe, T., Sato, Y.: Uncalibrated photometric stereo for unknown isotropic reflectances. In: CVPR (2013)Google Scholar
  30. 30.
    Lu, F., Matsushita, Y., Sato, I., Okabe, T., Sato, Y.: From intensity profile to surface normal: Photometric stereo for unknown light sources and isotropic reflectances. TPAMI 37, 1999–2012 (2015)Google Scholar
  31. 31.
    Matusik, W., Pfister, H., Brand, M., McMillan, L.: A data-driven reflectance model. In: SIGGRAPH (2003)Google Scholar
  32. 32.
    Midorikawa, K., Yamasaki, T., Aizawa, K.: Uncalibrated photometric stereo by stepwise optimization using principal components of isotropic BRDFs. In: CVPR (2016)Google Scholar
  33. 33.
    Mo, Z., Shi, B., Lu, F., Yeung, S.K., Matsushita, Y.: Uncalibrated photometric stereo under natural illumination. In: CVPR (2018)Google Scholar
  34. 34.
    Okabe, T., Sato, I., Sato, Y.: Attached shadow coding: estimating surface normals from shadows under unknown reflectance and lighting conditions. In: ICCV (2009)Google Scholar
  35. 35.
    Papadhimitri, T., Favaro, P.: A closed-form, consistent and robust solution to uncalibrated photometric stereo via local diffuse reflectance maxima. IJCV (2014)Google Scholar
  36. 36.
    Paszke, A., Gross, S., Chintala, S., Chanan, G.: PyTorch: tensors and dynamic neural networks in Python with strong GPU acceleration (2017)Google Scholar
  37. 37.
    Quéau, Y., Wu, T., Lauze, F., Durou, J.D., Cremers, D.: A non-convex variational approach to photometric stereo under inaccurate lighting. In: CVPR (2017)Google Scholar
  38. 38.
    Santo, H., Samejima, M., Sugano, Y., Shi, B., Matsushita, Y.: Deep photometric stereo network. In: ICCV Workshops (2017)Google Scholar
  39. 39.
    Santo, H., Waechter, M., Samejima, M., Sugano, Y., Matsushita, Y.: Light structure from pin motion: simple and accurate point light calibration for physics-based modeling. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11207, pp. 3–19. Springer, Cham (2018). Scholar
  40. 40.
    Sato, I., Okabe, T., Yu, Q., Sato, Y.: Shape reconstruction based on similarity in radiance changes under varying illumination. In: ICCV (2007)Google Scholar
  41. 41.
    Seitz, S.M., Curless, B., Diebel, J., Scharstein, D., Szeliski, R.: A comparison and evaluation of multi-view stereo reconstruction algorithms. In: CVPR (2006)Google Scholar
  42. 42.
    Shen, H.L., Cheng, Y.: Calibrating light sources by using a planar mirror. J. Electr. Imaging 20(1), 013002-1–013002-6 (2011)Google Scholar
  43. 43.
    Shi, B., Inose, K., Matsushita, Y., Tan, P., Yeung, S.K., Ikeuchi, K.: Photometric stereo using internet images. In: 3DV (2014)Google Scholar
  44. 44.
    Shi, B., Matsushita, Y., Wei, Y., Xu, C., Tan, P.: Self-calibrating photometric stereo. In: CVPR (2010)Google Scholar
  45. 45.
    Shi, B., Mo, Z., Wu, Z., Duan, D., Yeung, S.K., Tan, P.: A benchmark dataset and evaluation for non-Lambertian and uncalibrated photometric stereo. TPAMI 41, 271–284 (2019)Google Scholar
  46. 46.
    Shi, B., Tan, P., Matsushita, Y., Ikeuchi, K.: Bi-polynomial modeling of low-frequency reflectances. TPAMI 36, 1078–1091 (2014)Google Scholar
  47. 47.
    Silver, W.M.: Determining shape and reflectance using multiple images. Ph.d. thesis, Massachusetts Institute of Technology (1980)Google Scholar
  48. 48.
    Tan, P., Mallick, S.P., Quan, L., Kriegman, D.J., Zickler, T.: Isotropy, reciprocity and the generalized bas-relief ambiguity. In: CVPR (2007)Google Scholar
  49. 49.
    Taniai, T., Maehara, T.: Neural inverse rendering for general reflectance photometric stereo. In: ICML (2018)Google Scholar
  50. 50.
    Woodham, R.J.: Photometric method for determining surface orientation frommultiple images. Opt. Eng. (1980)Google Scholar
  51. 51.
    Wu, Z., Tan, P.: Calibrating photometric stereo by holistic reflectance symmetry analysis. In: CVPR (2013)Google Scholar
  52. 52.
    Yuille, A.L., Snow, D., Epstein, R., Belhumeur, P.N.: Determining generative models of objects under varying illumination: shape and albedo from multiple images using SVD and integrability. IJCV 35, 203–222 (1999)Google Scholar
  53. 53.
    Zheng, Q., Jia, Y., Shi, B., Jiang, X., Duan, L.Y., Kot, A.C.: SPLINE-Net: sparse photometric stereo through lighting interpolation and normal estimation networks. In: ICCV (2019)Google Scholar
  54. 54.
    Zhou, Z., Tan, P.: Ring-light photometric stereo. In: Daniilidis, K., Maragos, P., Paragios, N. (eds.) ECCV 2010. LNCS, vol. 6312, pp. 265–279. Springer, Heidelberg (2010). Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  1. 1.The University of Hong KongHong KongChina
  2. 2.Osaka UniversitySuitaJapan
  3. 3.Peking UniversityBeijingChina
  4. 4.Peng Cheng LaboratoryShenzhenChina

Personalised recommendations