Advertisement

PS-FCN: A Flexible Learning Framework for Photometric Stereo

  • Guanying Chen
  • Kai Han
  • Kwan-Yee K. Wong
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11213)

Abstract

This paper addresses the problem of photometric stereo for non-Lambertian surfaces. Existing approaches often adopt simplified reflectance models to make the problem more tractable, but this greatly hinders their applications on real-world objects. In this paper, we propose a deep fully convolutional network, called PS-FCN, that takes an arbitrary number of images of a static object captured under different light directions with a fixed camera as input, and predicts a normal map of the object in a fast feed-forward pass. Unlike the recently proposed learning based method, PS-FCN does not require a pre-defined set of light directions during training and testing, and can handle multiple images and light directions in an order-agnostic manner. Although we train PS-FCN on synthetic data, it can generalize well on real datasets. We further show that PS-FCN can be easily extended to handle the problem of uncalibrated photometric stereo. Extensive experiments on public real datasets show that PS-FCN outperforms existing approaches in calibrated photometric stereo, and promising results are achieved in uncalibrated scenario, clearly demonstrating its effectiveness.

Keywords

Photometric stereo Convolutional neural network 

Notes

Acknowledgments

We thank Hiroaki Santo for his help with the comparison to DPSN. We also thank Boxin Shi and Zhipeng Mo for their help with the evaluation on the DiLiGenT benchmark. We gratefully acknowledge the support of NVIDIA Corporation with the donation of the Titan X Pascal GPU used for this research. Kai Han is supported by EPSRC Programme Grant Seebibyte EP/M013774/1.

References

  1. 1.
    Woodham, R.J.: Photometric method for determining surface orientation from multiple images. Opt. Eng. 19, 191130 (1980)CrossRefGoogle Scholar
  2. 2.
    Silver, W.M.: Determining shape and reflectance using multiple images. Ph.D. thesis, Massachusetts Institute of Technology (1980)Google Scholar
  3. 3.
    Georghiades, A.S.: Incorporating the torrance and sparrow model of reflectance in uncalibrated photometric stereo. In: ICCV (2003)Google Scholar
  4. 4.
    Chung, H.S., Jia, J.: Efficient photometric stereo on glossy surfaces with wide specular lobes. In: CVPR (2008)Google Scholar
  5. 5.
    Ruiters, R., Klein, R.: Heightfield and spatially varying BRDF reconstruction for materials with interreflections. In: Computer Graphics Forum (2009)Google Scholar
  6. 6.
    Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. In: NIPS (2012)Google Scholar
  7. 7.
    LeCun, Y., Bottou, L., Bengio, Y., Haffner, P.: Gradient-based learning applied to document recognition. In: Proceedings of the IEEE (1998)Google Scholar
  8. 8.
    Santo, H., Samejima, M., Sugano, Y., Shi, B., Matsushita, Y.: Deep photometric stereo network. In: ICCV Workshops (2017)Google Scholar
  9. 9.
    Long, J., Shelhamer, E., Darrell, T.: Fully convolutional networks for semantic segmentation. In: CVPR (2015)Google Scholar
  10. 10.
    Johnson, M.K., Adelson, E.H.: Shape estimation in natural illumination. In: CVPR (2011)Google Scholar
  11. 11.
    Wiles, O., Zisserman, A.: SilNet: single-and multi-view reconstruction by learning from silhouettes. In: BMVC (2017)Google Scholar
  12. 12.
    Matusik, W., Pfister, H., Brand, M., McMillan, L.: A data-driven reflectance model. In: SIGGRAPH (2003)Google Scholar
  13. 13.
    Shi, B., Wu, Z., Mo, Z., Duan, D., Yeung, S.K., Tan, P.: A benchmark dataset and evaluation for non-lambertian and uncalibrated photometric stereo. IEEE TPAMI (2018)Google Scholar
  14. 14.
    Alldrin, N., Zickler, T., Kriegman, D.: Photometric stereo with non-parametric and spatially-varying reflectance. In: CVPR (2008)Google Scholar
  15. 15.
    Einarsson, P., et al.: Relighting human locomotion with flowed reflectance fields. In: EGSR (2006)Google Scholar
  16. 16.
    Herbort, S., Wöhler, C.: An introduction to image-based 3D surface reconstruction and a survey of photometric stereo methods. 3D Res. 2, 4 (2011)CrossRefGoogle Scholar
  17. 17.
    Wu, L., Ganesh, A., Shi, B., Matsushita, Y., Wang, Y., Ma, Y.: Robust photometric stereo via low-rank matrix completion and recovery. In: Kimmel, R., Klette, R., Sugimoto, A. (eds.) ACCV 2010 Part III. LNCS, vol. 6494, pp. 703–717. Springer, Heidelberg (2011).  https://doi.org/10.1007/978-3-642-19318-7_55CrossRefGoogle Scholar
  18. 18.
    Mukaigawa, Y., Ishii, Y., Shakunaga, T.: Analysis of photometric factors based on photometric linearization. JOSA A 24, 3326–3334 (2007)CrossRefGoogle Scholar
  19. 19.
    Miyazaki, D., Hara, K., Ikeuchi, K.: Median photometric stereo as applied to the segonko tumulus and museum objects. IJCV 86, 229 (2010)CrossRefGoogle Scholar
  20. 20.
    Wu, T.P., Tang, C.K.: Photometric stereo via expectation maximization. IEEE TPAMI 32, 546–560 (2010)CrossRefGoogle Scholar
  21. 21.
    Ikehata, S., Wipf, D., Matsushita, Y., Aizawa, K.: Robust photometric stereo using sparse regression. In: CVPR (2012)Google Scholar
  22. 22.
    Shi, B., Tan, P., Matsushita, Y., Ikeuchi, K.: Bi-polynomial modeling of low-frequency reflectances. IEEE TPAMI (2014)Google Scholar
  23. 23.
    Ikehata, S., Aizawa, K.: Photometric stereo using constrained bivariate regression for general isotropic surfaces. In: CVPR (2014)Google Scholar
  24. 24.
    Holroyd, M., Lawrence, J., Humphreys, G., Zickler, T.: A photometric approach for estimating normals and tangents. In: ACM TOG (2008)Google Scholar
  25. 25.
    Hertzmann, A., Seitz, S.M.: Example-based photometric stereo: shape reconstruction with general, varying BRDFs. IEEE TPAMI 27, 1254–1264 (2005)CrossRefGoogle Scholar
  26. 26.
    Hui, Z., Sankaranarayanan, A.C.: A dictionary-based approach for estimating shape and spatially-varying reflectance. In: ICCP (2015)Google Scholar
  27. 27.
    Alldrin, N.G., Mallick, S.P., Kriegman, D.J.: Resolving the generalized bas-relief ambiguity by entropy minimization. In: CVPR (2007)Google Scholar
  28. 28.
    Shi, B., Matsushita, Y., Wei, Y., Xu, C., Tan, P.: Self-calibrating photometric stereo. In: CVPR (2010)Google Scholar
  29. 29.
    Wu, Z., Tan, P.: Calibrating photometric stereo by holistic reflectance symmetry analysis. In: CVPR (2013)Google Scholar
  30. 30.
    Papadhimitri, T., Favaro, P.: A closed-form, consistent and robust solution to uncalibrated photometric stereo via local diffuse reflectance maxima. IJCV 107, 139–154 (2014)CrossRefGoogle Scholar
  31. 31.
    Lu, F., Matsushita, Y., Sato, I., Okabe, T., Sato, Y.: From intensity profile to surface normal: photometric stereo for unknown light sources and isotropic reflectances. IEEE TPAMI 37, 1999–2012 (2015)CrossRefGoogle Scholar
  32. 32.
    Lu, F., Chen, X., Sato, I., Sato, Y.: SymPS: BRDF symmetry guided photometric stereo for shape and light source estimation. IEEE TPAMI 40, 221–234 (2018)CrossRefGoogle Scholar
  33. 33.
    Eigen, D., Puhrsch, C., Fergus, R.: Depth map prediction from a single image using a multi-scale deep network. In: NIPS (2014)Google Scholar
  34. 34.
    Wang, X., Fouhey, D., Gupta, A.: Designing deep networks for surface normal estimation. In: CVPR (2015)Google Scholar
  35. 35.
    Choy, C.B., Xu, D., Gwak, J., Chen, K., Savarese, S.: 3D–R2N2: a unified approach for single and multi-view 3D object reconstruction. In: ECCV (2016)Google Scholar
  36. 36.
    Hartmann, W., Galliani, S., Havlena, M., Van Gool, L., Schindler, K.: Learned multi-patch similarity. In: ICCV (2017)Google Scholar
  37. 37.
    Bromley, J., Guyon, I., LeCun, Y., Säckinger, E., Shah, R.: Signature verification using a “siamese” time delay neural network. Int.J. Pattern Recogn. Artif. Intell. (1993)Google Scholar
  38. 38.
    Jakob, W.: Mitsuba renderer (2010)Google Scholar
  39. 39.
    Paszke, A., Gross, S., Chintala, S., Chanan, G.: PyTorch: tensors and dynamic neural networks in python with strong GPU acceleration (2017)Google Scholar
  40. 40.
    Kingma, D., Ba, J.: Adam: a method for stochastic optimization. In: ICLR (2015)Google Scholar
  41. 41.
    Frankot, R.T., Chellappa, R.: A method for enforcing integrability in shape from shading algorithms. IEEE TPAMI 10, 439–451 (1988)CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2018

Authors and Affiliations

  1. 1.The University of Hong KongPokfulamHong Kong
  2. 2.University of OxfordOxfordUK

Personalised recommendations