Advertisement

Journal of Mathematical Imaging and Vision

, Volume 60, Issue 3, pp 313–340 | Cite as

LED-Based Photometric Stereo: Modeling, Calibration and Numerical Solution

  • Yvain Quéau
  • Bastien Durix
  • Tao Wu
  • Daniel Cremers
  • François Lauze
  • Jean-Denis Durou
Article

Abstract

We conduct a thorough study of photometric stereo under nearby point light source illumination, from modeling to numerical solution, through calibration. In the classical formulation of photometric stereo, the luminous fluxes are assumed to be directional, which is very difficult to achieve in practice. Rather, we use light-emitting diodes to illuminate the scene to be reconstructed. Such point light sources are very convenient to use, yet they yield a more complex photometric stereo model which is arduous to solve. We first derive in a physically sound manner this model, and show how to calibrate its parameters. Then, we discuss two state-of-the-art numerical solutions. The first one alternatingly estimates the albedo and the normals, and then integrates the normals into a depth map. It is shown empirically to be independent from the initialization, but convergence of this sequential approach is not established. The second one directly recovers the depth, by formulating photometric stereo as a system of nonlinear partial differential equations (PDEs), which are linearized using image ratios. Although the sequential approach is avoided, initialization matters a lot and convergence is not established either. Therefore, we introduce a provably convergent alternating reweighted least-squares scheme for solving the original system of nonlinear PDEs. Finally, we extend this study to the case of RGB images.

Keywords

3D-reconstruction Photometric stereo Point light sources Variational methods Alternating reweighted least-squares 

Notes

Acknowledgements

Yvain Quéau, Tao Wu and Daniel Cremers were supported by the ERC Consolidator Grant “3D Reloaded”. Funding was provided by European Research Council (Grant No. 649323).

References

  1. 1.
    Ackermann, J., Fuhrmann, S., Goesele, M.: Geometric point light source calibration. In: Proceedings of the 18th International Workshop on Vision, Modeling and Visualization, pp. 161–168. Lugano, Switzerland (2013)Google Scholar
  2. 2.
    Ahmad, J., Sun, J., Smith, L., Smith, M.: An improved photometric stereo through distance estimation and light vector optimization from diffused maxima region. Pattern Recogn. Lett. 50, 15–22 (2014)CrossRefGoogle Scholar
  3. 3.
    Angelopoulou, M.E., Petrou, M.: Uncalibrated flatfielding and illumination vector estimation for photometric stereo face reconstruction. Mach. Vis. Appl. 25(5), 1317–1332 (2013)CrossRefGoogle Scholar
  4. 4.
    Aoto, T., Taketomi, T., Sato, T., Mukaigawa, Y., Yokoya, N.: Position estimation of near point light sources using a clear hollow sphere. In: Proceedings of the 21st International Conference on Pattern Recognition, pp. 3721–3724. Tsukuba, Japan (2012)Google Scholar
  5. 5.
    Barsky, S., Petrou, M.: The 4-source photometric stereo technique for three-dimensional surfaces in the presence of highlights and shadows. IEEE Trans. Pattern Anal. Mach. Intell. 25(10), 1239–1252 (2003)CrossRefGoogle Scholar
  6. 6.
    Basri, R., Jacobs, D.W.: Lambertian reflectance and linear subspaces. IEEE Trans. Pattern Anal. Mach. Intell. 25(2), 218–233 (2003)CrossRefGoogle Scholar
  7. 7.
    Bennahmias, M., Arik, E., Yu, K., Voloshenko, D., Chua, K., Pradhan, R., Forrester, T., Jannson, T.: Modeling of non-Lambertian sources in lighting applications. In: Proceedings of SPIE Optical Engineering and Applications, vol. 6669. San Diego, USA (2007)Google Scholar
  8. 8.
    Bony, A., Bringier, B., Khoudeir, M.: Tridimensional reconstruction by photometric stereo with near spot light sources. In: Proceedings of the 21st European Signal Processing Conference. Marrakech, Morocco (2013)Google Scholar
  9. 9.
    Boyd, S., Parikh, N., Chu, E., Peleato, B., Eckstein, J.: Distributed optimization and statistical learning via the alternating direction method of multipliers. Found. Trends Mach. Learn. 3(1), 1–122 (2011)CrossRefzbMATHGoogle Scholar
  10. 10.
    Bringier, B., Bony, A., Khoudeir, M.: Specularity and shadow detection for the multisource photometric reconstruction of a textured surface. J. Opt. Soc. Am. A 29(1), 11–21 (2012)CrossRefGoogle Scholar
  11. 11.
    Ciortan, I., Pintus, R., Marchioro, G., Daffara, C., Giachetti, A., Gobbetti, E.: A practical reflectance transformation imaging pipeline for surface characterization in cultural heritage. In: Proceedings of the 14th Eurographics Workshop on Graphics and Cultural Heritage. Genova, Italy (2016)Google Scholar
  12. 12.
    Clark, J.J.: Active photometric stereo. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 29–34 (1992)Google Scholar
  13. 13.
    Collins, T., Bartoli, A.: 3D Reconstruction in laparoscopy with close-range photometric stereo. In: Proceedings of the 15th International Conference on Medical Imaging and Computer Assisted Intervention, pp. 634–642. Nice, France (2012)Google Scholar
  14. 14.
    Durou, J.D., Aujol, J.F., Courteille, F.: Integrating the normal field of a surface in the presence of discontinuities. In: Proceedings of the 7th International Conference on Energy Minimization Methods in Computer Vision and Pattern Recognition, Lecture Notes in Computer Science, vol. 5681, pp. 261–273. Bonn, Germany (2009)Google Scholar
  15. 15.
    Gabay, D., Mercier, B.: A dual algorithm for the solution of nonlinear variational problems via finite element approximation. Comput. Math. Appl. 2(1), 17–40 (1976)CrossRefzbMATHGoogle Scholar
  16. 16.
    Gardner, I.C.: Validity of the cosine-fourth-power law of illumination. J. Res. Nat. Bur. Stand. 39, 213–219 (1947)CrossRefGoogle Scholar
  17. 17.
    Giachetti, A., Daffara, C., Reghelin C., Gobbetti, E., Pintus, R.: Light calibration and quality assessment methods for reflectance transformation imaging applied to artworks’ analysis. In: Proceedings of SPIE Optics for Arts, Architecture, and Archaeology V, vol. 9527. Munich, Germany (2015)Google Scholar
  18. 18.
    Glowinski, R., Marroco, A.: Sur l’approximation, par éléments finis d’ordre un, et la résolution, par pénalisation-dualité d’une classe de problèmes de Dirichlet non linéaires. ESAIM Math. Model. Numer. Anal. 9(2), 41–76 (1975)zbMATHGoogle Scholar
  19. 19.
    Golub, G.H., Van Loan, C.F.: Matrix Computations, 4th edn. John Hopkings University Press, Baltimore (2013)zbMATHGoogle Scholar
  20. 20.
    Gotardo, P.F.U., Simon, T., Sheikh, Y., Matthews, I.: Photogeometric scene flow for high-detail dynamic 3D reconstruction. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 846–854. Santiago, Chile (2015)Google Scholar
  21. 21.
    Gratton, S., Lawless, S., Nichols, N.K.: Approximate Gauss–Newton methods for nonlinear least squares problems. SIAM J. Optim. 18, 106–132 (2007)MathSciNetCrossRefzbMATHGoogle Scholar
  22. 22.
    Hara, K., Nishino, K., Ikeuchi, K.: Light source position and reflectance estimation from a single view without the distant illumination assumption. IEEE Trans. Pattern Anal. Mach. Intell. 27(4), 493–505 (2005)CrossRefGoogle Scholar
  23. 23.
    Hernández, C., Vogiatzis, G., Brostow, G.J., Stenger, B., Cipolla, R.: Non-rigid photometric stereo with colored lights. In: Proceedings of the 11th IEEE International Conference on Computer Vision. Rio de Janeiro, Brazil (2007)Google Scholar
  24. 24.
    Hernández, C., Vogiatzis, G., Cipolla, R.: Multiview photometric stereo. IEEE Trans. Pattern Anal. Mach. Intell. 30(3), 548–554 (2008)CrossRefGoogle Scholar
  25. 25.
    Hinkley, D.V.: On the ratio of two correlated normal random variables. Biometrika 56(3), 635–639 (1969)MathSciNetCrossRefzbMATHGoogle Scholar
  26. 26.
    Hoeltgen, L., Quéau, Y., Breuss, M., Radow, G.: Optimised photometric stereo via non-convex variational minimisation. In: Proceedings of the 27th British Machine Vision Conference. York, UK (2016)Google Scholar
  27. 27.
    Horn, B.K.P.: Robot Vision. MIT Press, Cambridge (1986)Google Scholar
  28. 28.
    Horn, B.K.P., Brooks, M.J. (eds.): Shape from Shading. MIT Press, Cambridge (1989)Google Scholar
  29. 29.
    Huang, X., Walton, M., Bearman, G., Cossairt, O.: Near light correction for image relighting and 3D shape recovery. In: Proceedings of the International Congress on Digital Heritage, vol. 1, pp. 215–222. Granada, Spain (2015)Google Scholar
  30. 30.
    Ikeda, O., Duan, Y.: Color photometric stereo for albedo and shape reconstruction. In: Proceedings of the IEEE Winter Conference on Applications of Computer Vision. Lake Placid, USA (2008)Google Scholar
  31. 31.
    Ikehata, S., Wipf, D., Matsushita, Y., Aizawa, K.: Photometric stereo using sparse Bayesian regression for general diffuse surfaces. IEEE Trans. Pattern Anal. Mach. Intell. 36(9), 1816–1831 (2014)CrossRefGoogle Scholar
  32. 32.
    Iwahori, Y., Sugie, H., Ishii, N.: Reconstructing shape from shading images under point light source illumination. In: Proceedings of the 19th International Conference on Pattern Recognition, vol. 1, pp. 83–87. Atlantic City, USA (1990)Google Scholar
  33. 33.
    Jiang, J., Liu, D., Gu, J., Süsstrunk, S.: What is the space of spectral sensitivity functions for digital color cameras? In: Proceedings of the IEEE Winter Conference on Applications of Computer Vision, pp. 168–179. Clearwater, USA (2013)Google Scholar
  34. 34.
    Kolagani, N., Fox, J.S., Blidberg, D.R.: Photometric stereo using point light sources. In: Proceedings of the 9th IEEE International Conference on Robotics and Automation, vol. 2, pp. 1759–1764. Nice, France (1992)Google Scholar
  35. 35.
    Kontsevich, L.L., Petrov, A.P., Vergelskaya, I.S.: Reconstruction of shape from shading in color images. J. Opt. Soc. Am. A 11(3), 1047–1052 (1994)CrossRefGoogle Scholar
  36. 36.
    Koppal, S.J., Narasimhan, S.G.: Novel depth cues from uncalibrated near-field lighting. In: Proceedings of the IEEE International Conference on Computer Vision (2007)Google Scholar
  37. 37.
    Liao, J., Buchholz, B., Thiery, J.M., Bauszat, P., Eisemann, E.: Indoor scene reconstruction using near-light photometric stereo. IEEE Trans. Image Process. 26(3), 1089–1101 (2016)MathSciNetCrossRefGoogle Scholar
  38. 38.
    Logothetis, F., Mecca, R., Cipolla, R.: Semi-calibrated near field photometric stereo. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Honolulu, USA (2017)Google Scholar
  39. 39.
    Logothetis, F., Mecca, R., Quéau, Y., Cipolla, R.: Near-field photometric stereo in ambient light. In: Proceedings of the 27th British Machine Vision Conference. York, UK (2016)Google Scholar
  40. 40.
    McGunnigle, G., Chantler, M.J.: Resolving handwriting from background printing using photometric stereo. Pattern Recogn. 36(8), 1869–1879 (2003)CrossRefGoogle Scholar
  41. 41.
    Mecca, R., Quéau, Y., Logothetis, F., Cipolla, R.: A single lobe photometric stereo approach for heterogeneous material. SIAM J. Imaging Sci. 9(4), 1858–1888 (2016)MathSciNetCrossRefzbMATHGoogle Scholar
  42. 42.
    Mecca, R., Rodolà, E., Cremers, D.: Realistic photometric stereo using partial differential irradiance equation ratios. Comput. Graph. 51, 8–16 (2015)CrossRefGoogle Scholar
  43. 43.
    Mecca, R., Wetzler, A., Bruckstein, A.M., Kimmel, R.: Near Field Photometric Stereo with Point Light Sources. SIAM J. Imaging Sci. 7(4), 2732–2770 (2014)MathSciNetCrossRefzbMATHGoogle Scholar
  44. 44.
    Migita, T., Ogino, S., Shakunaga, T.: Direct bundle estimation for recovery of shape, reflectance property and light position. In: Proceedings of the 10th European Conference on Computer Vision, Lecture Notes in Computer Science, vol. 5304, pp. 412–425. Marseille, France (2008)Google Scholar
  45. 45.
    Moreno, I., Avendaño Alejo, M., Tzonchev, R.I.: Designing light-emitting diode arrays for uniform near-field irradiance. Appl. Opt. 45(10), 2265–2272 (2006)CrossRefGoogle Scholar
  46. 46.
    Moreno, I., Sun, C.C.: Modeling the radiation pattern of LEDs. Opt. Express 16(3), 1808–1819 (2008)CrossRefGoogle Scholar
  47. 47.
    Nie, Y., Song, Z.: A novel photometric stereo method with nonisotropic point light sources. In: Proceedings of the 23rd International Conference on Pattern Recognition, pp. 1737–1742. Cancun, Mexico (2016)Google Scholar
  48. 48.
    Nie, Y., Song, Z., Ji, M., Zhu, L.: A novel calibration method for the photometric stereo system with non-isotropic LED lamps. In: Proceedings of the IEEE Conference on Real-time Computing and Robotics, pp. 289–294. Angkor Wat, Cambodia (2016)Google Scholar
  49. 49.
    Oren, M., Nayar, S.K.: Generalization of the Lambertian model and implications for machine vision. Int. J. Comput. Vis. 14(3), 227–251 (1995)CrossRefGoogle Scholar
  50. 50.
    Ortega, J.M., Rheinboldt, W.C.: Iterative Solution of Nonlinear Equations in Several Variables. Academic Press, New York (1970)zbMATHGoogle Scholar
  51. 51.
    Papadhimitri, T., Favaro, P.: Uncalibrated near-light photometric stereo. In: Proceedings of the 25th British Machine Vision Conference. Nottingham, UK (2014)Google Scholar
  52. 52.
    Pătrăucean, V., Gurdjos, P., Grompone von Gioi, R.: A parameterless line segment and elliptical arc detector with enhanced ellipse fitting. In: Proceedings of the 12th European Conference on Computer Vision, pp. 572–585. Florence, Italy (2012)Google Scholar
  53. 53.
    Pintus, R., Ciortan, I., Giachetti, A., Gobbetti, E.: Practical free-form RTI acquisition with local spot lights. In: Smart Tools and Applications for Graphics. Genova, Italy (2016)Google Scholar
  54. 54.
    Powell, M.W., Sarkar, S., Goldgof, D.: A simple strategy for calibrating the geometry of light sources. IEEE Trans. Pattern Anal. Mach. Intell. 23(9), 1022–1027 (2001)CrossRefGoogle Scholar
  55. 55.
    Quéau, Y., Durou, J.D., Aujol, J.F.: Normal Integration—Part I: A Survey (2016). https://hal.archives-ouvertes.fr/hal-01334349
  56. 56.
    Quéau, Y., Mecca, R., Durou, J.D.: Unbiased photometric stereo for colored surfaces: a variational approach. In: Proceedings of the 29th IEEE Conference on Computer Vision and Pattern Recognition, pp. 4350–4358. Las Vegas, USA (2016)Google Scholar
  57. 57.
    Quéau, Y., Wu, T., Cremers, D.: Semi-calibrated near-light photometric stereo. In: Proceedings of the 6th International Conference on Scale Space and Variational Methods in Computer Vision, Lecture Notes in Computer Science, vol. 10302, pp. 656–668. Kolding, Denmark (2017)Google Scholar
  58. 58.
    Quéau, Y., Wu, T., Lauze, F., Durou, J.D., Cremers, D.: A non-convex variational approach to photometric stereo under inaccurate lighting. In: Proceedings of the 30th IEEE Conference on Computer Vision and Pattern Recognition. Honolulu, USA (2017)Google Scholar
  59. 59.
    Shen, H.L., Cheng, Y.: Calibrating light sources by using a planar mirror. J. Electron. Imaging 20(1), 013002 (2011)Google Scholar
  60. 60.
    Smith, W., Fang, F.: Height from photometric ratio with model-based light source selection. Comput. Vis. Image Underst. 145, 128–138 (2016)CrossRefGoogle Scholar
  61. 61.
    Sun, J., Smith, M., Smith, L., Farooq, A.: Sampling Light Field for Photometric Stereo. Int. J. Comput. Theory Eng. 5(1), 14–18 (2013)CrossRefGoogle Scholar
  62. 62.
    Takai, T., Maki, A., Niinuma, K., Matsuyama, T.: Difference sphere: an approach to near light source estimation. Comput. Vis. Image Underst. 113(9), 966–978 (2009)CrossRefGoogle Scholar
  63. 63.
    Wolke, R., Schwetlick, H.: Iteratively reweighted least squares: algorithms, convergence analysis, and numerical comparisons. SIAM J. Sci. Stat. Comput. 9(5), 907–921 (1988)MathSciNetCrossRefzbMATHGoogle Scholar
  64. 64.
    Woodham, R.J.: Photometric method for determining surface orientation from multiple images. Opt. Eng. 19(1), 139–144 (1980)CrossRefGoogle Scholar
  65. 65.
    Wu, L., Ganesh, A., Shi, B., Matsushita, Y., Wang, Y., Ma, Y.: Robust photometric stereo via low-rank matrix completion and recovery. In: Proceedings of the Asian Conference on Computer Vision, Lecture Notes in Computer Science, vol. 6494, pp. 703–717. Queenstown, New-Zealand (2010)Google Scholar
  66. 66.
    Wu, Z., Li, L.: A line-integration based method for depth recovery from surface normals. Comput. Vis. Graph. Image Process. 43(1), 53–66 (1988)CrossRefGoogle Scholar
  67. 67.
    Xie, L., Song, Z., Jiao, G., Huang, X., Jia, K.: A practical means for calibrating an LED-based photometric stereo system. Opt. Lasers Eng. 64, 42–50 (2015)CrossRefGoogle Scholar
  68. 68.
    Xie, W., Dai, C., Wang, C.C.L.: Photometric stereo with near point lighting: a solution by mesh deformation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Boston, USA (2015)Google Scholar
  69. 69.
    Yeh, C.K., Matsuda, N., Huang, X., Li, F., Walton, M., Cossairt, O.: A streamlined photometric stereo framework for cultural heritage. In: Proceedings of the 14th European Conference on Computer Vision, pp. 738–752. Amsterdam, The Netherlands (2016)Google Scholar

Copyright information

© Springer Science+Business Media, LLC 2017

Authors and Affiliations

  1. 1.Department of Computer ScienceTechnical University of MunichGarchingGermany
  2. 2.IRIT, UMR CNRS 5505Université de ToulouseToulouseFrance
  3. 3.Department of Computer ScienceUniversity of CopenhagenCopenhagenDenmark

Personalised recommendations