Skip to main content
SpringerLink
Log in

Optimisation of Classic Photometric Stereo by Non-convex Variational Minimisation

  • Published:
Journal of Mathematical Imaging and Vision Aims and scope Submit manuscript

Abstract

Estimating shape and appearance of a three-dimensional object from a given set of images is a classic research topic that is still actively pursued. Among the various techniques available, photometric stereo is distinguished by the assumption that the underlying input images are taken from the same point of view but under different lighting conditions. The most common techniques are conceptually close to the classic photometric stereo problem, meaning that the modelling encompasses a linearisation step and that the shape information is computed in terms of surface normals. In this work, instead of linearising we aim to stick to the original formulation of the photometric stereo problem, and we propose to minimise a much more natural objective function, namely the reprojection error in terms of depth. Minimising the resulting non-trivial variational model for photometric stereo allows to recover the depth of the photographed scene directly. As a solving strategy, we follow an approach based on a recently published optimisation scheme for non-convex and non-smooth cost functions. The main contributions of our paper are of theoretical nature. A technical novelty in our framework is the usage of matrix differential calculus. We supplement our approach by a detailed convergence analysis of the resulting optimisation algorithm and discuss possibilities to ease the computational complexity. At hand of an experimental evaluation we discuss important properties of the method. Overall, our strategy achieves more accurate results than other approaches that rely on the classic photometric stereo assumptions. The experiments also highlight some practical aspects of the underlying optimisation algorithm that may be of interest in a more general context.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

Notes

  1. https://github.com/yqueau/optimized_ps.

References

  1. Bähr, M., Breuß, M., Quéau, Y., Boroujerdi, A.S., Durou, J.D.: Fast and accurate surface normal integration on non-rectangular domains. Comput. Vis. Media 3, 107–129 (2017)

    Article  Google Scholar 

  2. Bartal, O., Ofir, N., Lipman, Y., Basri, R.: Photometric stereo by hemispherical metric embedding. J. Math. Imaging Vis. 60(2), 148–162 (2018)

  3. Basri, R., Jacobs, D., Kemelmacher, I.: Photometric stereo with general, unknown lighting. Int. J. Comput. Vis. 72, 239–257 (2007)

    Article  Google Scholar 

  4. Chabrowski, J., Kewei, Z.: On variational approach to photometric stereo. Annales de l’Institut Henri Poincaré (C) Analyse non linéaire 10(4), 363–375 (1993)

    Article  MathSciNet  MATH  Google Scholar 

  5. Clark, J.J.: Active photometric stereo. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 29–34 (1992)

  6. Durou, J.D., Aujol, J.F., Courteille, F.: Integrating the normal field of a surface in the presence of discontinuities. In: Energy Minimization Methods in Computer Vision and Pattern Recognition (EMMCVPR). Lecture Notes in Computer Science, vol. 5681, pp. 261–273. Springer (2009)

  7. Gotardo, P.F.U., Simon, T., Sheikh, Y., Matthews, I.: Photogeometric scene flow for high-detail dynamic 3D reconstruction. In: Proceedings of IEEE International Conference on Computer Vision (ICCV), pp. 846–854 (2015)

  8. Harker, M., O’Leary, P.: Regularized reconstruction of a surface from its measured gradient field. J. Math. Imaging Vis. 51(1), 46–70 (2015)

    Article  MATH  Google Scholar 

  9. Hinkley, D.V.: On the ratio of two correlated normal random variables. Biometrika 56(3), 635–639 (1969)

    Article  MathSciNet  MATH  Google Scholar 

  10. Hoeltgen, L., Quéau, Y., Breuß, M., Radow, G.: Optimised photometric stereo via non-convex variational minimisation. In: British Machine Vision Conference (BMVC) (2016). https://doi.org/10.5244/C.30.36

  11. Horn, B.K.P.: Robot Vision. The MIT Press, Cambridge (1986)

    Google Scholar 

  12. Horn, B.K.P., Woodham, R.J., Silver, W.M.: Determining shape and reflectance using multiple images. Technical Report MIT AITR-490, MIT (1978)

  13. Horn, R.A., Johnson, C.R.: Topics in Matrix Analysis. Cambridge University Press, Cambridge (1994)

    MATH  Google Scholar 

  14. Ikehata, S., Wipf, D., Matsushita, Y., Aizawa, K.: Photometric stereo using sparse Bayesian regression for general diffuse surfaces. IEEE Trans. Pattern Anal. Mach. Intell. 36(9), 1816–1831 (2014)

    Article  Google Scholar 

  15. Ju, Y.C., Tozza, S., Breuß, M., Bruhn, A., Kleefeld, A.: Generalised perspective shape from shading with Oren–Nayar reflectance. In: British Machine Vision Conference (2013). http://doi.org/10.5244/C.27.42

  16. Khanian, M., Boroujerdi, A.S., Breuß, M.: Photometric stereo for strong specular highlights. Comput. Vis. Media 4(1), 83–102 (2018)

  17. Kozera, R.: Existence and uniqueness in photometric stereo. Appl. Math. Comput. 44, 1–103 (1991)

    MathSciNet  MATH  Google Scholar 

  18. Lambert, J.H.: Photometria. Klett, Augsburg (1760)

    Google Scholar 

  19. Magnus, J.R., Neudecker, H.: Matrix differential calculus with applications to simple, Hadamard, and Kronecker products. J. Math. Psychol. 29, 474–492 (1985)

    Article  MathSciNet  MATH  Google Scholar 

  20. Magnus, J.R., Neudecker, H.: Matrix Differential Calculus with Applications in Statistics and Econometrics, 3rd edn. Wiley, New York (2007)

    MATH  Google Scholar 

  21. Mecca, R., Quéau, Y., Logothetis, F., Cipolla, R.: A single-lobe photometric stereo approach for heterogeneous material. SIAM J. Imaging Sci. 9(4), 1858–1888 (2016)

    Article  MathSciNet  MATH  Google Scholar 

  22. Mecca, R., Rodolà, E., Cremers, D.: Realistic photometric stereo using partial differential irradiance equation ratios. Comput. Graph. 51, 8–16 (2015)

    Article  Google Scholar 

  23. Moreau, J.J.: Proximité et dualité dans un espace Hilbertien. Bulletin de la Société Mathématique de France 93, 273–299 (1965)

    Article  MathSciNet  MATH  Google Scholar 

  24. Ochs, P.: Unifying abstract inexact convergence theorems for descent methods and block coordinate variable metric iPiano. Saarland University, Technical report (2016)

  25. Ochs, P., Chen, Y., Brox, T., Pock, T.: iPiano: Inertial proximal algorithm for non-convex optimization. SIAM J. Imaging Sci. 7(2), 1388–1419 (2014)

    Article  MathSciNet  MATH  Google Scholar 

  26. Onn, R., Bruckstein, A.: Integrability disambiguates surface recovery in two-image photometric stereo. Int. J. Comput. Vis. 5, 105–113 (1990)

    Article  Google Scholar 

  27. Ortega, J.M., Rheinboldt, W.C.: Iterative Solutions of Nonlinear Equations in Several Variables. Academic, New York (1970)

    MATH  Google Scholar 

  28. Papadhimitri, T., Favaro, P.: Uncalibrated near-light photometric stereo. In: British Machine Vision Conference (2014). http://doi.org/10.5244/C.28.128

  29. Petersen, K.B., Pedersen, M.S.: The matrix cookbook (2012). https://www.math.uwaterloo.ca/~hwolkowi/matrixcookbook.pdf

  30. Pollock, D.S.G.: Tensor products and matrix differential calculus. Linear Algebra Appl. 67, 169–193 (1985)

    Article  MathSciNet  MATH  Google Scholar 

  31. Quéau, Y., Durix, B., Wu, T., Cremers, D., Lauze, F., Durou, J.D.: LED-based photometric stereo: modeling, calibration and numerical solution. J. Math. Imaging Vis. 60(3), 313–340 (2018)

  32. Quéau, Y., Lauze, F., Durou, J.D.: A \(L^1\)-TV algorithm for robust perspective photometric stereo with spatially-varying lightings. In: Scale Space and Variational Methods in Computer Vision (SSVM). Lecture Notes in Computer Science, vol. 9087, pp. 498–510 (2015)

  33. Quéau, Y., Mecca, R., Durou, J.D.: Unbiased photometric stereo for colored surfaces: a variational approach. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4359–4368 (2016)

  34. Quéau, Y., Wu, T., Lauze, F., Durou, J.D., Cremers, D.: A non-convex variational approach to photometric stereo under inaccurate lighting. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 99–108 (2017)

  35. Reddy, D., Agrawal, A., Chellappa, R.: Enforcing integrability by error correction using \(l_1\)-minimization. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2350–2357 (2009)

  36. Shi, B., Mo, Z., Wu, Z., Duan, D., Yeung, S.K., Tan, P.: A benchmark dataset and evaluation for non-Lambertian and uncalibrated photometric stereo. IEEE Trans. Pattern Anal. Mach. Intell. (to appear). https://doi.org/10.1109/tpami.2018.2799222

  37. Smith, W., Fang, F.: Height from photometric ratio with model-based light source selection. Comput. Vis. Image Underst. 145, 128–138 (2016)

    Article  Google Scholar 

  38. Tozza, S., Mecca, R., Duocastella, M., Del Bue, A.: Direct differential photometric stereo shape recovery of diffuse and specular surfaces. J. Math. Imaging Vis. 56(1), 57–76 (2016)

    Article  MathSciNet  MATH  Google Scholar 

  39. Wöhler, C.: 3D Computer Vision. Springer, Berlin (2013)

    Book  Google Scholar 

  40. Woodham, R.J.: Photometric stereo: a reflectance map technique for determining surface orientation from a single view. In: Proceedings of the 22nd SPIE Annual Technical Symposium. Proceedings of the International Society for Optical Engineering, vol. 155, pp. 136–143 (1978)

  41. Woodham, R.J.: Photometric method for determining surface orientation from multiple images. Opt. Eng. 19(1), 134–144 (1980)

    Article  Google Scholar 

  42. Wu, L., Ganesh, A., Shi, B., Matsushita, Y., Wang, Y., Ma, Y.: Robust photometric stereo via low-rank matrix completion and recovery. In: Asian Conference on Computer Vision (ACCV). Lecture Notes in Computer Science, vol. 6494, pp. 703–717. Springer, Berlin (2010)

  43. Zeisl, B., Zach, C., Pollefeys, M.: Variational regularization and fusion of surface normal maps. In: IEEE International Conference on 3D Vision (3DV), pp. 601–608 (2014)

Download references

Author information

Authors and Affiliations

  1. Chair for Applied Mathematics, BTU Cottbus-Senftenberg, Cottbus, Germany

    Georg Radow, Laurent Hoeltgen & Michael Breuß

  2. Vision Lab. ISEN Brest, L@bISEN Yncrea Ouest, Brest, France

    Yvain Quéau

Authors
  1. Georg Radow
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Laurent Hoeltgen
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yvain Quéau
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Michael Breuß
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Georg Radow.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Radow, G., Hoeltgen, L., Quéau, Y. et al. Optimisation of Classic Photometric Stereo by Non-convex Variational Minimisation. J Math Imaging Vis 61, 84–105 (2019). https://doi.org/10.1007/s10851-018-0828-7

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10851-018-0828-7

Keywords

Search

Navigation