Journal of Mathematical Imaging and Vision

, Volume 61, Issue 1, pp 84–105 | Cite as

Optimisation of Classic Photometric Stereo by Non-convex Variational Minimisation

  • Georg RadowEmail author
  • Laurent Hoeltgen
  • Yvain Quéau
  • Michael Breuß


Estimating shape and appearance of a three-dimensional object from a given set of images is a classic research topic that is still actively pursued. Among the various techniques available, photometric stereo is distinguished by the assumption that the underlying input images are taken from the same point of view but under different lighting conditions. The most common techniques are conceptually close to the classic photometric stereo problem, meaning that the modelling encompasses a linearisation step and that the shape information is computed in terms of surface normals. In this work, instead of linearising we aim to stick to the original formulation of the photometric stereo problem, and we propose to minimise a much more natural objective function, namely the reprojection error in terms of depth. Minimising the resulting non-trivial variational model for photometric stereo allows to recover the depth of the photographed scene directly. As a solving strategy, we follow an approach based on a recently published optimisation scheme for non-convex and non-smooth cost functions. The main contributions of our paper are of theoretical nature. A technical novelty in our framework is the usage of matrix differential calculus. We supplement our approach by a detailed convergence analysis of the resulting optimisation algorithm and discuss possibilities to ease the computational complexity. At hand of an experimental evaluation we discuss important properties of the method. Overall, our strategy achieves more accurate results than other approaches that rely on the classic photometric stereo assumptions. The experiments also highlight some practical aspects of the underlying optimisation algorithm that may be of interest in a more general context.


Non-convex minimisation Computer vision Photometric stereo 


  1. 1.
    Bähr, M., Breuß, M., Quéau, Y., Boroujerdi, A.S., Durou, J.D.: Fast and accurate surface normal integration on non-rectangular domains. Comput. Vis. Media 3, 107–129 (2017)CrossRefGoogle Scholar
  2. 2.
    Bartal, O., Ofir, N., Lipman, Y., Basri, R.: Photometric stereo by hemispherical metric embedding. J. Math. Imaging Vis. 60(2), 148–162 (2018)Google Scholar
  3. 3.
    Basri, R., Jacobs, D., Kemelmacher, I.: Photometric stereo with general, unknown lighting. Int. J. Comput. Vis. 72, 239–257 (2007)CrossRefGoogle Scholar
  4. 4.
    Chabrowski, J., Kewei, Z.: On variational approach to photometric stereo. Annales de l’Institut Henri Poincaré (C) Analyse non linéaire 10(4), 363–375 (1993)MathSciNetCrossRefzbMATHGoogle Scholar
  5. 5.
    Clark, J.J.: Active photometric stereo. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 29–34 (1992)Google Scholar
  6. 6.
    Durou, J.D., Aujol, J.F., Courteille, F.: Integrating the normal field of a surface in the presence of discontinuities. In: Energy Minimization Methods in Computer Vision and Pattern Recognition (EMMCVPR). Lecture Notes in Computer Science, vol. 5681, pp. 261–273. Springer (2009)Google Scholar
  7. 7.
    Gotardo, P.F.U., Simon, T., Sheikh, Y., Matthews, I.: Photogeometric scene flow for high-detail dynamic 3D reconstruction. In: Proceedings of IEEE International Conference on Computer Vision (ICCV), pp. 846–854 (2015)Google Scholar
  8. 8.
    Harker, M., O’Leary, P.: Regularized reconstruction of a surface from its measured gradient field. J. Math. Imaging Vis. 51(1), 46–70 (2015)CrossRefzbMATHGoogle Scholar
  9. 9.
    Hinkley, D.V.: On the ratio of two correlated normal random variables. Biometrika 56(3), 635–639 (1969)MathSciNetCrossRefzbMATHGoogle Scholar
  10. 10.
    Hoeltgen, L., Quéau, Y., Breuß, M., Radow, G.: Optimised photometric stereo via non-convex variational minimisation. In: British Machine Vision Conference (BMVC) (2016).
  11. 11.
    Horn, B.K.P.: Robot Vision. The MIT Press, Cambridge (1986)Google Scholar
  12. 12.
    Horn, B.K.P., Woodham, R.J., Silver, W.M.: Determining shape and reflectance using multiple images. Technical Report MIT AITR-490, MIT (1978)Google Scholar
  13. 13.
    Horn, R.A., Johnson, C.R.: Topics in Matrix Analysis. Cambridge University Press, Cambridge (1994)zbMATHGoogle Scholar
  14. 14.
    Ikehata, S., Wipf, D., Matsushita, Y., Aizawa, K.: Photometric stereo using sparse Bayesian regression for general diffuse surfaces. IEEE Trans. Pattern Anal. Mach. Intell. 36(9), 1816–1831 (2014)CrossRefGoogle Scholar
  15. 15.
    Ju, Y.C., Tozza, S., Breuß, M., Bruhn, A., Kleefeld, A.: Generalised perspective shape from shading with Oren–Nayar reflectance. In: British Machine Vision Conference (2013).
  16. 16.
    Khanian, M., Boroujerdi, A.S., Breuß, M.: Photometric stereo for strong specular highlights. Comput. Vis. Media 4(1), 83–102 (2018)Google Scholar
  17. 17.
    Kozera, R.: Existence and uniqueness in photometric stereo. Appl. Math. Comput. 44, 1–103 (1991)MathSciNetzbMATHGoogle Scholar
  18. 18.
    Lambert, J.H.: Photometria. Klett, Augsburg (1760)Google Scholar
  19. 19.
    Magnus, J.R., Neudecker, H.: Matrix differential calculus with applications to simple, Hadamard, and Kronecker products. J. Math. Psychol. 29, 474–492 (1985)MathSciNetCrossRefzbMATHGoogle Scholar
  20. 20.
    Magnus, J.R., Neudecker, H.: Matrix Differential Calculus with Applications in Statistics and Econometrics, 3rd edn. Wiley, New York (2007)zbMATHGoogle Scholar
  21. 21.
    Mecca, R., Quéau, Y., Logothetis, F., Cipolla, R.: A single-lobe photometric stereo approach for heterogeneous material. SIAM J. Imaging Sci. 9(4), 1858–1888 (2016)MathSciNetCrossRefzbMATHGoogle Scholar
  22. 22.
    Mecca, R., Rodolà, E., Cremers, D.: Realistic photometric stereo using partial differential irradiance equation ratios. Comput. Graph. 51, 8–16 (2015)CrossRefGoogle Scholar
  23. 23.
    Moreau, J.J.: Proximité et dualité dans un espace Hilbertien. Bulletin de la Société Mathématique de France 93, 273–299 (1965)MathSciNetCrossRefzbMATHGoogle Scholar
  24. 24.
    Ochs, P.: Unifying abstract inexact convergence theorems for descent methods and block coordinate variable metric iPiano. Saarland University, Technical report (2016)Google Scholar
  25. 25.
    Ochs, P., Chen, Y., Brox, T., Pock, T.: iPiano: Inertial proximal algorithm for non-convex optimization. SIAM J. Imaging Sci. 7(2), 1388–1419 (2014)MathSciNetCrossRefzbMATHGoogle Scholar
  26. 26.
    Onn, R., Bruckstein, A.: Integrability disambiguates surface recovery in two-image photometric stereo. Int. J. Comput. Vis. 5, 105–113 (1990)CrossRefGoogle Scholar
  27. 27.
    Ortega, J.M., Rheinboldt, W.C.: Iterative Solutions of Nonlinear Equations in Several Variables. Academic, New York (1970)zbMATHGoogle Scholar
  28. 28.
    Papadhimitri, T., Favaro, P.: Uncalibrated near-light photometric stereo. In: British Machine Vision Conference (2014).
  29. 29.
    Petersen, K.B., Pedersen, M.S.: The matrix cookbook (2012).
  30. 30.
    Pollock, D.S.G.: Tensor products and matrix differential calculus. Linear Algebra Appl. 67, 169–193 (1985)MathSciNetCrossRefzbMATHGoogle Scholar
  31. 31.
    Quéau, Y., Durix, B., Wu, T., Cremers, D., Lauze, F., Durou, J.D.: LED-based photometric stereo: modeling, calibration and numerical solution. J. Math. Imaging Vis. 60(3), 313–340 (2018)Google Scholar
  32. 32.
    Quéau, Y., Lauze, F., Durou, J.D.: A \(L^1\)-TV algorithm for robust perspective photometric stereo with spatially-varying lightings. In: Scale Space and Variational Methods in Computer Vision (SSVM). Lecture Notes in Computer Science, vol. 9087, pp. 498–510 (2015)Google Scholar
  33. 33.
    Quéau, Y., Mecca, R., Durou, J.D.: Unbiased photometric stereo for colored surfaces: a variational approach. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4359–4368 (2016)Google Scholar
  34. 34.
    Quéau, Y., Wu, T., Lauze, F., Durou, J.D., Cremers, D.: A non-convex variational approach to photometric stereo under inaccurate lighting. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 99–108 (2017)Google Scholar
  35. 35.
    Reddy, D., Agrawal, A., Chellappa, R.: Enforcing integrability by error correction using \(l_1\)-minimization. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2350–2357 (2009)Google Scholar
  36. 36.
    Shi, B., Mo, Z., Wu, Z., Duan, D., Yeung, S.K., Tan, P.: A benchmark dataset and evaluation for non-Lambertian and uncalibrated photometric stereo. IEEE Trans. Pattern Anal. Mach. Intell. (to appear).
  37. 37.
    Smith, W., Fang, F.: Height from photometric ratio with model-based light source selection. Comput. Vis. Image Underst. 145, 128–138 (2016)CrossRefGoogle Scholar
  38. 38.
    Tozza, S., Mecca, R., Duocastella, M., Del Bue, A.: Direct differential photometric stereo shape recovery of diffuse and specular surfaces. J. Math. Imaging Vis. 56(1), 57–76 (2016)MathSciNetCrossRefzbMATHGoogle Scholar
  39. 39.
    Wöhler, C.: 3D Computer Vision. Springer, Berlin (2013)CrossRefGoogle Scholar
  40. 40.
    Woodham, R.J.: Photometric stereo: a reflectance map technique for determining surface orientation from a single view. In: Proceedings of the 22nd SPIE Annual Technical Symposium. Proceedings of the International Society for Optical Engineering, vol. 155, pp. 136–143 (1978)Google Scholar
  41. 41.
    Woodham, R.J.: Photometric method for determining surface orientation from multiple images. Opt. Eng. 19(1), 134–144 (1980)CrossRefGoogle Scholar
  42. 42.
    Wu, L., Ganesh, A., Shi, B., Matsushita, Y., Wang, Y., Ma, Y.: Robust photometric stereo via low-rank matrix completion and recovery. In: Asian Conference on Computer Vision (ACCV). Lecture Notes in Computer Science, vol. 6494, pp. 703–717. Springer, Berlin (2010)Google Scholar
  43. 43.
    Zeisl, B., Zach, C., Pollefeys, M.: Variational regularization and fusion of surface normal maps. In: IEEE International Conference on 3D Vision (3DV), pp. 601–608 (2014)Google Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2018

Authors and Affiliations

  1. 1.Chair for Applied MathematicsBTU Cottbus-SenftenbergCottbusGermany
  2. 2.Vision Lab. ISEN BrestL@bISEN Yncrea OuestBrestFrance

Personalised recommendations