Journal of Mathematical Imaging and Vision

, Volume 60, Issue 4, pp 609–632 | Cite as

Variational Methods for Normal Integration

  • Yvain QuéauEmail author
  • Jean-Denis Durou
  • Jean-François Aujol


The need for an efficient method of integration of a dense normal field is inspired by several computer vision tasks, such as shape-from-shading, photometric stereo, deflectometry. Inspired by edge-preserving methods from image processing, we study in this paper several variational approaches for normal integration, with a focus on non-rectangular domains, free boundary and depth discontinuities. We first introduce a new discretization for quadratic integration, which is designed to ensure both fast recovery and the ability to handle non-rectangular domains with a free boundary. Yet, with this solver, discontinuous surfaces can be handled only if the scene is first segmented into pieces without discontinuity. Hence, we then discuss several discontinuity-preserving strategies. Those inspired, respectively, by the Mumford–Shah segmentation method and by anisotropic diffusion, are shown to be the most effective for recovering discontinuities.


3D-reconstruction Integration Normal field Gradient field Variational methods Photometric stereo Shape-from-shading 



We are grateful to the reviewers for the constructive discussion during the reviewing process.


  1. 1.
    Agrawal, A., Raskar, R., Chellappa, R.: What is the range of surface reconstructions from a gradient field? In: Proceedings of the 9th European Conference on Computer Vision (vol. 1), Lecture Notes in Computer Science, vol. 3951, pp. 578–591. Graz (2006)Google Scholar
  2. 2.
    Ambrosio, L., Tortorelli, V.M.: Approximation of functionals depending on jumps by elliptic functionals via \(\Gamma \)-convergence. Commun. Pure Appl. Math. 43, 999–1036 (1990)MathSciNetzbMATHCrossRefGoogle Scholar
  3. 3.
    Attouch, H., Buttazzo, G., Michaille, G.: Variational Analysis in Sobolev and BV Spaces: Applications to PDEs and Optimization, 2nd edn. SIAM (2014)Google Scholar
  4. 4.
    Aubert, G., Kornprobst, P. (eds.): Mathematical problems in image processing. In: Applied Mathematical Sciences, vol. 147, 2nd edn. Springer, Berlin (2006)Google Scholar
  5. 5.
    Badri, H., Yahia, H., Aboutajdine, D.: Robust surface reconstruction via triple sparsity. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2291–2298. Columbus (2014)Google Scholar
  6. 6.
    Bähr, M., Breuß, M., Quéau, Y., Bouroujerdi, A.S., Durou, J.D.: Fast and accurate surface normal integration on non-rectangular domains. Comput. Vis. Media 3, 107–129 (2017)CrossRefGoogle Scholar
  7. 7.
    Beck, A., Teboulle, M.: A fast iterative shrinkage-thresholding algorithm for linear inverse problems. SIAM J. Imaging Sci. 2(1), 183–202 (2009)MathSciNetzbMATHCrossRefGoogle Scholar
  8. 8.
    Belhachmi, B., Bucur, D., Burgeth, B., Weickert, J.: How to choose interpolation data in images. SIAM J. Appl. Math. 70(1), 333–352 (2009)MathSciNetzbMATHCrossRefGoogle Scholar
  9. 9.
    Boyd, S., Parikh, N., Chu, E., Peleato, B., Eckstein, J.: Distributed optimization and statistical learning via the alternating direction method of multipliers. Found. Trends Mach. Learn. 3(1), 1–122 (2011)zbMATHCrossRefGoogle Scholar
  10. 10.
    Bredies, K., Holler, M.: A TGV-based framework for variational image decompression, zooming, and reconstruction. Part I: analytics. SIAM J. Imaging Sci. 8(4), 2814–2850 (2015)MathSciNetzbMATHCrossRefGoogle Scholar
  11. 11.
    Catté, F., Lions, P.L., Morel, J.M., Coll, T.: Image selective smoothing and edge detection by nonlinear diffusion. SIAM J. Numer. Anal. 29(1), 182–193 (1992)MathSciNetzbMATHCrossRefGoogle Scholar
  12. 12.
    Chambolle, A.: Image segmentation by variational methods: Mumford and Shah functional and the discrete approximation. SIAM J. Appl. Math. 55(3), 827–863 (1995)MathSciNetzbMATHCrossRefGoogle Scholar
  13. 13.
    Chambolle, A., Pock, T.: A first-order primal-dual algorithm for convex problems with applications to imaging. J. Math. Imaging Vis. 40(1), 120–145 (2010)MathSciNetzbMATHCrossRefGoogle Scholar
  14. 14.
    Chan, T.F., Vese, L.A.: Active contours without edges. IEEE Trans. Image Process. 10(2), 266–277 (2001)zbMATHCrossRefGoogle Scholar
  15. 15.
    Chang, J.Y., Lee, K.M., Lee, S.U.: Multiview normal field integration using level set methods. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Workshop on Beyond Multiview Geometry: Robust Estimation and Organization of Shapes from Multiple Cues. Minneapolis (2007)Google Scholar
  16. 16.
    Charbonnier, P., Blanc-Féraud, L., Aubert, G., Barlaud, M.: Deterministic edge-preserving regularization in computed imaging. IEEE Trans. Image Process. 6(2), 298–311 (1997)CrossRefGoogle Scholar
  17. 17.
    Chen, S.S., Donoho, D.L., Saunders, M.A.: Atomic decomposition by basis pursuit. SIAM J. Sci. Comput. 20(1), 33–61 (1998)MathSciNetzbMATHCrossRefGoogle Scholar
  18. 18.
    Du, Z., Robles-Kelly, A., Lu, F.: Robust surface reconstruction from gradient field using the L1 norm. In: Proceedings of the 9th Biennial Conference of the Australian Pattern Recognition Society on Digital Image Computing Techniques and Applications, pp. 203–209. Glenelg (2007)Google Scholar
  19. 19.
    Durou, J.D., Aujol, J.F., Courteille, F.: Integration of a normal field in the presence of discontinuities. In: Proceedings of the 7th International Workshop on Energy Minimization Methods in Computer Vision and Pattern Recognition, Lecture Notes in Computer Science, vol. 5681, pp. 261–273. Bonn (2009)Google Scholar
  20. 20.
    Durou, J.D., Courteille, F.: Integration of a normal field without boundary condition. In: Proceedings of the 11th IEEE International Conference on Computer Vision, 1st Workshop on Photometric Analysis for Computer Vision. Rio de Janeiro (2007)Google Scholar
  21. 21.
    Gabay, D., Mercier, B.: A dual algorithm for the solution of nonlinear variational problems via finite element approximation. Comput. Math. Appl. 2(1), 17–40 (1976)zbMATHCrossRefGoogle Scholar
  22. 22.
    Geman, D., Reynolds, G.: Constrained restoration and recovery of discontinuities. IEEE Trans. Pattern Anal. Mach. Intell. 14(3), 367–383 (1992)CrossRefGoogle Scholar
  23. 23.
    Goldstein, T., O’Donoghue, B., Setzer, S., Baraniuk, R.: Fast alternating direction optimization methods. SIAM J. Imaging Sci. 7(3), 1588–1623 (2014)MathSciNetzbMATHCrossRefGoogle Scholar
  24. 24.
    Goldstein, T., Osher, S.: The split Bregman method for L1-regularized problems. SIAM J. Imaging Sci. 2(2), 323–343 (2009)MathSciNetzbMATHCrossRefGoogle Scholar
  25. 25.
    Haque, S.M., Chatterjee, A., Govindu, V.M.: High quality photometric reconstruction using a depth camera. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2283–2290. Columbus (2014)Google Scholar
  26. 26.
    Harker, M., O’Leary, P.: Regularized reconstruction of a surface from its measured gradient field. J. Math. Imaging Vis. 51(1), 46–70 (2015)zbMATHCrossRefGoogle Scholar
  27. 27.
    Hayya, J., Armstrong, D., Gressis, N.: A note on the ratio of two normally distributed variables. Manag. Sci. 21(11), 1338–1341 (1975)zbMATHCrossRefGoogle Scholar
  28. 28.
    Hoeltgen, L., Setzer, S., Weickert, J.: An optimal control approach to find sparse data for Laplace interpolation. In: Proceedings of the 9th International Workshop on Energy Minimization Methods in Computer Vision and Pattern Recognition, Lecture Notes in Computer Science, vol. 8081, pp. 151–164. Lund (2013)Google Scholar
  29. 29.
    Horn, B.K.P., Brooks, M.J.: The variational approach to shape from shading. Comput. Vis. Graph. Image Process. 33(2), 174–208 (1986)zbMATHCrossRefGoogle Scholar
  30. 30.
    Horovitz, I., Kiryati, N.: Depth from gradient fields and control points: bias correction in photometric stereo. Image Vis. Comput. 22(9), 681–694 (2004)CrossRefGoogle Scholar
  31. 31.
    Ikehata, S., Wipf, D., Matsushita, Y., Aizawa, K.: Photometric stereo using sparse Bayesian regression for general diffuse surfaces. IEEE Trans. Pattern Anal. Mach. Intell. 36(9), 1816–1831 (2014)CrossRefGoogle Scholar
  32. 32.
    Kadambi, A., Taamazyan, V., Shi, B., Raskar, R.: Polarized 3D: high-quality depth sensing with polarization cues. In: Proceedings of the 15th IEEE International Conference on Computer Vision, pp. 3370–3378. Santiago (2015)Google Scholar
  33. 33.
    Kimmel, R., Yavneh, I.: An algebraic multigrid approach for image analysis. SIAM J. Sci. Comput. 24(4), 1218–1231 (2003)MathSciNetzbMATHCrossRefGoogle Scholar
  34. 34.
    Kornprobst, P., Aubert, G.: Image sequence analysis via partial differential equations. J. Math. Imaging Vis. 11(1), 5–26 (1999)MathSciNetCrossRefGoogle Scholar
  35. 35.
    Koutis, I., Miller, G.L., Peng, R.: A nearly-m log n time solver for SDD linear systems. In: Proceedings of the IEEE Annual Symposium on Foundations of Computer Science, pp. 590–598. Palm Springs (2011)Google Scholar
  36. 36.
    Lanza, A., Morigi, S., Sgallari, F.: Convex image denoising via non-convex regularization with parameter selection. J. Math. Imaging Vis. 56(2), 195–220 (2016)MathSciNetzbMATHCrossRefGoogle Scholar
  37. 37.
    Mumford, D., Shah, J.: Optimal approximations by piecewise smooth functions and associated variational problems. Commun. Pure Appl. Math. 42(5), 577–685 (1989)MathSciNetzbMATHCrossRefGoogle Scholar
  38. 38.
    Nikolova, M.: Local strong homogeneity of a regularized estimator. SIAM J. Appl. Math. 61(2), 633–658 (2000)MathSciNetzbMATHCrossRefGoogle Scholar
  39. 39.
    Noakes, L., Kozera, R.: Nonlinearities and noise reduction in 3-source photometric stereo. J. Math. Imaging Vis. 18(2), 119–127 (2003)MathSciNetzbMATHCrossRefGoogle Scholar
  40. 40.
    Ochs, P., Brox, T., Pock, T.: iPiasco: inertial proximal algorithm for strongly convex optimization. J. Math. Imaging Vis. 53(2), 171–181 (2015)MathSciNetzbMATHCrossRefGoogle Scholar
  41. 41.
    Ochs, P., Chen, Y., Brox, T., Pock, T.: iPiano: inertial proximal algorithm for nonconvex optimization. SIAM J. Imaging Sci. 7(2), 1388–1419 (2014)MathSciNetzbMATHCrossRefGoogle Scholar
  42. 42.
    Or-el, R., Rosman, G., Wetzler, A., Kimmel, R., Bruckstein, A.M.: RGBD-Fusion: real-time high precision depth recovery. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5407–5416. Boston (2015)Google Scholar
  43. 43.
    Pérez, P., Gangnet, M., Blake, A.: Poisson image editing. ACM Trans. Graph. 22(3), 313–318 (2003)CrossRefGoogle Scholar
  44. 44.
    Perona, P., Malik, J.: Scale-space and edge detection using anisotropic diffusion. IEEE Trans. Pattern Anal. Mach. Intell. 12(7), 629–639 (1990)CrossRefGoogle Scholar
  45. 45.
    Peter, P., Hoffmann, S., Nedwed, F., Hoeltgen, L., Weickert, J.: Evaluating the true potential of diffusion-based inpainting in a compression context. Signal Process. Image Commun. 46, 40–53 (2016)CrossRefGoogle Scholar
  46. 46.
    Pock, T., Cremers, D., Bischof, H., Chambolle, A.: An algorithm for minimizing the Mumford–Shah functional. In: Proceedings of the 12th IEEE International Conference on Computer Vision, pp. 1133–1140. Kyoto (2009)Google Scholar
  47. 47.
    Quéau, Y., Durou, J.D.: Edge-preserving integration of a normal field: weighted least squares, TV and L1 approaches. In: Proceedings of the 5th International Conference on Scale Space and Variational Methods in Computer Vision, Lecture Notes in Computer Science, vol. 9087, pp. 576–588. Lège Cap-Ferret (2015)Google Scholar
  48. 48.
    Quéau, Y., Durou, J.D., Aujol, J.F.: Normal integration: a survey. J. Math. Imaging Vis. (2017).
  49. 49.
    Rudin, L.I., Osher, S., Fatemi, E.: Nonlinear total variation based noise removal algorithms. Phys. D Nonlinear Phenom. 60(1–4), 259–268 (1992)MathSciNetzbMATHCrossRefGoogle Scholar
  50. 50.
    Saracchini, R.F.V., Stolfi, J., Leitão, H.C.G., Atkinson, G.A., Smith, M.L.: A robust multi-scale integration method to obtain the depth from gradient maps. Comput. Vis. Image Underst. 116(8), 882–895 (2012)CrossRefGoogle Scholar
  51. 51.
    Shefi, R., Teboulle, M.: Rate of convergence analysis of decomposition methods based on the proximal method of multipliers for convex minimization. SIAM J. Optim. 24(1), 269–297 (2014)MathSciNetzbMATHCrossRefGoogle Scholar
  52. 52.
    Shi, B., Wu, Z., Mo, Z., Duan, D., Yeung, S.K., Tan, P.: A benchmark dataset and evaluation for non-Lambertian and uncalibrated photometric stereo. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas (2016)Google Scholar
  53. 53.
    Simchony, T., Chellappa, R., Shao, M.: Direct analytical methods for solving Poisson equations in computer vision problems. IEEE Trans. Pattern Anal. Mach. Intell. 12(5), 435–446 (1990)zbMATHCrossRefGoogle Scholar
  54. 54.
    Weickert, J.: Anisotropic Diffusion in Image Processing. Teubner, Stuttgart (1998)zbMATHGoogle Scholar
  55. 55.
    Woodham, R.J.: Photometric method for determining surface orientation from multiple images. Opt. Eng. 19(1), 139–144 (1980)CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2017

Authors and Affiliations

  1. 1.Technical University MunichGarchingGermany
  2. 2.IRITUniversité de ToulouseToulouseFrance
  3. 3.IMBUniversité de BordeauxTalenceFrance
  4. 4.Institut Universitaire de FranceParisFrance

Personalised recommendations