Skip to main content
SpringerLink
Log in

Iterative Regularization via Dual Diagonal Descent

  • Published:
Journal of Mathematical Imaging and Vision Aims and scope Submit manuscript

Abstract

In the context of linear inverse problems, we propose and study a general iterative regularization method allowing to consider large classes of data-fit terms and regularizers. The algorithm we propose is based on a primal-dual diagonal descent method. Our analysis establishes convergence as well as stability results. Theoretical findings are complemented with numerical experiments showing state-of-the-art performances.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13

Similar content being viewed by others

Notes

  1. In [2], the authors show that the proposed proximal method can be used to solve the dual problem, but the regularization method they consider is the exponential barrier, which is not of interest here.

  2. Here, the term qualification condition shall be understood as in the optimization literature. It is a sufficient condition ensuring, in our case, strong duality between the problems (P) and (D). It should not be confused with the notion of qualification used in the inverse problem literature, which is a property for a regularization method [46, Remark 4.6].

  3. To see this, it is enough to write the optimality condition of (P) and use the Moreau-Rockafellar Theorem [62, Theorem 3.30].

  4. http://www.guillaume-garrigos.com/database/image_processing_512.zip

References

  1. Alart, P., Lemaire, B.: Penalization in non-classical convex programming via variational convergence. Math. Program. 51, 307–331 (1991)

    Article  MATH  Google Scholar 

  2. Alvarez, F., Cominetti, R.: Primal and dual convergence of a proximal point exponential penalty method for linear programming. Math. Program. 93, 87–96 (2002)

    Article  MathSciNet  MATH  Google Scholar 

  3. Attouch, H.: Viscosity solutions of minimization problems. SIAM J. Optim. 6, 769–806 (1996)

    Article  MathSciNet  MATH  Google Scholar 

  4. Attouch, H., Cabot, A., Czarnecki, M.-O.: Asymptotic behavior of non-autonomous monotone and subgradient evolution equations. arXiv:1601.00767 (2016)

  5. Attouch, H., Czarnecki, M.-O.: Asymptotic behavior of coupled dynamical systems with multiscale aspects. J. Differ. Equ. 248, 1315–1344 (2010)

    Article  MathSciNet  MATH  Google Scholar 

  6. Attouch, H., Czarnecki, M.-O., Peypouquet, J.: Prox-penalization and splitting methods for constrained variational problems. SIAM J. Optim. 21, 149–173 (2011)

    Article  MathSciNet  MATH  Google Scholar 

  7. Attouch, H., Czarnecki, M.-O., Peypouquet, J.: Coupling forward–backward with penalty schemes and parallel splitting for constrained variational inequalities. SIAM J. Optim. 21, 1251–1274 (2011)

    Article  MathSciNet  MATH  Google Scholar 

  8. Auslender, A., Crouzeix, J.-P., Fedit, P.: Penalty-proximal methods in convex programming. J. Optim. Theory Appl. 55, 1–21 (1987)

    Article  MathSciNet  MATH  Google Scholar 

  9. Bachmayr, M., Burger, M.: Iterative total variation schemes for nonlinear inverse problems. Inverse Probl. 25, 105004 (2009)

    Article  MathSciNet  MATH  Google Scholar 

  10. Bahraoui, M.A., Lemaire, B.: Convergence of diagonally stationary sequences in convex optimization. Set-Valued Anal. 2, 49–61 (1994)

    Article  MathSciNet  MATH  Google Scholar 

  11. Bakushinsky, A.B., Kokurin, MYu.: Iterative Methods for Approximate Solution of Inverse Problems. Springer, New York (2004)

    MATH  Google Scholar 

  12. Bauschke, H.H., Borwein, J.: Joint and separate convexity of the Bregman distance, inherently parallel algorithms in feasibility and optimization and their applications. Stud. Comput. Math. 8, 23–36 (2001)

    Article  MATH  Google Scholar 

  13. Bauschke, H.H., Combettes, P.: Convex Analysis and Monotone Operator Theory. Springer, New York (2011)

    MATH  Google Scholar 

  14. Beck, A., Sabach, S.: A first order method for finding minimal norm-like solutions of convex optimization problems. Math. Program. 147, 25–46 (2014)

    Article  MathSciNet  MATH  Google Scholar 

  15. Beck, A., Teboulle, M.: Mirror descent and nonlinear projected subgradient methods for convex optimization. Oper. Res. Lett. 31, 167–175 (2003)

    Article  MathSciNet  MATH  Google Scholar 

  16. Becker, S., Bobin, J., Candès, E.: NESTA: a fast and accurate first-order method for sparse recovery. SIAM J. Imaging Sci. 4, 1–39 (2011)

    Article  MathSciNet  MATH  Google Scholar 

  17. Bertero, M., Boccacci, P.: Introduction to Inverse Problems in Imaging. IOP Publishing, Bristol (1998)

    Book  MATH  Google Scholar 

  18. Bolte, J., Nguyen, T.P., Peypouquet, J., Suter, B.: From error bounds to the complexity of first-order descent methods for convex functions. arXiv:1510.08234 (2015)

  19. Bot, R.I., Hofmann, B.: The impact of a curious type of smoothness conditions on convergence rates in l1-regularization. Eurasian J. Math. Comput. Appl. 1, 29–40 (2013)

    Google Scholar 

  20. Bot, R.I., Hein, T.: Iterative regularization with a general penalty term: theory and application to L1 and TV regularization. Inverse Prob. 28, 1–19 (2012)

    Article  MATH  Google Scholar 

  21. Boyer, R.: Quelques algorithmes diagonaux en optimisation convexe. Ph.D., Université de Provence (1974)

  22. Bredies, K., Kunisch, K., Pock, T.: Total generalized variation. SIAM J. Imaging Sci. 3, 492–526 (2010)

    Article  MathSciNet  MATH  Google Scholar 

  23. Briceño-Arias, L., Combettes, P.L.: A monotone + skew splitting model for composite monotone inclusions in duality. SIAM J. Optim. 21, 1230–1250 (2011)

    Article  MathSciNet  MATH  Google Scholar 

  24. Bruck Jr., R.E.: A strongly convergent iterative solution of \(0 \in U(x)\) for a maximal monotone operator \(U\) in Hilbert space. J. Math. Anal. Appl. 48, 114–126 (1974)

    Article  MathSciNet  MATH  Google Scholar 

  25. Burger, M., Osher, S. (ed.): A guide to the TV zoo. In: Level Set and PDE Based Reconstruction Methods in Imaging, pp. 1–70. Springer, Berlin (2013)

  26. Burger, M., Resmerita, E., He, L.: Error estimation for Bregman iterations and inverse scale space methods in image restoration. Comput. 81, 109–135 (2007)

    Article  MathSciNet  MATH  Google Scholar 

  27. Cabot, A.: The steepest descent dynamical system with control. Applications to constrained minimization. ESAIM Control Optim. Calc. Var. 10, 243–258 (2004)

    Article  MathSciNet  MATH  Google Scholar 

  28. Cabot, A.: Proximal point algorithm controlled by a slowly vanishing term: applications to hierarchical minimization. SIAM J. Optim. 15, 555–572 (2005)

    Article  MathSciNet  MATH  Google Scholar 

  29. Calatroni, L., De Los Reyes, J.-C., Schönlieb, C.-B.: Infimal convolution of data discrepancies for mixed noise removal. arXiv:1611.00690 (2016)

  30. Chaux, C., Combettes, P.L., Pesquet, J.-C., Wajs, V.: A variational formulation for frame-based inverse problems. Inverse Probl. 23, 1495–1518 (2007)

    Article  MathSciNet  MATH  Google Scholar 

  31. Chambolle, A., Lions, P.L.: Image recovery via total variation minimization and related problems. Numer. Math. 76, 167–188 (1997)

    Article  MathSciNet  MATH  Google Scholar 

  32. Chambolle, A., Pock, T.: A first-order primal-dual algorithm for convex problems with applications to imaging. J. Math. Imaging Vis. 40, 120–145 (2011)

    Article  MathSciNet  MATH  Google Scholar 

  33. Chambolle, A., Pock, T.: A remark on accelerated block coordinate descent for computing the proximity operators of a sum of convex functions, preprint hal-01099182v2 (2015)

  34. Combettes, P.L.: Quasi-Fejérian analysis of some optimization algorithms. In: Butnariu, D., Censor, Y., Reich, S. (eds.) Inherently Parallel Algorithms in Feasibility and Optimization and Their Applications, pp. 115–152. Elsevier, New York (2001)

    Chapter  Google Scholar 

  35. Combettes, P.L., Dũng, D., Vũ, B.C.: Dualization of signal recovery problems. Set-Valued Var. Anal. 18, 373–404 (2010)

    Article  MathSciNet  MATH  Google Scholar 

  36. Combettes, P.L., Pesquet, J.-C.: Proximal splitting methods in signal processing. In: Bauschke, H. H., Burachik, R. S., Combettes, P. L., Elser, V., Luke, D. R., Wolkowicz, H. (eds.) Fixed-Point Algorithms for Inverse Problems in Science and Engineering, pp. 185–212. Springer, New York (2011)

  37. Combettes, P.L., Pesquet, J.-C.: Primal-dual splitting algorithm for solving inclusions with mixtures of composite, Lipschitzian, and parallel-sum type monotone operators. Set-Valued Var. Anal. 20, 307–330 (2012)

    Article  MathSciNet  MATH  Google Scholar 

  38. Combettes, P.L., Wajs, V.: Signal recovery by proximal forward–backward splitting. Multiscale Model. Simul. 4, 1168–1200 (2005)

    Article  MathSciNet  MATH  Google Scholar 

  39. Cominetti, R., Peypouquet, J., Sorin, S.: Strong asymptotic convergence of evolution equations governed by maximal monotone operators with Tikhonov regularization. J. Differ. Equ. 245, 3753–3763 (2008)

    Article  MathSciNet  MATH  Google Scholar 

  40. Cominetti, R.: Coupling the proximal point algorithm with approximation methods. J. Optim. Theory Appl. 95, 581–600 (1997)

    Article  MathSciNet  MATH  Google Scholar 

  41. Czarnecki, M.-O., Noun, N., Peypouquet, J.: Splitting forward–backward penalty scheme for constrained variational problems. arXiv:1408.0974 (2014)

  42. Daubechies, I., Defrise, M., De Mol, C.: An iterative thresholding algorithm for linear inverse problems with a sparsity constraint. Commun. Pure Appl. Math. 57, 1413–1457 (2004)

    Article  MathSciNet  MATH  Google Scholar 

  43. Deledalle, C.-A., Vaiter, S., Fadili, J.-M., Peyré, G.: Stein Unbiased GrAdient estimator of the Risk (SUGAR) for multiple parameter selection. SIAM J. Imaging Sci. 7, 2448–2487 (2014)

    Article  MathSciNet  MATH  Google Scholar 

  44. Dontchev, A., Zolezzi, T.: Well-Posed Optimization Problems. Springer, Berlin (1993)

    Book  MATH  Google Scholar 

  45. Dupé, F.-X., Fadili, J., Starck, J.-L.: Deconvolution under Poisson noise using exact data-fit function and synthesis or analysis sparsity priors. Stat. Methodol. 9, 4–18 (2012)

    Article  MathSciNet  MATH  Google Scholar 

  46. Engl, H., Hanke, M., Neubauer, A.: Regularization of Inverse Problems. Kluwer, Dordrecht (1996)

    Book  MATH  Google Scholar 

  47. Hale, E., Yin, W., Zhang, Y.: Fixed-point continuation for \(\ell _1\)-minimization: methodology and convergence. SIAM J. Optim. 19, 1107–1130 (2008)

    Article  MathSciNet  MATH  Google Scholar 

  48. Hintermüller, M., Langer, A.: Subspace correction methods for a class of non-smooth and non-additive convex variational problems with mixed \(\ell ^1/\ell ^2\) data-fidelity in image processing. SIAM J. Imaging Sci. 6, 2134–2173 (2013)

    Article  MathSciNet  MATH  Google Scholar 

  49. Kaltenbacher, B., Neubauer, A., Scherzer, O.: Iterative Regularization Methods for Nonlinear Ill-Posed Problems. De Gruyter, Berlin (2008)

    Book  MATH  Google Scholar 

  50. Kaplan, A.A.: On convex programming with internal regularization. Sov. Math. Dokl. Akad. Nauk 19, 795–799 (1975)

    MATH  Google Scholar 

  51. Kaplan, A.A.: Iteration processes of convex programming with internal regularization. Sib. Math. J. 20, 219–226 (1979)

    Article  MATH  Google Scholar 

  52. Langer, A.: Automated parameter selection for total variation minimization in image restoration. J. Math. Imaging Vis. 57, 239–268 (2017)

    Article  MathSciNet  MATH  Google Scholar 

  53. Le, T., Chartran, R., Asaki, T.: A variational approach to reconstructing images corrupted by Poisson noise. J. Math. Imaging Vis. 27, 257–63 (2007)

    Article  MathSciNet  Google Scholar 

  54. Lemaire, B.: Coupling optimization methods and variational convergence. Trends Math. Optim. Int. Ser. Numer. Math. 84, 163–179 (1988)

    MathSciNet  MATH  Google Scholar 

  55. Lemaire, B.: On the convergence of some iterative methods for convex minimization. Recent Dev. Optim. Lect. Notes Econ. Math. Syst. 429, 252–268 (1995)

    Article  MathSciNet  MATH  Google Scholar 

  56. Lemaire, B.: Well-posedness, conditioning and regularization of minimization, inclusion and fixed-point problems. Pliska Studia Mathematica Bulgarica 12, 71–84 (1998)

    MathSciNet  MATH  Google Scholar 

  57. Mallat, S.: A Wavelet Tour of Signal Processing, 3rd edn. Elsevier/Academic Press, Amsterdam (2009)

    MATH  Google Scholar 

  58. Martinet, B.: Perturbation des mthodes d’optimisation. Applications. R.A.I.R.O Analyse numrique 12, 153–171 (1978)

    MATH  Google Scholar 

  59. Matet, S., Rosasco, L., Villa, S., Vũ, B. C.: Don’t relax: early stopping for convex regularization. arXiv:1707.05422 (2016)

  60. Nikolova, M.: Minimizers of cost-functions involving non- smooth data-fidelity terms. Application to the processing of outliers. SIAM J. Numer. Anal. 40, 965–994 (2002)

    Article  MathSciNet  MATH  Google Scholar 

  61. Peypouquet, J.: Coupling the gradient method with a general exterior penalization scheme for convex minimization. J. Optim. Theory Appl. 153, 123–138 (2011)

    Article  MathSciNet  MATH  Google Scholar 

  62. Peypouquet, J.: Convex Optimization in Normed Spaces. Theory, Methods and Examples. Springer, New York (2015)

    MATH  Google Scholar 

  63. Ramlau, R.: TIGRA—an iterative algorithm for regularizing nonlinear ill-posed problems. Inverse Prob. 19, 433–465 (2003)

    Article  MathSciNet  MATH  Google Scholar 

  64. Rudin, L., Osher, S., Fatemi, E.: Nonlinear total variation based noise removal algorithms. Physica D 60, 259–268 (1992)

    Article  MathSciNet  MATH  Google Scholar 

  65. Scherzer, O.: A modified Landweber iteration for solving parameter estimation problems. Appl. Math. Optim. 38, 45–68 (1998)

    Article  MathSciNet  MATH  Google Scholar 

  66. Stein, C.: Estimation of the mean of a multivariate normal distribution. Ann. Stat. 9, 1135–1151 (1981)

    Article  MathSciNet  MATH  Google Scholar 

  67. Steinwart, I., Christmann, A.: Support Vector Machines. Springer, New York (2008)

    MATH  Google Scholar 

  68. Tossings, P.: The perturbed Tikhonov’s algorithm and some of its applications. ESAIM Math. Model. Numer. Anal. 28, 189–221 (1994)

    Article  MathSciNet  MATH  Google Scholar 

  69. Tseng, P.: Applications of a splitting algorithm to decomposition in convex programming and variational inequalities. SIAM J. Control Optim. 29, 119–138 (1991)

    Article  MathSciNet  MATH  Google Scholar 

  70. Uzawa, H.: Iterative Methods for Concave Programming. Studies in Linear and Nonlinear Programming. Stanford University Press, Stanford (1958)

    Google Scholar 

  71. Vainberg, M.M.: Le problème de la minimisation des fonctionelles non linéaires. C.I.M.E. IV ciclo (1970)

  72. Yamada, I., Yukawa, M., Yamagishi, M.: Minimizing the Moreau envelope of nonsmooth convex functions over the fixed point set of certain quasi-nonexpansive mappings. In: Bauschke, H. H., Burachik, R. S., Combettes, P. L., Elser, V., Luke, D. R., Wolkowicz, H. (eds.) Fixed-Point Algorithms for Inverse Problems in Science and Engineering. Springer, New York (2011)

  73. Zolezzi, T.: On equiwellset minimum problems. Appl. Math. Optim. 4, 209–223 (1978)

    Article  MathSciNet  MATH  Google Scholar 

  74. Zou, Z., Hastie, T.: Regularization and variable selection via the elastic net. J. R. Stat. Soc. B 67, 301–320 (2005)

    Article  MathSciNet  MATH  Google Scholar 

  75. Zalinescu, C.: Convex Analysis in General Vector Spaces. World Scientific, Singapore (2002)

    Book  MATH  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Laboratory for Computational and Statistical Learning, Istituto Italiano di Tecnologia and Massachusetts Institute of Technology, 43 Vassar Street, Cambridge, MA, 02139, USA

    Guillaume Garrigos & Lorenzo Rosasco

  2. DIBRIS, Università Degli Studi di Genova, Via Dodecaneso 35, 16146, Genova, Italy

    Lorenzo Rosasco

  3. Dipartimento di Matematica, Politecnico di Milano, Via Bonardi 9, 20133, Milan, Italy

    Silvia Villa

Authors
  1. Guillaume Garrigos
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Lorenzo Rosasco
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Silvia Villa
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Guillaume Garrigos.

Additional information

The authors would like to thank NVIDIA Corporation for the donation of a Tesla K40c GPU used for this research. This material is based upon work supported by the Center for Brains, Minds and Machines (CBMM), funded by NSF STC award CCF-1231216, and the Air Force project FA9550-17-1-0390 (European Office of Aerospace Research and Development). Second author acknowledges the financial support of the Italian Ministry of Education, University and Research FIRB project RBFR12M3AC. Third author acknowledges the financial support of the INDAMGNAMPA research project 2017 Algoritmi di ottimizzazione ed equazioni di evoluzione ereditarie.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Garrigos, G., Rosasco, L. & Villa, S. Iterative Regularization via Dual Diagonal Descent. J Math Imaging Vis 60, 189–215 (2018). https://doi.org/10.1007/s10851-017-0754-0

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10851-017-0754-0

Keywords

Search

Navigation