Alternating forward–backward splitting for linearly constrained optimization problems

Abstract

We present an alternating forward–backward splitting method for solving linearly constrained structured optimization problems. The algorithm takes advantage of the separable structure and possibly asymmetric regularity properties of the objective functions involved. We also describe some applications to the study of non-Newtonian fluids and image reconstruction problems. We conclude with a numerical example, and its comparison with Condat’s algorithm. An acceleration heuristic is also briefly outlined.

This is a preview of subscription content, log in to check access.

Fig. 1
Fig. 2

Notes

  1. 1.

    This quantity is finite, in view of Proposition 9, and the fact that \((\lambda _n)\in \ell ^2\) [an upper bound can be obtained from (22)].

References

  1. 1.

    Acker, F., Prestel, M.A.: Convergence d’un schéma de minimisation alternée. Ann. Faculté des Sci. Toulouse V Sér. Math. 2, 1–9 (1980)

    MATH  Google Scholar 

  2. 2.

    Attouch, H., Bolte, J., Redont, P., Soubeyran, A.: Alternating proximal algorithms for weakly coupled convex minimization problems. Applications to dynamical games and PDE’s. J. Convex Anal. 15(3), 485–506 (2008)

    MathSciNet  MATH  Google Scholar 

  3. 3.

    Attouch, H., Brézis, H.: Duality for the sum of convex functions in general Banach spaces. Asp. Math. Appl. J. Barroso Ed. 34, 125–133 (1986)

    MathSciNet  MATH  Google Scholar 

  4. 4.

    Attouch, H., Cabot, A., Frankel, P., Peypouquet, J.: Alternating proximal algorithms for constrained variational inequalities. Application to domain decomposition for PDE’s. Nonlinear Anal. 74(18), 7455–7473 (2011)

    MathSciNet  MATH  Google Scholar 

  5. 5.

    Attouch, H., Czarnecki, M.-O., Peypouquet, J.: Prox-penalization and splitting methods for constrained variational problems. SIAM J. Optim. 21, 149–173 (2011)

    MathSciNet  MATH  Google Scholar 

  6. 6.

    Attouch, H., Czarnecki, M.-O., Peypouquet, J.: Coupling forward–backward with penalty schemes and parallel splitting for constrained variational inequalities. SIAM J. Optim. 21(4), 1251–1274 (2011)

    MathSciNet  MATH  Google Scholar 

  7. 7.

    Attouch, H., Peypouquet, J.: The rate of convergence of Nesterov’s accelerated forward–backward method is actually faster than 1/k2. SIAM J. Optim. 26(3), 1824–1834 (2016)

    MathSciNet  MATH  Google Scholar 

  8. 8.

    Attouch, H., Redont, P., Soubeyran, A.: A new class of alternating proximal minimization algorithms with costs-to-move. SIAM J. Optim. 18, 1061–1081 (2007)

    MathSciNet  MATH  Google Scholar 

  9. 9.

    Auslender, A., Crouzeix, J.P., Fedit, P.: Penalty-proximal methods in convex programming. J. Optim. Theory Appl. 55, 1–21 (1987)

    MathSciNet  MATH  Google Scholar 

  10. 10.

    Bauschke, H.H., Combettes, P.L., Reich, S.: The asymptotic behavior of the composition of two resolvents. Nonlinear Anal. 60, 283–301 (2005)

    MathSciNet  MATH  Google Scholar 

  11. 11.

    Beck, A., Teboulle, M.: Fast gradient-based algorithms for constrained total variation image denoising and deblurring problems. IEEE Trans. Image Process. 18(11), 2419–2434 (2009)

    MathSciNet  MATH  Google Scholar 

  12. 12.

    Bildhauer, M., Fuchs, M.: A variational approach to the denoising of images based on different variants of the TV-regularization. Appl. Math. Optim. 66, 331–361 (2012)

    MathSciNet  MATH  Google Scholar 

  13. 13.

    Bertels, S.: Total variation minimization with finite elements: convergence and iterative solution. SIAM J. Numer. Anal. 50(3), 1162–1180 (2012)

    MathSciNet  Google Scholar 

  14. 14.

    Briceño-Arias, L.: Forward-Douglas–Rachford splitting and forward-partial inverse method for solving monotone inclusions. Optimization 64(5), 1239–1261 (2015)

    MathSciNet  MATH  Google Scholar 

  15. 15.

    Briceño-Arias, L., López, S.: A projected primal-dual splitting for solving constrained monotone inclusions. arXiv:1805.11687v1 [math.OC]

  16. 16.

    Cabot, A.: Proximal point algorithm controlled by a slowly vanishing term: applications to hierarchical minimization. SIAM J. Optim. 15(2), 555–572 (2005)

    MathSciNet  MATH  Google Scholar 

  17. 17.

    Candès, E.J., Eldar, Y.C., Needell, D., Randall, P.: Compressed sensing with coherent and redundant dictionaries. Appl. Comput. Harmon. Anal. 31(1), 59–73 (2011)

    MathSciNet  MATH  Google Scholar 

  18. 18.

    Carter, J.L.: Dual methods for total variation-based image restoration. Ph.D. Thesis, UCLA (2001)

  19. 19.

    Cauchy, A.L.: Méthode générale pour la résolution des systèmes d’équations simultanées. C. R. Acad. Sci. Paris 25, 536–538 (1847)

    Google Scholar 

  20. 20.

    Chambolle, A., Dossal, C.: On the convergence of the iterates of the “fast iterative shrinkage/thresholding algorithm”. J. Optim. Theory Appl. 166(3), 968–982 (2015)

    MathSciNet  MATH  Google Scholar 

  21. 21.

    Chambolle, A., Lions, P.L.: Image recovery via total variation minimization and related problems. Numer. Math. 76, 167–188 (1997)

    MathSciNet  MATH  Google Scholar 

  22. 22.

    Chambolle, A., Pock, T.: A first-order primal-dual algorithm for convex problems with applications to imaging. J. Math. Imaging Vis. 40(1), 120–145 (2011)

    MathSciNet  MATH  Google Scholar 

  23. 23.

    Chan, T., Esedoglu, S., Park, F., Yip, A.: Total variation image restoration: overview and recent developments. In: Paragios, N., Chen, Y., Faugeras, O.D. (eds.) Handbook of mathematical models in computer vision, pp. 17–31. Springer, New York (2006)

    Google Scholar 

  24. 24.

    Chan, T.F., Golub, G.H., Mulet, P.: A nonlinear primal-dual method for total variation-based image restoration. SIAM J. Sci. Comput. 20(6), 1964–1977 (1999)

    MathSciNet  MATH  Google Scholar 

  25. 25.

    Chen, G., Teboulle, M.: A proximal-based decomposition method for convex minimization problems. Math. Program. 64(1–3), 81–101 (1994)

    MathSciNet  MATH  Google Scholar 

  26. 26.

    Combettes, P.L.: Iterative construction of the resolvent of a sum of maximal monotone operators. J. Convex Anal. 16(3–4), 727–748 (2009)

    MathSciNet  MATH  Google Scholar 

  27. 27.

    Combettes, P.L., Pesquet, J.C.: A proximal decomposition method for solving convex variational inverse problems. Inverse Probl. 24(26) (2008)

  28. 28.

    Combettes, P.L., Pesquet, J.C.: Proximal splitting methods in signal processing. Fixed-point algorithms for inverse problems in science and engineering. In: Bauschke, H.H., Burachik, R.S.S., Combettes, P.L., Elser, V., Luke, D.R., Wolkowicz, H. (eds.) Springer Optimization and Its Applications, vol. 49, pp. 185–212. Springer, New York (2011)

  29. 29.

    Combettes, P.L., Pesquet, J.C.: Primal-dual splitting algorithm for solving inclusions with mixtures of composite, Lipschitzian, and parallel-sum type monotone operators. Set Valued Var. Anal. 20(2), 307–330 (2012)

    MathSciNet  MATH  Google Scholar 

  30. 30.

    Combettes, P.L., Wajs, V.R.: Signal recovery by proximal forward–backward splitting. Multiscale Model. Simul. 4(4), 1168–1200 (2005)

    MathSciNet  MATH  Google Scholar 

  31. 31.

    Condat, L.: A primal-dual splitting method for convex optimization involving Lipschitzian, proximable and linear composite terms. J. Optim. Theory Appl. 158(2), 460–479 (2013)

    MathSciNet  MATH  Google Scholar 

  32. 32.

    Daubechies, I., Defrise, M., De Mol, C.: An iterative thresholding algorithm for linear inverse problems with a sparsity constraint. Commun. Pure Appl. Math. 57(11), 1413–1457 (2004)

    MathSciNet  MATH  Google Scholar 

  33. 33.

    De Oliveira, W., Sagastizábal, C., Lemaréchal, C.: Convex proximal bundle methods in depth: a unified analysis for inexact oracles. Math. Program. Ser. B 148(1–2), 241–277 (2014)

    MathSciNet  MATH  Google Scholar 

  34. 34.

    Dobson, D.C., Vogel, C.R.: Convergence of an iterative method for total variation denoising. SIAM J. Numer. Anal. 34(5), 1779–1791 (1997)

    MathSciNet  MATH  Google Scholar 

  35. 35.

    Douglas, J., Rachford, H.H.: On the numerical solution of heat conduction problems in two and three space variables. Trans. Am. Math. Soc. 82(2), 421–439 (1956)

    MathSciNet  MATH  Google Scholar 

  36. 36.

    Eckstein, J., Silva, P.J.S.: Proximal methods for nonlinear programming: double regularization and inexact subproblems. Comput. Optim. Appl. 46(2), 279–304 (2010)

    MathSciNet  MATH  Google Scholar 

  37. 37.

    Garrigos, G., Rosasco, L., Villa, S.: Iterative regularization via dual diagonal descent. J. Math. Imaging Vis. 60, 189–215 (2017). https://doi.org/10.1007/s10851-017-0754-0

    MathSciNet  Article  MATH  Google Scholar 

  38. 38.

    Goldstein, A.A.: Convex programming in Hilbert space. Bull. Am. Math. Soc. 70(5), 709–710 (1964)

    MathSciNet  MATH  Google Scholar 

  39. 39.

    Goldstein, T., Osher, S.: The split Bregman method for L1-regularized problems. SIAM J. Imaging Sci. 2(2), 323–343 (2009)

    MathSciNet  MATH  Google Scholar 

  40. 40.

    González-Andrade, S.: A preconditioned descent algorithm for variational inequalities of the second kind involving the \(p\)-Laplacian operator. Comput. Optim. Appl. 66(1), 123–162 (2017)

    MathSciNet  MATH  Google Scholar 

  41. 41.

    Hare, W., Planiden, C.: Computing proximal points of convex functions with inexact subgradients. Set Valued Var. Anal. 26(3), 469–492 (2018)

    MathSciNet  MATH  Google Scholar 

  42. 42.

    Hare, W., Sagastizábal, C., Solodov, M.: A proximal bundle method for nonsmooth nonconvex functions with inexact information. Comput. Optim. Appl. 63(1), 1–28 (2016)

    MathSciNet  MATH  Google Scholar 

  43. 43.

    Lai, M.J., Messi, L.M.: Piecewise linear approximation of the continuous Rudin–Osher–Fatemi model for image denoising. SIAM J. Numer. Anal. 50(5), 2446–2466 (2012)

    MathSciNet  MATH  Google Scholar 

  44. 44.

    Levitin, E.S., Polyak, B.T.: Constrained minimization problems. USSR Comput. Math. Math. Phys. 6, 1–50 (1966) (English translation of original paper in Zh. Vychisl. Mat. Fiz. 6, 787–823, 1965)

  45. 45.

    Martinet, B.: Régularisation d’inéquations variationnelles par approximations successives. Rev. Française Informat. Recherche Opérationnelle 4, 154–158 (1970)

    MathSciNet  MATH  Google Scholar 

  46. 46.

    Molinari, C., Peypouquet, J.: Lagrangian penalization algorithm with parallel forward–backward splitting. J. Optim. Theory Appl. 177(2), 413–447 (2018)

    MathSciNet  MATH  Google Scholar 

  47. 47.

    Moreau, J.J.: Fonctionnelles Convexes. Cours Collège de France, Paris (1967)

    Google Scholar 

  48. 48.

    Nesterov, Y.: A method of solving a convex programming problem with convergence rate \(O(1/k^2)\). Sov. Math. Dokl. 27, 372–376 (1983)

    MATH  Google Scholar 

  49. 49.

    Opial, Z.: Weak convergence of the sequence of successive approximations for nonexpansive mappings. Bull. Am. Math. Soc. 73, 591–597 (1967)

    MathSciNet  MATH  Google Scholar 

  50. 50.

    Parikh, N., Boyd, S.: Proximal algorithms. Found. Trends Optim. 1(3), 123–231 (2014)

    Google Scholar 

  51. 51.

    Peypouquet, J.: Convex Optimization in Normed Spaces. Theory, Methods and Examples. With a Foreword by Hedy Attouch. Springer, Berlin (2015)

  52. 52.

    Reem, D., Reich, S.: Solutions to inexact resolvent inclusion problems with applications to nonlinear analysis and optimization. Rend. Circ. Mat. Palermo (2) 67(2), 337–371 (2018)

    MathSciNet  MATH  Google Scholar 

  53. 53.

    Rockafellar, R.T.: Convex Analysis. Princeton University Press, Princeton (1970)

    Google Scholar 

  54. 54.

    Rockafellar, R.T.: Monotone operators and the proximal point algorithm. SIAM J. Control Optim. 14, 877–898 (1976)

    MathSciNet  MATH  Google Scholar 

  55. 55.

    Rudin, L.I., Osher, S., Fatemi, E.: Nonlinear total variation based noise removal algorithms. Physica D 60, 259–268 (1992)

    MathSciNet  MATH  Google Scholar 

  56. 56.

    Shen, J., Xia, Z.Q., Pang, L.P.: A proximal bundle method with inexact data for convex nondifferentiable minimization. Nonlinear Anal. 66(9), 2016–2027 (2007)

    MathSciNet  MATH  Google Scholar 

  57. 57.

    Su, W., Boyd, S., Candès, E.J.: A differential equation for modeling Nesterov’s accelerated gradient method: theory and insights. NIPS 27, 2510–2518 (2014)

    MATH  Google Scholar 

  58. 58.

    Tai, X.C., Wu, C.: Augmented Lagrangian method, dual methods and split Bregman iteration for ROF model. In: Tai, X.C., Moerken, K., Lysaker, M., Lie, K.A. (eds.) Scale space and variational methods in computer vision, pp. 502–513 (2009)

  59. 59.

    Vogel, C.R., Oman, M.E.: Iterative methods for total variation denoising. SIAM J. Sci. Comput. 17(1), 227–238 (1996)

    MathSciNet  MATH  Google Scholar 

  60. 60.

    Vu, B.C.: A splitting algorithm for coupled system of primal-dual monotone inclusions. J. Optim. Theory Appl. 164(3), 993–1025 (2015)

    MathSciNet  MATH  Google Scholar 

  61. 61.

    Wu, C., Zhang, J., Tai, X.C.: Augmented Lagrangian method for total variation restoration with non-quadratic fidelity. Inverse Probl. Imaging 5(1), 237–261 (2011)

    MathSciNet  MATH  Google Scholar 

  62. 62.

    Zhu, M., Wright, S.J., Chan, T.F.: Duality-based algorithms for total-variation-regularized image restoration. Comput. Optim. Appl. 47, 377–400 (2010)

    MathSciNet  MATH  Google Scholar 

Download references

Acknowledgements

Supported by Fondecyt Grant 1181179 and Basal Project CMM Universidad de Chile. The first author was also supported by CONICYT-PCHA/Doctorado Nacional/2016 scholarship.

Author information

Affiliations

Authors

Corresponding author

Correspondence to Juan Peypouquet.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Molinari, C., Peypouquet, J. & Roldan, F. Alternating forward–backward splitting for linearly constrained optimization problems. Optim Lett 14, 1071–1088 (2020). https://doi.org/10.1007/s11590-019-01388-y

Download citation

Keywords

  • Convex optimization
  • Forward–backward splitting
  • Structured problems