Alternating forward–backward splitting for linearly constrained optimization problems

  • Cesare Molinari
  • Juan PeypouquetEmail author
  • Fernando Roldan


We present an alternating forward–backward splitting method for solving linearly constrained structured optimization problems. The algorithm takes advantage of the separable structure and possibly asymmetric regularity properties of the objective functions involved. We also describe some applications to the study of non-Newtonian fluids and image reconstruction problems. We conclude with a numerical example, and its comparison with Condat’s algorithm. An acceleration heuristic is also briefly outlined.


Convex optimization Forward–backward splitting Structured problems 



Supported by Fondecyt Grant 1181179 and Basal Project CMM Universidad de Chile. The first author was also supported by CONICYT-PCHA/Doctorado Nacional/2016 scholarship.


  1. 1.
    Acker, F., Prestel, M.A.: Convergence d’un schéma de minimisation alternée. Ann. Faculté des Sci. Toulouse V Sér. Math. 2, 1–9 (1980)zbMATHGoogle Scholar
  2. 2.
    Attouch, H., Bolte, J., Redont, P., Soubeyran, A.: Alternating proximal algorithms for weakly coupled convex minimization problems. Applications to dynamical games and PDE’s. J. Convex Anal. 15(3), 485–506 (2008)MathSciNetzbMATHGoogle Scholar
  3. 3.
    Attouch, H., Brézis, H.: Duality for the sum of convex functions in general Banach spaces. Asp. Math. Appl. J. Barroso Ed. 34, 125–133 (1986)MathSciNetzbMATHGoogle Scholar
  4. 4.
    Attouch, H., Cabot, A., Frankel, P., Peypouquet, J.: Alternating proximal algorithms for constrained variational inequalities. Application to domain decomposition for PDE’s. Nonlinear Anal. 74(18), 7455–7473 (2011)MathSciNetzbMATHGoogle Scholar
  5. 5.
    Attouch, H., Czarnecki, M.-O., Peypouquet, J.: Prox-penalization and splitting methods for constrained variational problems. SIAM J. Optim. 21, 149–173 (2011)MathSciNetzbMATHGoogle Scholar
  6. 6.
    Attouch, H., Czarnecki, M.-O., Peypouquet, J.: Coupling forward–backward with penalty schemes and parallel splitting for constrained variational inequalities. SIAM J. Optim. 21(4), 1251–1274 (2011)MathSciNetzbMATHGoogle Scholar
  7. 7.
    Attouch, H., Peypouquet, J.: The rate of convergence of Nesterov’s accelerated forward–backward method is actually faster than 1/k2. SIAM J. Optim. 26(3), 1824–1834 (2016)MathSciNetzbMATHGoogle Scholar
  8. 8.
    Attouch, H., Redont, P., Soubeyran, A.: A new class of alternating proximal minimization algorithms with costs-to-move. SIAM J. Optim. 18, 1061–1081 (2007)MathSciNetzbMATHGoogle Scholar
  9. 9.
    Auslender, A., Crouzeix, J.P., Fedit, P.: Penalty-proximal methods in convex programming. J. Optim. Theory Appl. 55, 1–21 (1987)MathSciNetzbMATHGoogle Scholar
  10. 10.
    Bauschke, H.H., Combettes, P.L., Reich, S.: The asymptotic behavior of the composition of two resolvents. Nonlinear Anal. 60, 283–301 (2005)MathSciNetzbMATHGoogle Scholar
  11. 11.
    Beck, A., Teboulle, M.: Fast gradient-based algorithms for constrained total variation image denoising and deblurring problems. IEEE Trans. Image Process. 18(11), 2419–2434 (2009)MathSciNetzbMATHGoogle Scholar
  12. 12.
    Bildhauer, M., Fuchs, M.: A variational approach to the denoising of images based on different variants of the TV-regularization. Appl. Math. Optim. 66, 331–361 (2012)MathSciNetzbMATHGoogle Scholar
  13. 13.
    Bertels, S.: Total variation minimization with finite elements: convergence and iterative solution. SIAM J. Numer. Anal. 50(3), 1162–1180 (2012)MathSciNetGoogle Scholar
  14. 14.
    Briceño-Arias, L.: Forward-Douglas–Rachford splitting and forward-partial inverse method for solving monotone inclusions. Optimization 64(5), 1239–1261 (2015)MathSciNetzbMATHGoogle Scholar
  15. 15.
    Briceño-Arias, L., López, S.: A projected primal-dual splitting for solving constrained monotone inclusions. arXiv:1805.11687v1 [math.OC]
  16. 16.
    Cabot, A.: Proximal point algorithm controlled by a slowly vanishing term: applications to hierarchical minimization. SIAM J. Optim. 15(2), 555–572 (2005)MathSciNetzbMATHGoogle Scholar
  17. 17.
    Candès, E.J., Eldar, Y.C., Needell, D., Randall, P.: Compressed sensing with coherent and redundant dictionaries. Appl. Comput. Harmon. Anal. 31(1), 59–73 (2011)MathSciNetzbMATHGoogle Scholar
  18. 18.
    Carter, J.L.: Dual methods for total variation-based image restoration. Ph.D. Thesis, UCLA (2001)Google Scholar
  19. 19.
    Cauchy, A.L.: Méthode générale pour la résolution des systèmes d’équations simultanées. C. R. Acad. Sci. Paris 25, 536–538 (1847)Google Scholar
  20. 20.
    Chambolle, A., Dossal, C.: On the convergence of the iterates of the “fast iterative shrinkage/thresholding algorithm”. J. Optim. Theory Appl. 166(3), 968–982 (2015)MathSciNetzbMATHGoogle Scholar
  21. 21.
    Chambolle, A., Lions, P.L.: Image recovery via total variation minimization and related problems. Numer. Math. 76, 167–188 (1997)MathSciNetzbMATHGoogle Scholar
  22. 22.
    Chambolle, A., Pock, T.: A first-order primal-dual algorithm for convex problems with applications to imaging. J. Math. Imaging Vis. 40(1), 120–145 (2011)MathSciNetzbMATHGoogle Scholar
  23. 23.
    Chan, T., Esedoglu, S., Park, F., Yip, A.: Total variation image restoration: overview and recent developments. In: Paragios, N., Chen, Y., Faugeras, O.D. (eds.) Handbook of mathematical models in computer vision, pp. 17–31. Springer, New York (2006)Google Scholar
  24. 24.
    Chan, T.F., Golub, G.H., Mulet, P.: A nonlinear primal-dual method for total variation-based image restoration. SIAM J. Sci. Comput. 20(6), 1964–1977 (1999)MathSciNetzbMATHGoogle Scholar
  25. 25.
    Chen, G., Teboulle, M.: A proximal-based decomposition method for convex minimization problems. Math. Program. 64(1–3), 81–101 (1994)MathSciNetzbMATHGoogle Scholar
  26. 26.
    Combettes, P.L.: Iterative construction of the resolvent of a sum of maximal monotone operators. J. Convex Anal. 16(3–4), 727–748 (2009)MathSciNetzbMATHGoogle Scholar
  27. 27.
    Combettes, P.L., Pesquet, J.C.: A proximal decomposition method for solving convex variational inverse problems. Inverse Probl. 24(26) (2008)Google Scholar
  28. 28.
    Combettes, P.L., Pesquet, J.C.: Proximal splitting methods in signal processing. Fixed-point algorithms for inverse problems in science and engineering. In: Bauschke, H.H., Burachik, R.S.S., Combettes, P.L., Elser, V., Luke, D.R., Wolkowicz, H. (eds.) Springer Optimization and Its Applications, vol. 49, pp. 185–212. Springer, New York (2011)Google Scholar
  29. 29.
    Combettes, P.L., Pesquet, J.C.: Primal-dual splitting algorithm for solving inclusions with mixtures of composite, Lipschitzian, and parallel-sum type monotone operators. Set Valued Var. Anal. 20(2), 307–330 (2012)MathSciNetzbMATHGoogle Scholar
  30. 30.
    Combettes, P.L., Wajs, V.R.: Signal recovery by proximal forward–backward splitting. Multiscale Model. Simul. 4(4), 1168–1200 (2005)MathSciNetzbMATHGoogle Scholar
  31. 31.
    Condat, L.: A primal-dual splitting method for convex optimization involving Lipschitzian, proximable and linear composite terms. J. Optim. Theory Appl. 158(2), 460–479 (2013)MathSciNetzbMATHGoogle Scholar
  32. 32.
    Daubechies, I., Defrise, M., De Mol, C.: An iterative thresholding algorithm for linear inverse problems with a sparsity constraint. Commun. Pure Appl. Math. 57(11), 1413–1457 (2004)MathSciNetzbMATHGoogle Scholar
  33. 33.
    De Oliveira, W., Sagastizábal, C., Lemaréchal, C.: Convex proximal bundle methods in depth: a unified analysis for inexact oracles. Math. Program. Ser. B 148(1–2), 241–277 (2014)MathSciNetzbMATHGoogle Scholar
  34. 34.
    Dobson, D.C., Vogel, C.R.: Convergence of an iterative method for total variation denoising. SIAM J. Numer. Anal. 34(5), 1779–1791 (1997)MathSciNetzbMATHGoogle Scholar
  35. 35.
    Douglas, J., Rachford, H.H.: On the numerical solution of heat conduction problems in two and three space variables. Trans. Am. Math. Soc. 82(2), 421–439 (1956)MathSciNetzbMATHGoogle Scholar
  36. 36.
    Eckstein, J., Silva, P.J.S.: Proximal methods for nonlinear programming: double regularization and inexact subproblems. Comput. Optim. Appl. 46(2), 279–304 (2010)MathSciNetzbMATHGoogle Scholar
  37. 37.
    Garrigos, G., Rosasco, L., Villa, S.: Iterative regularization via dual diagonal descent. J. Math. Imaging Vis. 60, 189–215 (2017). MathSciNetzbMATHGoogle Scholar
  38. 38.
    Goldstein, A.A.: Convex programming in Hilbert space. Bull. Am. Math. Soc. 70(5), 709–710 (1964)MathSciNetzbMATHGoogle Scholar
  39. 39.
    Goldstein, T., Osher, S.: The split Bregman method for L1-regularized problems. SIAM J. Imaging Sci. 2(2), 323–343 (2009)MathSciNetzbMATHGoogle Scholar
  40. 40.
    González-Andrade, S.: A preconditioned descent algorithm for variational inequalities of the second kind involving the \(p\)-Laplacian operator. Comput. Optim. Appl. 66(1), 123–162 (2017)MathSciNetzbMATHGoogle Scholar
  41. 41.
    Hare, W., Planiden, C.: Computing proximal points of convex functions with inexact subgradients. Set Valued Var. Anal. 26(3), 469–492 (2018)MathSciNetzbMATHGoogle Scholar
  42. 42.
    Hare, W., Sagastizábal, C., Solodov, M.: A proximal bundle method for nonsmooth nonconvex functions with inexact information. Comput. Optim. Appl. 63(1), 1–28 (2016)MathSciNetzbMATHGoogle Scholar
  43. 43.
    Lai, M.J., Messi, L.M.: Piecewise linear approximation of the continuous Rudin–Osher–Fatemi model for image denoising. SIAM J. Numer. Anal. 50(5), 2446–2466 (2012)MathSciNetzbMATHGoogle Scholar
  44. 44.
    Levitin, E.S., Polyak, B.T.: Constrained minimization problems. USSR Comput. Math. Math. Phys. 6, 1–50 (1966) (English translation of original paper in Zh. Vychisl. Mat. Fiz. 6, 787–823, 1965)Google Scholar
  45. 45.
    Martinet, B.: Régularisation d’inéquations variationnelles par approximations successives. Rev. Française Informat. Recherche Opérationnelle 4, 154–158 (1970)MathSciNetzbMATHGoogle Scholar
  46. 46.
    Molinari, C., Peypouquet, J.: Lagrangian penalization algorithm with parallel forward–backward splitting. J. Optim. Theory Appl. 177(2), 413–447 (2018)MathSciNetzbMATHGoogle Scholar
  47. 47.
    Moreau, J.J.: Fonctionnelles Convexes. Cours Collège de France, Paris (1967)Google Scholar
  48. 48.
    Nesterov, Y.: A method of solving a convex programming problem with convergence rate \(O(1/k^2)\). Sov. Math. Dokl. 27, 372–376 (1983)zbMATHGoogle Scholar
  49. 49.
    Opial, Z.: Weak convergence of the sequence of successive approximations for nonexpansive mappings. Bull. Am. Math. Soc. 73, 591–597 (1967)MathSciNetzbMATHGoogle Scholar
  50. 50.
    Parikh, N., Boyd, S.: Proximal algorithms. Found. Trends Optim. 1(3), 123–231 (2014)Google Scholar
  51. 51.
    Peypouquet, J.: Convex Optimization in Normed Spaces. Theory, Methods and Examples. With a Foreword by Hedy Attouch. Springer, Berlin (2015)Google Scholar
  52. 52.
    Reem, D., Reich, S.: Solutions to inexact resolvent inclusion problems with applications to nonlinear analysis and optimization. Rend. Circ. Mat. Palermo (2) 67(2), 337–371 (2018)MathSciNetzbMATHGoogle Scholar
  53. 53.
    Rockafellar, R.T.: Convex Analysis. Princeton University Press, Princeton (1970)zbMATHGoogle Scholar
  54. 54.
    Rockafellar, R.T.: Monotone operators and the proximal point algorithm. SIAM J. Control Optim. 14, 877–898 (1976)MathSciNetzbMATHGoogle Scholar
  55. 55.
    Rudin, L.I., Osher, S., Fatemi, E.: Nonlinear total variation based noise removal algorithms. Physica D 60, 259–268 (1992)MathSciNetzbMATHGoogle Scholar
  56. 56.
    Shen, J., Xia, Z.Q., Pang, L.P.: A proximal bundle method with inexact data for convex nondifferentiable minimization. Nonlinear Anal. 66(9), 2016–2027 (2007)MathSciNetzbMATHGoogle Scholar
  57. 57.
    Su, W., Boyd, S., Candès, E.J.: A differential equation for modeling Nesterov’s accelerated gradient method: theory and insights. NIPS 27, 2510–2518 (2014)zbMATHGoogle Scholar
  58. 58.
    Tai, X.C., Wu, C.: Augmented Lagrangian method, dual methods and split Bregman iteration for ROF model. In: Tai, X.C., Moerken, K., Lysaker, M., Lie, K.A. (eds.) Scale space and variational methods in computer vision, pp. 502–513 (2009)Google Scholar
  59. 59.
    Vogel, C.R., Oman, M.E.: Iterative methods for total variation denoising. SIAM J. Sci. Comput. 17(1), 227–238 (1996)MathSciNetzbMATHGoogle Scholar
  60. 60.
    Vu, B.C.: A splitting algorithm for coupled system of primal-dual monotone inclusions. J. Optim. Theory Appl. 164(3), 993–1025 (2015)MathSciNetzbMATHGoogle Scholar
  61. 61.
    Wu, C., Zhang, J., Tai, X.C.: Augmented Lagrangian method for total variation restoration with non-quadratic fidelity. Inverse Probl. Imaging 5(1), 237–261 (2011)MathSciNetzbMATHGoogle Scholar
  62. 62.
    Zhu, M., Wright, S.J., Chan, T.F.: Duality-based algorithms for total-variation-regularized image restoration. Comput. Optim. Appl. 47, 377–400 (2010)MathSciNetzbMATHGoogle Scholar

Copyright information

© Springer-Verlag GmbH Germany, part of Springer Nature 2019

Authors and Affiliations

  1. 1.ENSICAEN, CNRS, GREYCNormandie UniversitéCaen Cedex 4France
  2. 2.Departamento de Ingeniería Matemática, Centro de Modelamiento Matemático (CNRS UMI2807), FCFMUniversidad de ChileSantiagoChile
  3. 3.Departamento de MatemáticaUniversidad Técnica Federico Santa MaríaValparaisoChile

Personalised recommendations