Lagrangian Penalization Scheme with Parallel Forward–Backward Splitting

  • Cesare Molinari
  • Juan Peypouquet


We propose a new iterative algorithm for the numerical approximation of the solutions to convex optimization problems and constrained variational inequalities, especially when the functions and operators involved have a separable structure on a product space, and exhibit some dissymmetry in terms of their component-wise regularity. Our method combines Lagrangian techniques and a penalization scheme with bounded parameters, with parallel forward–backward iterations. Conveniently combined, these techniques allow us to take advantage of the particular structure of the problem. We prove the weak convergence of the sequence generated by this scheme, along with worst-case convergence rates in the convex optimization setting, and for the strongly non-degenerate monotone operator case. Implementation issues related to the penalization of the constraint set are discussed, as well as applications in image recovery and non-Newtonian fluids modeling. A numerical illustration is also given, in order to prove the performance of the algorithm.


Convex programming Forward–backward Lagrange multipliers Penalization 

Mathematics Subject Classification

49M37 90C25 



Supported by Fondecyt Grant 1140829 and Basal Project CMM Universidad de Chile. The first author was also supported by CONICYT scholarship CONICYT-PCHA/Doctorado Nacional/2016.


  1. 1.
    Goldstein, A.A.: Convex programming in Hilbert space. Bull. Am. Math. Soc. 70, 709–710 (1964)MathSciNetCrossRefzbMATHGoogle Scholar
  2. 2.
    Levitin, E.S., Polyak, B.T.: Constrained minimization problems. USSR Comput. Math. Math. Phys. 6, 1–50 (1966)CrossRefGoogle Scholar
  3. 3.
    Auslender, A., Crouzeix, J.-P., Fedit, P.: Penalty-proximal methods in convex programming. J. Optim. Theory Appl. 55(1), 1–21 (1987)MathSciNetCrossRefzbMATHGoogle Scholar
  4. 4.
    Cominetti, R., Courdurier, M.: Coupling general penalty schemes for convex programming with the steepest descent method and the proximal point algorithm. SIAM J. Optim. 13, 745–765 (2002)MathSciNetCrossRefzbMATHGoogle Scholar
  5. 5.
    Gabay, D., Mercier, B.: A dual algorithm for the solution of nonlinear variational problems via finite elements approximations. Comp. Math. Appl. 2, 17–40 (1976)CrossRefzbMATHGoogle Scholar
  6. 6.
    Glowinski, R., Marrocco, A.: Sur l’approximation par éléments finis d’ordre un, et la résolution par pénalisation-dualité d’une classe de problèmes de Dirichlet non linéaires. RAIRO 2, 41–76 (1975)zbMATHGoogle Scholar
  7. 7.
    Gabay, D.: Applications of the method of multipliers to variational inequalities. In: Fortin, M., Glowinski, R. (eds.) Augmented Lagrange Methods: Applications to the Solution of Boundary Valued Problems, pp. 299–331. North-Holland, Amsterdam (1983)CrossRefGoogle Scholar
  8. 8.
    Glowinski, R.: Numerical Methods for Nonlinear Variational Problems. Springer Series in Computational Physics. Springer, New York (1984)CrossRefGoogle Scholar
  9. 9.
    Glowinski, R., Le Tallec, P.: Augmented lagrangian and operator-splitting methods in nonlinear mechanics. SIAM Stud. Appl. Math. 9 (1989)Google Scholar
  10. 10.
    Gol’shtein, E.G., Tret’yakov, N.V.: Modified Lagrangians in convex programming and their generalizations. Point-to-set maps and mathematical programming. Math. Program. Stud. 10, 86–97 (1979)CrossRefGoogle Scholar
  11. 11.
    Rockafellar, R.T.: Augmented lagrangians and applications of the proximal point algorithm in convex programming. Math. Oper. Res. 1, 97–116 (1976)MathSciNetCrossRefzbMATHGoogle Scholar
  12. 12.
    Chen, G., Teboulle, M.: A proximal-based decomposition method for convex minimization problems. Math. Program. 64, 81–101 (1994)MathSciNetCrossRefzbMATHGoogle Scholar
  13. 13.
    Tseng, P.: Alternating projection-proximal methods for convex programming and variational inequalities. SIAM J. Optim. 7, 951–965 (1997)MathSciNetCrossRefzbMATHGoogle Scholar
  14. 14.
    Attouch, H., Soueycatt, M.: Augmented Lagrangian and proximal alternating direction methods of multipliers in Hilbert spaces. Applications to games, PDE’s and control. Pac. J. Optim. 5(1), 17–37 (2009)MathSciNetzbMATHGoogle Scholar
  15. 15.
    Xu, M.H.: Proximal alternating directions method for structured variational inequalities. J. Optim. Theory Appl. 134, 107–117 (2007)MathSciNetCrossRefzbMATHGoogle Scholar
  16. 16.
    Han, D., Yuan, X., Zhang, W.: An augmented Lagrangian based parallel splitting method for separable convex minimization with applications to image processing. Math. Comp. 83(289), 2263–2291 (2014)MathSciNetCrossRefzbMATHGoogle Scholar
  17. 17.
    Nocedal, J., Wright, S.J.: Numerical Optimization, 2nd edn. Springer, New York (2000)zbMATHGoogle Scholar
  18. 18.
    Ben-Tal, A., Zibulevsky, M.: Penalty/barrier multiplier methods for convex programming problems. SIAM J. Optim. 7(2), 347–366 (1997)MathSciNetCrossRefzbMATHGoogle Scholar
  19. 19.
    Frankel, P., Peypouquet, J.: Lagrangian-penalization algorithm for constrained optimization and variational inequalities. Set-Valued Var. Anal. 20(2), 169–185 (2012)MathSciNetCrossRefzbMATHGoogle Scholar
  20. 20.
    Necoara, I., Suykens, J.A.K.: Interior-point Lagrangian decomposition method for separable convex optimization. J. Optim. Theory Appl. 143(3), 567–588 (2009)MathSciNetCrossRefzbMATHGoogle Scholar
  21. 21.
    Sarmiento, O., Papa Quiroz, E.A., Oliveira, P.R.: A proximal multiplier method for separable convex minimization. Optimization 65(2), 501–537 (2016)MathSciNetCrossRefzbMATHGoogle Scholar
  22. 22.
    Cauchy, A.-L.: Méthode générale pour la résolution des systèmes d’équations simultanées. C. R. Acad. Sci. Paris 25, 536–538 (1847)Google Scholar
  23. 23.
    Martinet, B.: Régularisation d’inéquations variationnelles par approximations successives. Rev. Franaise Informat. Recherche Oprationnelle 4, 154–158 (1970)zbMATHGoogle Scholar
  24. 24.
    Brézis, H., Lions, P.L.: Produits infinis de résolvantes. Isr. J. Math. 29, 329–345 (1978)CrossRefzbMATHGoogle Scholar
  25. 25.
    Rockafellar, R.T.: Monotone operators and the proximal point algorithm. SIAM J. Control Optim. 14, 877–898 (1976)MathSciNetCrossRefzbMATHGoogle Scholar
  26. 26.
    Peypouquet, J.: Convex Optimization in Normed Spaces: Theory, Methods and Examples. Springer, New York (2015)CrossRefzbMATHGoogle Scholar
  27. 27.
    Nesterov, Y.: A method of solving a convex programming problem with convergence rate \(O(1/k^2)\). Sov. Math. Dokl. 27, 372–376 (1983)zbMATHGoogle Scholar
  28. 28.
    Attouch, H., Peypouquet, J.: The rate of convergence of Nesterov’s accelerated forward-backward method is actually faster than \(1/k^2\). SIAM J. Optim. 26(3), 1824–1834 (2016)MathSciNetCrossRefzbMATHGoogle Scholar
  29. 29.
    Chambolle, A., Dossal, C.: On the convergence of the iterates of the ”fast iterative shrinkage/thresholding algorithm”. J. Optim. Theory Appl. 166(3), 968–982 (2015)MathSciNetCrossRefzbMATHGoogle Scholar
  30. 30.
    Su, W., Boyd, S., Candès, E.J.: A differential equation for modeling Nesterov’s accelerated gradient method: theory and insights. NIPS 27, 2510–2518 (2014)zbMATHGoogle Scholar
  31. 31.
    Martinez-Legaz, J.E., Théra, M.: \(\varepsilon \)-subdifferentials in terms of subdifferentials. Set-Valued Anal. 4(4), 327–332 (1996)MathSciNetCrossRefzbMATHGoogle Scholar
  32. 32.
    Burachik, R.S., Iusem, A.N., Svaiter, B.F.: Enlargement of monotone operators with applications to variational inequalities. Set-Valued Anal. 5(2), 159–180 (1997)MathSciNetCrossRefzbMATHGoogle Scholar
  33. 33.
    Brézis, H.: Functional Analysis, Sobolev Spaces and Partial Differential Equations. Universitext, Springer, New York (2011)zbMATHGoogle Scholar
  34. 34.
    Bauschke, H.H., Combettes, P.L.: Convex Analysis and Monotone Operator Theory in Hilbert Spaces. Springer, CMS Books in Mathematics, New York (2011)Google Scholar
  35. 35.
    Boyd, S.P., Vandenberghe, L.: Convex Optimization. Cambridge University Press, Cambridge (2004)CrossRefzbMATHGoogle Scholar
  36. 36.
    Raguet, H., Fadili, J., Peyré, G.: Generalized forward–backward splitting. SIAM J. Imaging Sci. 6(3), 1199–1226 (2013)MathSciNetCrossRefzbMATHGoogle Scholar
  37. 37.
    Hansen, P.C., Nagy, J.G., O’Leary, D.P.: Deblurring images: matrices, spectra, and filtering. Fundam. Algorithms 3, SIAM (2006)Google Scholar
  38. 38.
    Beck, A., Teboulle, M.: A fast iterative shrinkage-thresholding algorithm for linear inverse problems. SIAM J. Imaging Sci. 2(1), 183–202 (2009)MathSciNetCrossRefzbMATHGoogle Scholar
  39. 39.
    Rudin, L.I., Osher, S., Fatemi, E.: Nonlinear total variation based noise removal algorithms. Phys. D 60, 259–268 (1992)MathSciNetCrossRefzbMATHGoogle Scholar
  40. 40.
    Chambolle, A., Lions, P.L.: Image recovery via total variation minimization and related problems. Proc. Numer. Math. 76, 167–188 (1997)MathSciNetCrossRefzbMATHGoogle Scholar
  41. 41.
    Beck, A., Teboulle, M.: Fast gradient-based algorithms for constrained total variation image denoising and deblurring problems. IEEE Trans. Image Proc. 18(11), 2419–2434 (2009)MathSciNetCrossRefzbMATHGoogle Scholar
  42. 42.
    Le, T., Chartrand, R., Asaki, T.J.: A variational approach to reconstructing images corrupted by Poisson noise. J. Math. Imaging Vision 27(3), 257–263 (2007)MathSciNetCrossRefGoogle Scholar
  43. 43.
    Sawatzky, A., Brune, C., Muller, J., Burger, M.: Total variation processing of images with poisson statistics. In: International Conference on Computer Analysis of Images and Patterns, pp. 533–540 (2009)Google Scholar
  44. 44.
    De los Reyes, J.C., Schönlieb, C.B.: Image denoising: learning the noise model via nonsmooth PDE-constrained optimization. Inverse Probl. Imaging 7(4), 1183–1214 (2013)MathSciNetCrossRefzbMATHGoogle Scholar
  45. 45.
    Gonzlez-Andrade, S.: A preconditioned descent algorithm for variational inequalities of the second kind involving the p-Laplacian operator. Comput. Optim. Appl. 66(1), 123–162 (2017)MathSciNetCrossRefzbMATHGoogle Scholar
  46. 46.
    Opial, Z.: Weak convergence of the sequence of successive approximations for nonexpansive mappings. Bull. Am. Math Soc. 73(4), 591–597 (1967)MathSciNetCrossRefzbMATHGoogle Scholar
  47. 47.
    Knopp, K.: Theory and Application of Infinite Series. Courier Corporation (2013)Google Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2018

Authors and Affiliations

  1. 1.Departamento de MatemáticaUniversidad Técnica Federico Santa MaríaValparaisoChile
  2. 2.GREYC CNRS UMR 6072, ENSICAENNormandie UniversitéCaen CedexFrance
  3. 3.Departamento de Ingeniería Matemática & Centro de Modelamiento Matemático (CNRS UMI2807), FCFMUniversidad de ChileSantiagoChile

Personalised recommendations