Lagrangian Penalization Scheme with Parallel Forward–Backward Splitting

Article
  • 69 Downloads

Abstract

We propose a new iterative algorithm for the numerical approximation of the solutions to convex optimization problems and constrained variational inequalities, especially when the functions and operators involved have a separable structure on a product space, and exhibit some dissymmetry in terms of their component-wise regularity. Our method combines Lagrangian techniques and a penalization scheme with bounded parameters, with parallel forward–backward iterations. Conveniently combined, these techniques allow us to take advantage of the particular structure of the problem. We prove the weak convergence of the sequence generated by this scheme, along with worst-case convergence rates in the convex optimization setting, and for the strongly non-degenerate monotone operator case. Implementation issues related to the penalization of the constraint set are discussed, as well as applications in image recovery and non-Newtonian fluids modeling. A numerical illustration is also given, in order to prove the performance of the algorithm.

Keywords

Convex programming Forward–backward Lagrange multipliers Penalization 

Mathematics Subject Classification

49M37 90C25 

Notes

Acknowledgements

Supported by Fondecyt Grant 1140829 and Basal Project CMM Universidad de Chile. The first author was also supported by CONICYT scholarship CONICYT-PCHA/Doctorado Nacional/2016.

References

  1. 1.
    Goldstein, A.A.: Convex programming in Hilbert space. Bull. Am. Math. Soc. 70, 709–710 (1964)MathSciNetCrossRefMATHGoogle Scholar
  2. 2.
    Levitin, E.S., Polyak, B.T.: Constrained minimization problems. USSR Comput. Math. Math. Phys. 6, 1–50 (1966)CrossRefGoogle Scholar
  3. 3.
    Auslender, A., Crouzeix, J.-P., Fedit, P.: Penalty-proximal methods in convex programming. J. Optim. Theory Appl. 55(1), 1–21 (1987)MathSciNetCrossRefMATHGoogle Scholar
  4. 4.
    Cominetti, R., Courdurier, M.: Coupling general penalty schemes for convex programming with the steepest descent method and the proximal point algorithm. SIAM J. Optim. 13, 745–765 (2002)MathSciNetCrossRefMATHGoogle Scholar
  5. 5.
    Gabay, D., Mercier, B.: A dual algorithm for the solution of nonlinear variational problems via finite elements approximations. Comp. Math. Appl. 2, 17–40 (1976)CrossRefMATHGoogle Scholar
  6. 6.
    Glowinski, R., Marrocco, A.: Sur l’approximation par éléments finis d’ordre un, et la résolution par pénalisation-dualité d’une classe de problèmes de Dirichlet non linéaires. RAIRO 2, 41–76 (1975)MATHGoogle Scholar
  7. 7.
    Gabay, D.: Applications of the method of multipliers to variational inequalities. In: Fortin, M., Glowinski, R. (eds.) Augmented Lagrange Methods: Applications to the Solution of Boundary Valued Problems, pp. 299–331. North-Holland, Amsterdam (1983)CrossRefGoogle Scholar
  8. 8.
    Glowinski, R.: Numerical Methods for Nonlinear Variational Problems. Springer Series in Computational Physics. Springer, New York (1984)CrossRefGoogle Scholar
  9. 9.
    Glowinski, R., Le Tallec, P.: Augmented lagrangian and operator-splitting methods in nonlinear mechanics. SIAM Stud. Appl. Math. 9 (1989)Google Scholar
  10. 10.
    Gol’shtein, E.G., Tret’yakov, N.V.: Modified Lagrangians in convex programming and their generalizations. Point-to-set maps and mathematical programming. Math. Program. Stud. 10, 86–97 (1979)CrossRefGoogle Scholar
  11. 11.
    Rockafellar, R.T.: Augmented lagrangians and applications of the proximal point algorithm in convex programming. Math. Oper. Res. 1, 97–116 (1976)MathSciNetCrossRefMATHGoogle Scholar
  12. 12.
    Chen, G., Teboulle, M.: A proximal-based decomposition method for convex minimization problems. Math. Program. 64, 81–101 (1994)MathSciNetCrossRefMATHGoogle Scholar
  13. 13.
    Tseng, P.: Alternating projection-proximal methods for convex programming and variational inequalities. SIAM J. Optim. 7, 951–965 (1997)MathSciNetCrossRefMATHGoogle Scholar
  14. 14.
    Attouch, H., Soueycatt, M.: Augmented Lagrangian and proximal alternating direction methods of multipliers in Hilbert spaces. Applications to games, PDE’s and control. Pac. J. Optim. 5(1), 17–37 (2009)MathSciNetMATHGoogle Scholar
  15. 15.
    Xu, M.H.: Proximal alternating directions method for structured variational inequalities. J. Optim. Theory Appl. 134, 107–117 (2007)MathSciNetCrossRefMATHGoogle Scholar
  16. 16.
    Han, D., Yuan, X., Zhang, W.: An augmented Lagrangian based parallel splitting method for separable convex minimization with applications to image processing. Math. Comp. 83(289), 2263–2291 (2014)MathSciNetCrossRefMATHGoogle Scholar
  17. 17.
    Nocedal, J., Wright, S.J.: Numerical Optimization, 2nd edn. Springer, New York (2000)MATHGoogle Scholar
  18. 18.
    Ben-Tal, A., Zibulevsky, M.: Penalty/barrier multiplier methods for convex programming problems. SIAM J. Optim. 7(2), 347–366 (1997)MathSciNetCrossRefMATHGoogle Scholar
  19. 19.
    Frankel, P., Peypouquet, J.: Lagrangian-penalization algorithm for constrained optimization and variational inequalities. Set-Valued Var. Anal. 20(2), 169–185 (2012)MathSciNetCrossRefMATHGoogle Scholar
  20. 20.
    Necoara, I., Suykens, J.A.K.: Interior-point Lagrangian decomposition method for separable convex optimization. J. Optim. Theory Appl. 143(3), 567–588 (2009)MathSciNetCrossRefMATHGoogle Scholar
  21. 21.
    Sarmiento, O., Papa Quiroz, E.A., Oliveira, P.R.: A proximal multiplier method for separable convex minimization. Optimization 65(2), 501–537 (2016)MathSciNetCrossRefMATHGoogle Scholar
  22. 22.
    Cauchy, A.-L.: Méthode générale pour la résolution des systèmes d’équations simultanées. C. R. Acad. Sci. Paris 25, 536–538 (1847)Google Scholar
  23. 23.
    Martinet, B.: Régularisation d’inéquations variationnelles par approximations successives. Rev. Franaise Informat. Recherche Oprationnelle 4, 154–158 (1970)MATHGoogle Scholar
  24. 24.
    Brézis, H., Lions, P.L.: Produits infinis de résolvantes. Isr. J. Math. 29, 329–345 (1978)CrossRefMATHGoogle Scholar
  25. 25.
    Rockafellar, R.T.: Monotone operators and the proximal point algorithm. SIAM J. Control Optim. 14, 877–898 (1976)MathSciNetCrossRefMATHGoogle Scholar
  26. 26.
    Peypouquet, J.: Convex Optimization in Normed Spaces: Theory, Methods and Examples. Springer, New York (2015)CrossRefMATHGoogle Scholar
  27. 27.
    Nesterov, Y.: A method of solving a convex programming problem with convergence rate \(O(1/k^2)\). Sov. Math. Dokl. 27, 372–376 (1983)MATHGoogle Scholar
  28. 28.
    Attouch, H., Peypouquet, J.: The rate of convergence of Nesterov’s accelerated forward-backward method is actually faster than \(1/k^2\). SIAM J. Optim. 26(3), 1824–1834 (2016)MathSciNetCrossRefMATHGoogle Scholar
  29. 29.
    Chambolle, A., Dossal, C.: On the convergence of the iterates of the ”fast iterative shrinkage/thresholding algorithm”. J. Optim. Theory Appl. 166(3), 968–982 (2015)MathSciNetCrossRefMATHGoogle Scholar
  30. 30.
    Su, W., Boyd, S., Candès, E.J.: A differential equation for modeling Nesterov’s accelerated gradient method: theory and insights. NIPS 27, 2510–2518 (2014)MATHGoogle Scholar
  31. 31.
    Martinez-Legaz, J.E., Théra, M.: \(\varepsilon \)-subdifferentials in terms of subdifferentials. Set-Valued Anal. 4(4), 327–332 (1996)MathSciNetCrossRefMATHGoogle Scholar
  32. 32.
    Burachik, R.S., Iusem, A.N., Svaiter, B.F.: Enlargement of monotone operators with applications to variational inequalities. Set-Valued Anal. 5(2), 159–180 (1997)MathSciNetCrossRefMATHGoogle Scholar
  33. 33.
    Brézis, H.: Functional Analysis, Sobolev Spaces and Partial Differential Equations. Universitext, Springer, New York (2011)MATHGoogle Scholar
  34. 34.
    Bauschke, H.H., Combettes, P.L.: Convex Analysis and Monotone Operator Theory in Hilbert Spaces. Springer, CMS Books in Mathematics, New York (2011)Google Scholar
  35. 35.
    Boyd, S.P., Vandenberghe, L.: Convex Optimization. Cambridge University Press, Cambridge (2004)CrossRefMATHGoogle Scholar
  36. 36.
    Raguet, H., Fadili, J., Peyré, G.: Generalized forward–backward splitting. SIAM J. Imaging Sci. 6(3), 1199–1226 (2013)MathSciNetCrossRefMATHGoogle Scholar
  37. 37.
    Hansen, P.C., Nagy, J.G., O’Leary, D.P.: Deblurring images: matrices, spectra, and filtering. Fundam. Algorithms 3, SIAM (2006)Google Scholar
  38. 38.
    Beck, A., Teboulle, M.: A fast iterative shrinkage-thresholding algorithm for linear inverse problems. SIAM J. Imaging Sci. 2(1), 183–202 (2009)MathSciNetCrossRefMATHGoogle Scholar
  39. 39.
    Rudin, L.I., Osher, S., Fatemi, E.: Nonlinear total variation based noise removal algorithms. Phys. D 60, 259–268 (1992)MathSciNetCrossRefMATHGoogle Scholar
  40. 40.
    Chambolle, A., Lions, P.L.: Image recovery via total variation minimization and related problems. Proc. Numer. Math. 76, 167–188 (1997)MathSciNetCrossRefMATHGoogle Scholar
  41. 41.
    Beck, A., Teboulle, M.: Fast gradient-based algorithms for constrained total variation image denoising and deblurring problems. IEEE Trans. Image Proc. 18(11), 2419–2434 (2009)MathSciNetCrossRefMATHGoogle Scholar
  42. 42.
    Le, T., Chartrand, R., Asaki, T.J.: A variational approach to reconstructing images corrupted by Poisson noise. J. Math. Imaging Vision 27(3), 257–263 (2007)MathSciNetCrossRefGoogle Scholar
  43. 43.
    Sawatzky, A., Brune, C., Muller, J., Burger, M.: Total variation processing of images with poisson statistics. In: International Conference on Computer Analysis of Images and Patterns, pp. 533–540 (2009)Google Scholar
  44. 44.
    De los Reyes, J.C., Schönlieb, C.B.: Image denoising: learning the noise model via nonsmooth PDE-constrained optimization. Inverse Probl. Imaging 7(4), 1183–1214 (2013)MathSciNetCrossRefMATHGoogle Scholar
  45. 45.
    Gonzlez-Andrade, S.: A preconditioned descent algorithm for variational inequalities of the second kind involving the p-Laplacian operator. Comput. Optim. Appl. 66(1), 123–162 (2017)MathSciNetCrossRefMATHGoogle Scholar
  46. 46.
    Opial, Z.: Weak convergence of the sequence of successive approximations for nonexpansive mappings. Bull. Am. Math Soc. 73(4), 591–597 (1967)MathSciNetCrossRefMATHGoogle Scholar
  47. 47.
    Knopp, K.: Theory and Application of Infinite Series. Courier Corporation (2013)Google Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2018

Authors and Affiliations

  1. 1.Departamento de MatemáticaUniversidad Técnica Federico Santa MaríaValparaisoChile
  2. 2.GREYC CNRS UMR 6072, ENSICAENNormandie UniversitéCaen CedexFrance
  3. 3.Departamento de Ingeniería Matemática & Centro de Modelamiento Matemático (CNRS UMI2807), FCFMUniversidad de ChileSantiagoChile

Personalised recommendations