Advertisement

Applied Mathematics & Optimization

, Volume 80, Issue 3, pp 547–598 | Cite as

Convergence of a Relaxed Inertial Forward–Backward Algorithm for Structured Monotone Inclusions

  • Hedy AttouchEmail author
  • Alexandre Cabot
Article

Abstract

In a Hilbert space \({{\mathcal {H}}}\), we study the convergence properties of a class of relaxed inertial forward–backward algorithms. They aim to solve structured monotone inclusions of the form \(Ax + Bx \ni 0\) where \(A:{{\mathcal {H}}}\rightarrow 2^{{\mathcal {H}}}\) is a maximally monotone operator and \(B:{{\mathcal {H}}}\rightarrow {{\mathcal {H}}}\) is a cocoercive operator. We extend to this class of problems the acceleration techniques initially introduced by Nesterov, then developed by Beck and Teboulle in the case of structured convex minimization (FISTA). As an important element of our approach, we develop an inertial and parametric version of the Krasnoselskii–Mann theorem, where joint adjustment of the inertia and relaxation parameters plays a central role. This study comes as a natural extension of the techniques introduced by the authors for the study of relaxed inertial proximal algorithms. An illustration is given to the inertial Nash equilibration of a game combining non-cooperative and cooperative aspects.

Keywords

Structured monotone inclusions Inertial forward–backward algorithms Cocoercive operators Relaxation Convergence rate Inertial Krasnoselskii–Mann iteration Nash equilibration 

Mathematics Subject Classification

49M37 65K05 65K10 90C25 

Notes

References

  1. 1.
    Álvarez, F.: On the minimizing property of a second-order dissipative system in Hilbert spaces. SIAM J. Control Optim. 38(4), 1102–1119 (2000)MathSciNetCrossRefGoogle Scholar
  2. 2.
    Álvarez, F.: Weak convergence of a relaxed and inertial hybrid projection-proximal point algorithm for maximal monotone operators in Hilbert space. SIAM J. Optim. 14, 773–782 (2004)MathSciNetCrossRefGoogle Scholar
  3. 3.
    Álvarez, F., Attouch, H.: The Heavy Ball with Friction Dynamical System for Convex Constrained Minimization Problems, Optimization, Namur, 1998. Lecture Notes in Economics and Mathematical Systems, vol. 481, pp. 25–35. Springer, Berlin (2000)Google Scholar
  4. 4.
    Álvarez, F., Attouch, H.: An inertial proximal method for maximal monotone operators via discretization of a nonlinear oscillator with damping. Set-Valued Anal. 9(1–2), 3–11 (2001)MathSciNetCrossRefGoogle Scholar
  5. 5.
    Attouch, H., Cabot, A.: Convergence rates of inertial forward–backward algorithms. SIAM J. Optim. 28(1), 849–874 (2018)MathSciNetCrossRefGoogle Scholar
  6. 6.
    Attouch, H., Cabot, A.: Convergence of a relaxed inertial proximal algorithm for maximally monotone operators, 2018, HAL-01708905 (2018)Google Scholar
  7. 7.
    Attouch, H., Chbani, Z., Peypouquet, J., Redont, P.: Fast convergence of inertial dynamics and algorithms with asymptotic vanishing damping. Math. Progr. Ser. B 168, 123–175 (2018)Google Scholar
  8. 8.
    Attouch, H., Maingé, P.E.: Asymptotic behavior of second-order dissipative evolution equations combining potential with non-potential effects, ESAIM: control. Optim. Calc. Var. 17, 836–857 (2010)CrossRefGoogle Scholar
  9. 9.
    Attouch, H., Peypouquet, J.: The rate of convergence of Nesterov’s accelerated forward-backward method is actually faster than \(\frac{1}{k^2}\). SIAM J. Optim. 26(3), 1824–1834 (2016)MathSciNetCrossRefGoogle Scholar
  10. 10.
    Attouch, H., Peypouquet, J.: Convergence of inertial dynamics and proximal algorithms governed by maximal monotone operators. Math. Program., Ser. B.  https://doi.org/10.1007/s10107-018-1252-x MathSciNetCrossRefGoogle Scholar
  11. 11.
    Aujol, J.-F., Dossal, C.: Stability of over-relaxations for the forward–backward algorithm, application to FISTA. SIAM J. Optim. 25(4), 2408–2433 (2015)MathSciNetCrossRefGoogle Scholar
  12. 12.
    Baillon, J.B., Brézis, H.: Une remarque sur le comportement asymptotique des semigroupes non linéaires. Houston J. Math. 2(1), 5–7 (1976)MathSciNetzbMATHGoogle Scholar
  13. 13.
    Baillon, J.B., Haddad, G.: Quelques propriétés des opérateurs angle-bornés et \(n\)-cycliquement monotones. Israël J. Math. 26, 137–150 (1977)MathSciNetCrossRefGoogle Scholar
  14. 14.
    Bauschke, H., Combettes, P.: Convex Analysis and Monotone Operator Theory in Hilbert Spaces. CMS Books in Mathematics. Springer, New York (2011)CrossRefGoogle Scholar
  15. 15.
    Beck, A., Teboulle, M.: A fast iterative shrinkage-thresholding algorithm for linear inverse problems. SIAM J. Imaging Sci. 2(1), 183–202 (2009)MathSciNetCrossRefGoogle Scholar
  16. 16.
    Bot, R.I., Csetnek, E.R.: Second order forward–backward dynamical systems for monotone inclusion problems. SIAM J. Control Optim. 54(3), 1423–1443 (2016)MathSciNetCrossRefGoogle Scholar
  17. 17.
    Bot, R.I., Csetnek, E.R., Hendrich, C.: Inertial Douglas-Rachford splitting for monotone inclusion problems. Appl. Math. Comput. 256, 472–487 (2015)MathSciNetzbMATHGoogle Scholar
  18. 18.
    Brézis, H.: Opérateurs maximaux monotones dans les espaces de Hilbert et équations d’évolution. Lecture Notes 5, North Holland (1972)Google Scholar
  19. 19.
    Brézis, H., Browder, F.E.: Nonlinear ergodic theorems. Bull. Am. Math. Soc. 82(6), 959–961 (1976)MathSciNetCrossRefGoogle Scholar
  20. 20.
    Briceño-Arias, L.M., Combettes, P.L.: A monotone\(+\)skew splitting model for composite monotone inclusions in duality. SIAM J. Optim. 21, 1230–1250 (2011)MathSciNetCrossRefGoogle Scholar
  21. 21.
    Chambolle, A., Dossal, Ch.: On the convergence of the iterates of the “fast iterative shrinkage/thresholding algorithm”. J. Optim. Theory Appl. 166, 968–982 (2015)MathSciNetCrossRefGoogle Scholar
  22. 22.
    Chen, G., Rockafellar, R.T.: Convergence rates in forward–backward splitting. SIAM J. Optim. 7, 421–444 (1997)MathSciNetCrossRefGoogle Scholar
  23. 23.
    Combettes, P.L., Wajs, V.R.: Signal recovery by proximal forward–backward splitting. Multiscale Model. Simul. 4, 1168–1200 (2005)MathSciNetCrossRefGoogle Scholar
  24. 24.
    Combettes, P.L., Yamada, I.: Compositions and convex combinations of averaged nonexpansive operators. J. Math. Anal. Appl. 425, 55–70 (2015)MathSciNetCrossRefGoogle Scholar
  25. 25.
    Eckstein, J.: Splitting methods for monotone operators with applications to parallel optimization, PhD thesis, Massachusetts Institute of Technology (1989)Google Scholar
  26. 26.
    Eckstein, J., Bertsekas, D.P.: On the Douglas-Rachford splitting method and the proximal point algorithm for maximal monotone operators. Math. Progr. 55, 293–318 (1992)MathSciNetCrossRefGoogle Scholar
  27. 27.
    Gabay, D.: Applications of the Method of Multipliers to Variational Inequalities. In: Fortin, M., Glowinski, R. (eds.) Augmented Lagrangian Methods: Applications to the Solution of Boundary Value Problems, pp. 299–331. North-Holland, Amsterdam (1983)CrossRefGoogle Scholar
  28. 28.
    Hofbauer, J., Sorin, S.: Best response dynamics for continuous zero-sum games. Discret. Contin. Dyn. Syst. Ser. B 6, 215–224 (2006)MathSciNetzbMATHGoogle Scholar
  29. 29.
    Iutzeler, F., Hendrickx, J.M.: Generic online acceleration scheme for optimization algorithms via relaxation and inertia (2017), arXiv:1603.05398v3 [math.OC]MathSciNetCrossRefGoogle Scholar
  30. 30.
    Kim, D., Fessler, J.A.: Optimized first-order methods for smooth convex minimization. Math. Progr. 159(1), 81–107 (2016)MathSciNetCrossRefGoogle Scholar
  31. 31.
    Liang, J., Fadili, J., Peyré, G.: Local linear convergence of forward-backward under partial smoothness. In: Advances in Neural Information Processing Systems, pp. 1970–1978 (2014)Google Scholar
  32. 32.
    Lions, P.L., Mercier, B.: Splitting algorithms for the sum of two nonlinear operators. SIAM J. Numer. Anal. 16, 964–979 (1979)MathSciNetCrossRefGoogle Scholar
  33. 33.
    Lorenz, D.A., Pock, T.: An inertial forward–backward algorithm for monotone inclusions. J. Math. Imaging Vis. 51, 311–325 (2015)MathSciNetCrossRefGoogle Scholar
  34. 34.
    Maingé, P.-E.: Convergence theorems for inertial KM-type algorithms. J. Comput. Appl. Math. 219, 223–236 (2008)MathSciNetCrossRefGoogle Scholar
  35. 35.
    Moudafi, A., Oliny, M.: Convergence of a splitting inertial proximal method for monotone operators. J. Comput. Appl. Math. 155, 447–454 (2003)MathSciNetCrossRefGoogle Scholar
  36. 36.
    Nesterov, Y.: A method of solving a convex programming problem with convergence rate \(O(1/k^2)\). Soviet Math. Doklady 27, 372–376 (1983)zbMATHGoogle Scholar
  37. 37.
    Nesterov, Y.: Introductory Lectures on Convex Optimization: A Basic Course, Volume 87 of Applied Optimization. Kluwer Academic Publishers, Boston (2004)Google Scholar
  38. 38.
    Opial, Z.: Weak convergence of the sequence of successive approximations for nonexpansive mappings. Bull. Am. Math. Soc. 73, 591–597 (1967)MathSciNetCrossRefGoogle Scholar
  39. 39.
    Parikh, N., Boyd, S.: Proximal algorithms. Found. Trends Optim. 1, 123–231 (2013)Google Scholar
  40. 40.
    Passty, G.B.: Ergodic convergence to a zero of the sum of monotone operators in Hilbert space. J. Math. Anal. Appl. 72, 383–390 (1979)MathSciNetCrossRefGoogle Scholar
  41. 41.
    Polyak, B.T.: Some methods of speeding up the convergence of iteration methods. USSR Comput. Math. Math. Phys. 4(5), 1–17 (1964)CrossRefGoogle Scholar
  42. 42.
    Raguet, H., Fadili, J., Peyré, G.: A generalized forward–backward splitting. SIAM J. Imaging Sci. 6, 1199–1226 (2013)MathSciNetCrossRefGoogle Scholar
  43. 43.
    Rockafellar, R.T.: Monotone Operators Associated with Saddle-functions and Minimax Problems. Nonlinear Functional Analysis, Vol. I. In: Browder, F.E. (ed.) Proceedings of Symposia in Pure Mathematical American Mathematical Society (1970)Google Scholar
  44. 44.
    Su, W., Boyd, S., Candès, E.J.: A differential equation for modeling Nesterov’s accelerated gradient method: theory and insights. Neural Inf. Process. Syst. 27, 2510–2518 (2014)zbMATHGoogle Scholar
  45. 45.
    Tseng, P.: Applications of a splitting algorithm to decomposition in convex programming and variational inequalities. SIAM J. Control Optim. 29, 119–138 (1991)MathSciNetCrossRefGoogle Scholar
  46. 46.
    Tseng, P.: A modified forward–backward splitting method for maximal monotone mappings. SIAM J. Control Optim. 38, 431–446 (2000)MathSciNetCrossRefGoogle Scholar
  47. 47.
    Tseng, P.: On accelerated proximal gradient methods for convex-concave optimization, 2008. Technical report (2008)Google Scholar
  48. 48.
    Villa, S., Salzo, S., Baldassarres, L., Verri, A.: Accelerated and inexact forward–backward. SIAM J. Optim. 23(3), 1607–1633 (2013)MathSciNetCrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2019

Authors and Affiliations

  1. 1.Institut Montpelliérain Alexander Grothendieck, UMR 5149 CNRSUniversité MontpellierMontpellier Cedex 5France
  2. 2.Institut de Mathématiques de Bourgogne, UMR 5584 CNRSUniversité Bourgogne Franche-ComtéDijonFrance

Personalised recommendations