Skip to main content
SpringerLink
Log in

Convergence of Inertial Dynamics Driven by Sums of Potential and Nonpotential Operators with Implicit Newton-Like Damping

  • Published:
Journal of Optimization Theory and Applications Aims and scope Submit manuscript

Abstract

We analyze the convergence properties when the time t tends to infinity of the trajectories generated by damped inertial dynamics which are driven by the sum of potential and nonpotential operators. Precisely, we seek to reach asymptotically the zeros of a maximally monotone operator which is the sum of a potential operator (the gradient of a continuously differentiable convex function) and of a monotone and cocoercive nonpotential operator. As an original feature, in addition to the viscous friction, the dynamic involves implicit Newton-type damping. This contrasts with the authors’ previous study where explicit Newton-type damping was considered, which, for the potential term corresponds to Hessian-driven damping. We show the weak convergence, as time tends to infinity, of the generated trajectories toward the zeros of the sum of the potential and nonpotential operators. Our results are based on Lyapunov analysis and appropriate setting of the damping parameters. The introduction of geometric dampings allows to control and attenuate the oscillations known for the viscous damping of inertial methods. Rewriting the second-order evolution equation as a system involving only first-order derivative in time and space allows us to extend the convergence analysis to nonsmooth convex potentials. The main part of our study concerns the autonomous case with positive fixed parameters. We complete it with some first results concerning the nonautonomous case, and which are based on a recent acceleration method using time scaling and averaging. These results open the door to the design of new first-order accelerated algorithms in optimization taking into account the specific properties of potential and nonpotential terms. The proofs and techniques are original due to the presence of the nonpotential term.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

Notes

  1. At several places, assumption (A1) will be relaxed, just assuming \(\nabla f \) to be Lipschitz continuous on the bounded sets

  2. We thank the anonymous reviewer for this judicious suggestion

References

  1. Abbas, B., Attouch, H.: Dynamical systems and forward-backward algorithms associated with the sum of a convex subdifferential and a monotone cocoercive operator. Optimization 64(10), 2223–2252 (2015)

    Article  MathSciNet  MATH  Google Scholar 

  2. Abbas, B., Attouch, H., Svaiter, B.F.: Newton-like dynamics and forward-backward methods for structured monotone inclusions in Hilbert spaces. J. Optim. Theor. Appl. 161(2), 331–360 (2014)

    Article  MathSciNet  MATH  Google Scholar 

  3. Adly, S., Attouch, H.: Finite convergence of proximal-gradient inertial algorithms combining dry friction with Hessian-driven damping. SIAM J. Optim. 30(3), 2134–2162 (2020)

    Article  MathSciNet  MATH  Google Scholar 

  4. Adly, S., Attouch, H., Vo, V.N.: Asymptotic behavior of Newton-like inertial dynamics involving the sum of potential and nonpotential terms. Fixed Point Theor. Algorithms Sci. Eng. 17, 30 (2021)

    MathSciNet  MATH  Google Scholar 

  5. Adly, S., Attouch, H., Vo, V.N.: Newton-type inertial algorithms for solving monotone equations governed by sums of potential and nonpotential operators. Appl. Math. Optim. 85(3), 31 (2022)

    Article  MathSciNet  MATH  Google Scholar 

  6. Alecsa, C.D., László, S., Pinta, T.: An extension of the second order dynamical system that models Nesterov’s convex gradient method. Appl. Math. Optim. 84(2), 1687–1716 (2021)

    Article  MathSciNet  MATH  Google Scholar 

  7. Alvarez, F.: On the minimizing property of a second order dissipative system in Hilbert space. SIAM J. Control Optim. 38(4), 1102–1119 (2000)

    Article  MathSciNet  MATH  Google Scholar 

  8. Apidopoulos, V., Aujol, J.-F., Dossal, Ch.: The differential inclusion modeling FISTA algorithm and optimality of convergence rate in the case \(b\le 3\). SIAM J. Optim. 28, 551–574 (2018)

    Article  MathSciNet  MATH  Google Scholar 

  9. Alvarez, F., Attouch, H.: An inertial proximal method for maximal monotone operators via discretization of a nonlinear oscillator with damping. Set Valued Anal. 9(1–2), 3–11 (2001)

    Article  MathSciNet  MATH  Google Scholar 

  10. Attouch H., Boţ R.I., Nguyen D.-K., Fast convex optimization via time scale and averaging of the steepest descent, arXiv:2208.08260v1 [math.OC] Aug (2022)

  11. Alvarez, F., Attouch, H., Bolte, J., Redont, P.: A second-order gradient-like dissipative dynamical system with Hessian-driven damping: application to optimization and mechanics. J. Math. Pures Appl. 81(8), 747–779 (2002)

    Article  MathSciNet  MATH  Google Scholar 

  12. Attouch, H., Cabot, A.: Convergence rates of inertial forward-backward algorithms. SIAM J. Optim. 28(1), 849–874 (2018)

    Article  MathSciNet  MATH  Google Scholar 

  13. Attouch, H., Chbani, Z., Peypouquet, J., Redont, P.: Fast convergence of inertial dynamics and algorithms with asymptotic vanishing viscosity. Math. Program. Ser. B 168, 123–175 (2018)

    Article  MathSciNet  MATH  Google Scholar 

  14. Attouch, H., Chbani, Z., Riahi, H.: Rate of convergence of the Nesterov accelerated gradient method in the subcritical case \(\alpha \le 3\). ESAIM Control Optim. Calc. Var. 25(2), 34 (2019)

    MathSciNet  MATH  Google Scholar 

  15. Attouch, H., Chbani, Z., Fadili, J., Riahi, H.: First-order algorithms via inertial systems with Hessian driven damping. Math. Program. Ser. A 193(1), 113–155 (2022)

    Article  MathSciNet  MATH  Google Scholar 

  16. Attouch, H., Chbani, Z., Fadili, J., Riahi, H.: Convergence of iterates for first-order optimization algorithms with inertia and Hessian driven damping. Optimization (2021). https://doi.org/10.1080/02331934.2021.2009828

    Article  MATH  Google Scholar 

  17. Attouch, H., Chbani, Z., Fadili, J., Riahi, H.: Fast convergence of dynamical ADMM via time scaling of damped inertial dynamics. J. Optim. Theor. Appl. 193, 704–736 (2022)

    Article  MathSciNet  MATH  Google Scholar 

  18. Attouch, H., Fadili, J., Kungurtsev, V.: On the effect of perturbations in first-order optimization methods with inertia and Hessian driven damping. Evolution Equations and Control Theory. 12(1), (2022)

  19. Attouch, H., László, S.C.: Continuous Newton-like inertial dynamics for monotone inclusions. Set Valued Var. Anal. 29(3), 555–581 (2021)

    Article  MathSciNet  MATH  Google Scholar 

  20. Attouch, H., László, S.C.: Newton-like inertial dynamics and proximal algorithms governed by maximally monotone operators. SIAM J. Optim. 30(4), 3252–3283 (2020)

    Article  MathSciNet  MATH  Google Scholar 

  21. Attouch, H., Maingé, P.E.: Asymptotic behavior of second order dissipative evolution equations combining potential with nonpotential effects. ESAIM Control Optim. Calc. Var. 17(3), 836–857 (2011)

    Article  MathSciNet  MATH  Google Scholar 

  22. Attouch, H., Marques Alves, M., Svaiter, B.F.: A dynamic approach to a proximal-Newton method for monotone inclusions in Hilbert Spaces, with complexity \(\cal{O} (1/n^2)\). J. Convex Anal. 23(1), 139–180 (2016)

    MathSciNet  MATH  Google Scholar 

  23. Attouch, H., Peypouquet, J.: Convergence of inertial dynamics and proximal algorithms governed by maximal monotone operators. Math. Program. 174(1–2), 391–432 (2019)

    Article  MathSciNet  MATH  Google Scholar 

  24. Attouch, H., Peypouquet, J., Redont, P.: Fast convex minimization via inertial dynamics with Hessian driven damping. J. Differ. Equ. 261(10), 5734–5783 (2016)

    Article  MATH  Google Scholar 

  25. Attouch, H., Redont, P., Svaiter, B.F.: Global convergence of a closed-loop regularized Newton method for solving monotone inclusions in Hilbert spaces. J. Optim. Theor. Appl. 157(3), 624–650 (2013)

    Article  MathSciNet  MATH  Google Scholar 

  26. Attouch, H., Svaiter, B.F.: A continuous dynamical Newton-like approach to solving monotone inclusions. SIAM J. Control Optim. 49(2), 574–598 (2011)

    Article  MathSciNet  MATH  Google Scholar 

  27. Aujol, J.-F., Dossal, C., Hoàng, V., Labarrière, H., Rondepierre A.: Fast convergence of inertial dynamics with hessian-driven damping under geometry assumptions. arXiv:2206.06853, (2022)

  28. Baillon, J.-B., Haddad, G.: Quelques propriétés des opérateurs angles-bornés et n-cycliquement monotones. Israel J. Math. 26, 137–150 (1977)

    Article  MathSciNet  MATH  Google Scholar 

  29. Bauschke, H., Combettes, P.L.: Convex Analysis and Monotone Operator Theory in Hilbert spaces. In: CMS Books in Mathematics. Springer (2011)

    Google Scholar 

  30. Beck, A., Teboulle, M.: A fast iterative shrinkage-thresholding algorithm for linear inverse problems. SIAM J. Imaging Sci. 2(1), 183–202 (2009)

    Article  MathSciNet  MATH  Google Scholar 

  31. Boţ, R.I., Csetnek, E.R.: Second order forward-backward dynamical systems for monotone inclusion problems. SIAM J. Control Optim. 54, 1423–1443 (2016)

    Article  MathSciNet  MATH  Google Scholar 

  32. Boţ, R.I., Csetnek, E.R., László, S.C.: Tikhonov regularization of a second order dynamical system with Hessian damping. Math. Program. Ser. B 189(1–2), 151–186 (2021)

    Article  MathSciNet  MATH  Google Scholar 

  33. Brézis, H.: Opérateurs maximaux monotones dans les espaces de Hilbert et équations d’évolution. Lecture Notes 5. North Holland (1972)

  34. Castera, C., Bolte, J., Févotte, C., Pauwels, E.: An Inertial Newton algorithm for deep learning. (2019), HAL-02140748

  35. Chambolle, A., Dossal, Ch.: On the convergence of the iterates of the fast iterative shrinkage thresholding algorithm. J. Optim. Theor. Appl. 166, 968–982 (2015)

    Article  MathSciNet  MATH  Google Scholar 

  36. Gelfand I.M. , Tsetlin M.: Printszip nelokalnogo poiska v sistemah avtomatich. Optimizatsii, Dokl. AN SSSR. 137, 295–298 (1961) (in Russian)

  37. Lin, T., Jordan, M.I.: A control-theoretic perspective on optimal high-order optimization. Math. Program. 195, 929–975 (2022)

    Article  MathSciNet  MATH  Google Scholar 

  38. Muehlebach, M., Jordan, M. I.: A Dynamical systems perspective on nesterov acceleration. in Proceedings of the International Conference on Machine Learning, (2019)

  39. Nesterov, Y.: A method for solving the convex programming problem with convergence rate \(O(1/k^2)\). in Dokl. Akad. Nauk SSSR, (Russian) 269(3), 543–547 (1983)

  40. Polyak, B.T.: Some methods of speeding up the convergence of iteration methods. USSR Comput. Math. Math. Phys. 4, 1–17 (1964)

    Article  Google Scholar 

  41. Peypouquet, J., Sorin, S.: Evolution equations for maximal monotone operators asymptotic analysis in continuous and discrete time. J. Convex Anal. 17(3–4), 1113–1163 (2010)

    MathSciNet  MATH  Google Scholar 

  42. Shi, B., Du, S.S., Jordan, M.I., Su, W.J.: Understanding the acceleration phenomenon via high-resolution differential equations. Math. Program. 195, 79–148 (2022)

    Article  MathSciNet  MATH  Google Scholar 

  43. Su, W., Boyd, S., Candès, E.J.: A differential equation for modeling Nesterov’s accelerated gradient method. J. Mach. Learn. Res. 17(153), 43 (2016)

    MathSciNet  MATH  Google Scholar 

  44. Villa, S., Salzo, S., Baldassarres, L., Verri, A.: Accelerated and inexact forward-backward. SIAM J. Optim. 23(3), 1607–1633 (2013)

    Article  MathSciNet  MATH  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Laboratoire XLIM, Université de Limoges, 123 Avenue A. Thomas, 87060, Limoges, France

    Samir Adly & Van Nam Vo

  2. IMAG, CNRS, Université Montpellier, Montpellier, France

    Hedy Attouch

Authors
  1. Samir Adly
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Hedy Attouch
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Van Nam Vo
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Samir Adly.

Additional information

Communicated by Radu Ioan Bot.

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Adly, S., Attouch, H. & Vo, V.N. Convergence of Inertial Dynamics Driven by Sums of Potential and Nonpotential Operators with Implicit Newton-Like Damping. J Optim Theory Appl 198, 290–331 (2023). https://doi.org/10.1007/s10957-023-02228-x

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10957-023-02228-x

Keywords

Mathematics Subject Classification

Search

Navigation