Skip to main content
Log in

The complexity of first-order optimization methods from a metric perspective

  • Full Length Paper
  • Series A
  • Published:
Mathematical Programming Submit manuscript

Abstract

A central tool for understanding first-order optimization algorithms is the Kurdyka–Łojasiewicz inequality. Standard approaches to such methods rely crucially on this inequality to leverage sufficient decrease conditions involving gradients or subgradients. However, the KL property fundamentally concerns not subgradients but rather “slope”, a purely metric notion. By highlighting this view, and avoiding any use of subgradients, we present a simple and concise complexity analysis for first-order optimization algorithms on metric spaces. This subgradient-free perspective also frames a short and focused proof of the KL property for nonsmooth semi-algebraic functions.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Subscribe and save

Springer+ Basic
EUR 32.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or Ebook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Similar content being viewed by others

References

  1. Absil, P.-A., Mahony, R., Andrews, B.: Convergence of the iterates of descent methods for analytic cost functions. SIAM J. Optim. 6(2), 531–547 (2005)

    Article  MathSciNet  Google Scholar 

  2. Absil, P.-A., Mahony, R., Sepulchre, R.: Optimization Algorithms on Matrix Manifolds. Princeton University Press, Princeton (2008)

    Book  Google Scholar 

  3. Ambrosio, L., Gigli, N., Savaré, G.: Gradient Flows in Metric Spaces and in the Space of Probability Measures, 2nd edn. Birkhäuser, Basel (2008)

    Google Scholar 

  4. Attouch, H., Bolte, J.: On the convergence of the proximal algorithm for non-smooth functions involving analytic features. Math. Program. 116, 5–16 (2009)

    Article  MathSciNet  Google Scholar 

  5. Attouch, H., Bolte, J., Redont, P., Soubeyran, A.: Proximal alternating minimization and projection methods for nonconvex problems: an approach based on the Kurdyka–Lojasiewicz inequality. Math. Oper. Res. 35, 438–457 (2010)

    Article  MathSciNet  Google Scholar 

  6. Attouch, H., Bolte, J., Svaiter, B.F.: Convergence of descent methods for semi-algebraic and tame problems: proximal algorithms, forward–backward splitting, and regularized Gauss–Seidel methods. Math. Program. 137(1–2), 91–129 (2013)

    Article  MathSciNet  Google Scholar 

  7. Auslender, A., Shefi, R., Teboulle, M.: A moving balls approximation method for a class of smooth constrained minimization problems. SIAM J. Optim. 20(6), 3232–3259 (2010)

    Article  MathSciNet  Google Scholar 

  8. Azé, D., Corvellec, J.-N.: Characterizations of error bounds for lower semi-continuous functions on metric spaces. ESAIM COCV 10, 409–425 (2004)

    Article  Google Scholar 

  9. Azé, D., Corvellec, J.-N.: Nonlinear error bounds via a change of function. J. Optim. Theory Appl. 172, 9–32 (2017)

    Article  MathSciNet  Google Scholar 

  10. Barakat, A., Bianchi, P.: Convergence rates of a momentum algorithm with bounded adaptive step size for nonconvex optimization. Proc. Mach. Learn. Res. 129, 225–240 (2020)

    Google Scholar 

  11. Bac̆ák, M.: The proximal point algorithm in metric spaces. Isr. J. Math. 194, 689–701 (2013)

  12. Bac̆ák, M.: Convex Analysis and Optimization in Hadamard Spaces. De Gruyter, Berlin (2014)

  13. Bento, G.C., Ferreira, O.P., Melo, J.G.: Iteration-complexity of gradient, sub-gradient and proximal point methods on Riemannian manifolds. J. Optim. Theory Appl. 173, 548–562 (2017)

    Article  MathSciNet  Google Scholar 

  14. Blanchet, A., Bolte, J.: A family of functional inequalities: Lojasiewicz inequalities and displacement convex functions. J. Funct. Anal. 275, 1650–1673 (2018)

    Article  MathSciNet  Google Scholar 

  15. Bolte, J., Daniilidis, A., Lewis, A.S.: The Łojasiewicz inequality for nonsmooth subanalytic functions with applications to subgradient dynamical systems. SIAM J. Optim. 17, 1205–1223 (2006)

    Article  MathSciNet  Google Scholar 

  16. Bolte, J., Daniilidis, A., Lewis, A.S., Shiota, M.: Clarke subgradients of stratifiable functions. SIAM J. Optim. 18(2), 556–572 (2007)

    Article  MathSciNet  Google Scholar 

  17. Bolte, J., Daniilidis, A., Ley, O., Mazet, L.: Characterizations of Łojasiewicz inequalities: subgradient flows, talweg, convexity. TAMS 362, 3319–3363 (2010)

    Article  Google Scholar 

  18. Bolte, J., Nguyen, T.P., Peypouquet, J., Suter, B.: From error bounds to the complexity of first order descent methods for convex functions. Math. Program. 165, 471–507 (2017)

    Article  MathSciNet  Google Scholar 

  19. Bolte, J., Pauwels, E.: Majorization-minimization procedures and convergence of SQP methods for semi-algebraic and tame programs. Math. Oper. Res. 41, 442–465 (2016)

    Article  MathSciNet  Google Scholar 

  20. Boumal, N.: An Introduction to Optimization on Smooth Manifolds. Cambridge University Press, Cambridge (2023)

    Book  Google Scholar 

  21. Bridson, M.R., Haefliger, A.: Metric Spaces of Non-positive Curvature. Springer, Berlin (1999)

    Book  Google Scholar 

  22. Chen, S., Ma, S., So, A.M.C., Zhang, T.: Proximal gradient method for nonsmooth optimization over the Stiefel manifold. SIAM J. Optim. 30, 210–239 (2020)

    Article  MathSciNet  Google Scholar 

  23. Coste, M.: An Introduction to O-minimal Geometry. In: RAAG Notes, Institut de Recherche Mathématiques de Rennes, November 1999

  24. De Giorgi, E., Marino, A., Tosques, M.: Problems of evolution in metric spaces and maximal decreasing curve. Atti. Accad. Naz. Lincei Rend Cl Sci. Fis. Mat. Nat. 68, 180–187 (1980)

    MathSciNet  Google Scholar 

  25. Dempster, A.P., Laird, N.M., Rubin, D.B.: Maximum likelihood from incomplete data via the EM algorithm. J. R. Stat. Soc. Ser. B 39, 1–38 (1977)

    Article  MathSciNet  Google Scholar 

  26. Drusvyatskiy, D., Ioffe, A.D., Lewis, A.S.: Curves of descent. SIAM J. Control Optim. 53(1), 114–138 (2015)

    Article  MathSciNet  Google Scholar 

  27. Drusvyatskiy, D., Ioffe, A.D., Lewis, A.S.: Nonsmooth optimization using Taylor-like models: error bounds, convergence, and termination criteria. Math. Program. 185, 357–383 (2021)

    Article  MathSciNet  Google Scholar 

  28. Ekeland, I.: On the variational principle. J. Math. Anal. Appl. 47, 324–353 (1974)

    Article  MathSciNet  Google Scholar 

  29. Güler, O.: On the convergence of the proximal point algorithm for convex minimization. SIAM J. Control Optim. 29, 403–419 (1991)

    Article  MathSciNet  Google Scholar 

  30. Hauer, D., Mazón, J.M.: Kurdyka–Lojasiewicz-Simon inequality for gradient flows in metric spaces. TAMS 372, 4917–4976 (2019)

    Article  Google Scholar 

  31. Ioffe, A.D.: Variational Analysis of Regular Mappings. Springer, Berlin (2017)

    Book  Google Scholar 

  32. Jost, J.: Convex functionals and generalized harmonic maps into spaces of non positive curvature. Comment. Math. Helvetici 70, 659–673 (1995)

    Article  MathSciNet  Google Scholar 

  33. Kurdyka, K.: On gradients of functions definable in o-minimal structures. Ann. Inst. Fourier (Grenoble) 48(3), 769–783 (1998)

    Article  MathSciNet  Google Scholar 

  34. Lewis, A.S., López, G., Nicolae, A.: Basic convex analysis in metric spaces with bounded curvature. SIAM J. Optim. arxiv:2302.03588 (2023)

  35. Lewis, A.S., Tian, T.: Identifiability, the KL property in metric spaces, and subgradient curves. arXiv:2205.02868 (2022)

  36. Lewis, A.S., Wright, S.J.: A proximal method for composite minimization. Math. Program. 1–46 (2015)

  37. Łojasiewicz, S.: Ensembles Semi-analytiques. IHES (1965)

  38. Luo, Z.-Q., Tseng, P.: Error bounds and convergence analysis of feasible descent methods: a general approach. Ann. Oper. Res. 46/47(1-4):157–178 (1993). Degeneracy in optimization problems

  39. Mayer, U.F.: Gradient flows on nonpositively curved metric spaces and harmonic maps. Commun. Anal. Geom. 6, 199–253 (1998)

    Article  MathSciNet  Google Scholar 

  40. Minty, G.J.: Monotone (nonlinear) operators in Hilbert space. Duke Math. J. 29, 341–346 (1962)

    Article  MathSciNet  Google Scholar 

  41. Myers, S.B.: Arcs and geodesics in metric spaces. Trans. Am. Math. Soc. 57, 217–227 (1945)

    Article  MathSciNet  Google Scholar 

  42. Nedić, A.: Random projection algorithms for convex set intersection problems. In: 49th IEEE Conference on Decision and Control (CDC), pp. 7655–7660 (2010)

  43. Noll, D.: Convergence of non-smooth descent methods using the Kurdyka–Lojasiewicz inequality. J. Optim. Theory Appl. 160, 553–572 (2014)

    Article  MathSciNet  Google Scholar 

  44. Peyré, G., Cuturi, M.: Computational optimal transport: with applications to data science. Found. Trends Mach. Learn. 11, 355–607 (2019)

    Article  Google Scholar 

  45. Rockafellar, R.T.: Lipschitzian properties of multi-functions. Nonlinear Anal. 9, 867–885 (1985)

    Article  MathSciNet  Google Scholar 

  46. Rockafellar, R.T., Wets, R.J-B.: Variational analysis. In: Grundlehren der mathematischen Wissenschaften, vol. 317. Springer, Berlin (1998)

  47. Valette, A.: Łojasiewicz inequality at singular points. Proc. Am. Math. Soc. 147, 1109–1117 (2019)

    Article  Google Scholar 

  48. van den Dries, L.: Tame topology and o-minimal structures. In: LMS Lecture Note Series, vol. 248. Cambridge University Press, Cambridge (1998)

  49. van den Dries, L., Miller, C.: Geometric categories and o-minimal structures. Duke Math. J. 84, 497–540 (1996)

    MathSciNet  Google Scholar 

  50. Zhang, H., Sra, S.: First-order methods for geo-desically convex optimization. J. Mach. Learn. Res. 49, 1–22 (2016)

    Google Scholar 

Download references

Funding

Research of A.S. Lewis supported in part by National Science Foundation Grant DMS-2006990.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to A. S. Lewis.

Ethics declarations

Conflict of interest

The authors declare that the results presented are new, and there is no Conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Research supported in part by National Science Foundation Grant DMS-2006990.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Lewis, A.S., Tian, T. The complexity of first-order optimization methods from a metric perspective. Math. Program. (2024). https://doi.org/10.1007/s10107-024-02091-2

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s10107-024-02091-2

Keywords

Mathematics Subject Classification

Navigation