Abstract
The article discusses the class of gradient-concordant numerical methods for smooth unconstrained minimization where the descent direction is restricted to a subset of the descent cone. This class covers a wide range of well-known optimization methods, such as gradient descent, conjugate gradient method, and Newton’s method. While previous research has demonstrated the linear convergence rate of gradient-concordant methods for strongly convex functions, many practical functions do not meet this criterion. Our research explores the convergence of gradient-concordant methods for a broader class of functions. We prove that the Polyak-Łojasiewicz condition is sufficient for the linear convergence of the gradient-concordant method. Additionally, we show sublinear convergence for convex functions that are not necessarily strongly convex.
Supported by Russian Science Foundation (project No. 21-71-30005) https://rscf.ru/en/project/21-71-30005/.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Ortega, J.M., Rheinbolt, W.G.: Iterative Solution of Nonlinear Equations in Several Variables. Academic Press, New York (1970)
Birjukov, A.G., Chernov, A.V.: Gradient-concordant methods for solving unconstrained minimization problems. In: Proceedings of MIPT, pp. 113–125 (2017). (in Russian)
Polyak, B.T.: Gradient methods for minimizing functionals. Zh. Vychisl. Mat. Mat. Fiz. 643–653 (1963). (in Russian)
Łojasiewicz., S.: A topological property of real analytic subsets. Coll. du CNRS, Les ’equations aux d’eriv’ees partielles, 87–89 (1963). (in French)
Ajalloeian, A., Stich, S.: Analysis of SGD with biased gradient estimators. In: Workshop on “Beyond First Order Methods in ML Systems” at the 37th ICML, Vienna, Austria (2020)
Kim, S., Madden, L., Dall’Anese, E.: Online stochastic gradient methods under sub-weibull noise and the Polyak-Łojasiewicz condition. In: 2022 IEEE 61st Conference on Decision and Control (CDC), Cancun, Mexico, pp. 3499–3506 (2022)
Nesterov, Y.: Efficiency of coordinate descent methods on huge-scale optimization problems. SIAM J. Optim. 341–362 (2012)
Karimi, H., Nutini, J., Schmidt, M.: Linear convergence of gradient and proximal-gradient methods under the Polyak-Łojasiewicz condition. In: Frasconi, P., Landwehr, N., Manco, G., Vreeken, J. (eds.) ECML PKDD 2016. LNCS (LNAI), vol. 9851, pp. 795–811. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46128-1_50
Apidopoulos, V., Ginatta, N., Villa, S.: Convergence rates for the heavy-ball continuous dynamics for non-convex optimization, under Polyak-Łojasiewicz condition. J. Glob. Optim. 84, 563–589 (2022)
Wang, J.K., et al.: Provable acceleration of heavy ball beyond quadratics for a class of Polyak-Lojasiewicz functions when the non-convexity is averaged-out. In: ICML, pp. 22839–22864. PMLR (2022)
Garber, D., Hazan, E.: Faster rates for the Frank-Wolfe method over strongly-convex sets. In: ICML, pp. 541–549 (2015)
Sukharev, A.G., Timokhov, A.V., Fedorov, V.V.: Course in Optimization Methods. Physmatlit, Moscow (2005)
Polyak, B.T.: Introduction to Optimization, 2nd edn. Lenand, Moscow (2014). (in Russian)
Acknowledgements
We would like to thank the late Dr. Alexander Birjukov for his considerable help that made this work possible.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Chernov, A., Lisachenko, A. (2023). Convergence Rate of Gradient-Concordant Methods for Smooth Unconstrained Optimization. In: Olenev, N., Evtushenko, Y., Jaćimović, M., Khachay, M., Malkova, V. (eds) Optimization and Applications. OPTIMA 2023. Lecture Notes in Computer Science, vol 14395. Springer, Cham. https://doi.org/10.1007/978-3-031-47859-8_3
Download citation
DOI: https://doi.org/10.1007/978-3-031-47859-8_3
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-47858-1
Online ISBN: 978-3-031-47859-8
eBook Packages: Computer ScienceComputer Science (R0)