Abstract
We show that the iterates generated by a generic first-order meta-algorithm satisfy a canonical perturbed Fenchel duality inequality. The latter in turn readily yields a unified derivation of the best known convergence rates for various popular first-order algorithms including the conditional gradient method as well as the main kinds of Bregman proximal methods: subgradient, gradient, fast gradient, and universal gradient methods.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Abernethy, J., Lai,K., Levy,K. , Wang,J.: Faster rates for convex-concave games. arXiv preprint arXiv:1805.06792, 2018
Abernethy, J.,Wang,J.: On Frank-Wolfe and equilibrium computation. In Advances in Neural Information Processing Systems, pages 6584–6593, 2017
Bach, F.: Duality between subgradient and conditional gradient methods. SIAM J. Optim. 25(1), 115–129 (2015)
Bauschke, H., Bolte, J., Teboulle, M.: A descent lemma beyond Lipschitz gradient continuity: first-order methods revisited and applications. Math. Oper. Res. 42(2), 330–348 (2016)
Bauschke, H., Combettes,P.: Convex Analysis and Monotone Operator Theory in Hilbert Spaces, volume 408. Springer, 2011
Beck,A.: First-Order Methods in Optimization, volume 25. SIAM, 2017
Beck, A., Teboulle, M.: Mirror descent and nonlinear projected subgradient methods for convex optimization. Oper. Res. Lett. 31(3), 167–175 (2003)
Beck, A., Teboulle, M.: A fast iterative shrinkage-thresholding algorithm for linear inverse problems. SIAM J. Imag. Sci. 2(1), 183–202 (2009)
Bello-Cruz, J.: On proximal subgradient splitting method for minimizing the sum of two nonsmooth convex functions. Set-Valued and Variational Analysis 25(2), 245–263 (2017)
Borwein,J., Lewis,A.: Convex Analysis and Nonlinear Optimization. Springer, New York, 2000
Bubeck,S., Lee,Y. , Singh,M.: A geometric alternative to Nesterov’s accelerated gradient descent. arXiv preprint arXiv:1506.08187, 2015
Chen, G., Teboulle, M.: Convergence analysis of a proximal-like minimization algorithm using Bregman functions. SIAM J. Optim. 3(3), 538–543 (1993)
Diakonikolas, J., Orecchia, L.: The approximate duality gap technique: a unified theory of first-order methods. SIAM J. Optim. 29(1), 660–689 (2019)
Dragomir,R., Taylor,A., d’Aspremont,A., Bolte,J.: Optimal complexity and certification of Bregman first-order methods. Mathematical Programming, pages 1–43, 2021
Drusvyatskiy, D., Fazel, M., Roy, S.: An optimal first order method based on optimal quadratic averaging. SIAM J. Optim. 28(1), 251–271 (2018)
Freund, R., Grigas, P.: New analysis and results for the Frank-Wolfe method. Math. Program. 155(1–2), 199–230 (2016)
Gutman, D., Peña, J.: A unified framework for Bregman proximal methods: subgradient, gradient, and accelerated gradient schemes. arXiv preprint arXiv:1812.10198, 2018
Gutman, D., Peña, J.: Convergence rates of proximal gradient methods via the convex conjugate. SIAM J. Opt., 29(1):162–174, 2019
Hanzely, F., Richtarik, P., Xiao, L.: Accelerated Bregman proximal gradient methods for relatively smooth convex optimization. Comput. Optim. Appl. 79(2), 405–440 (2021)
Hiriart-Urruty,J., Lemaréchal,C.: Convex Analysis and Minimization Algorithms. Springer-Verlag, Berlin, 1993
Jaggi,M.: Revisiting Frank-Wolfe: Projection-free sparse convex optimization. In ICML, volume 28 of JMLR Proceedings, pages 427–435, 2013
Lessard, L., Recht, B., Packard, A.: Analysis and design of optimization algorithms via integral quadratic constraints. SIAM J. Optim. 26(1), 57–95 (2016)
Lu,H.:“Relative continuity”for non-Lipschitz nonsmooth convex optimization using stochastic (or deterministic) mirror descent. INFORMS J. Opt., 1(4):288–303, 2019
Lu, H., Freund, R., Nesterov, Y.: Relatively smooth convex optimization by first-order methods, and applications. SIAM J. Optim. 28(1), 333–354 (2018)
Nemirovsky,A., Yudin,D.: Problem Complexity and Method Efficiency in Optimization. Wiley, 1983
Nesterov,Y.: A method for solving the convex programming problem with convergence rate \(\cal{O}(1/k^2)\). Doklady AN SSSR (in Russian). (English translation. Soviet Math. Dokl.), 269:543–547, 1983
Nesterov, Y.: Smooth minimization of non-smooth functions. Math. Program. 103(1), 127–152 (2005)
Nesterov, Y.: Gradient methods for minimizing composite functions. Math. Program. 140(1), 125–161 (2013)
Nesterov, Y.: Universal gradient methods for convex optimization problems. Math. Program. 152(1–2), 381–404 (2015)
Nesterov, Y.: Complexity bounds for primal-dual methods minimizing the model of objective function. Math. Program. 171(1), 311–330 (2018)
Peña. J.: Convergence of first-order methods via the convex conjugate. Oper. Res. Lett. 45, 561–564 (2017)
Rockafellar, T.: Convex Analysis. Princeton University Press, Princeton, 1970
Su,W., Boyd,S., Candès,E.: A differential equation for modeling Nesterov’s accelerated gradient method: Theory and insights. In Advances in Neural Information Processing Systems, pages 2510–2518, 2014
Teboulle,M.: A simplified view of first order methods for optimization. Math. Program., p 1–30, 2018
Van Nguyen, Q.: Variable quasi-Bregman monotone sequences. Numeric. Alg. 73(4), 1107–1130 (2016)
Van Nguyen, Q.: Forward-backward splitting with Bregman distances. Vietnam J. Math. 45(3), 519–539 (2017)
Wang,J., Abernethy,J.: Acceleration through optimistic no-regret dynamics. In Advances in Neural Information Processing Systems, p 3824–3834, 2018
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Gutman, D.H., Peña, J.F. Perturbed Fenchel duality and first-order methods. Math. Program. 198, 443–469 (2023). https://doi.org/10.1007/s10107-022-01779-7
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10107-022-01779-7