Abstract
A fast gradient method requiring only one projection is proposed for smooth convex optimization problems. The method has a visual geometric interpretation, so it is called the method of similar triangles (MST). Composite, adaptive, and universal versions of MST are suggested. Based on MST, a universal method is proposed for the first time for strongly convex problems (this method is continuous with respect to the strong convexity parameter of the smooth part of the functional). It is shown how the universal version of MST can be applied to stochastic optimization problems.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Yu. Nesterov, “Smooth minimization of nonsmooth function,” Math. Prog. Ser. A 103 (1), 127–152 (2005).
Yu. Nesterov, “Gradient methods for minimizing composite functions,” Math. Prog. 140 (1), 125–161 (2013).
O. Devolder, PhD Thesis (CORE UCL, 2013)
O. Devolder, F. Glineur, and Yu. Nesterov, “First order methods of smooth convex optimization with inexact oracle,” Math. Prog. Ser. A 146 (1–2), 37–75 (2014).
O. Devolder, F. Glineur, and Yu. Nesterov, “Intermediate gradient methods for smooth convex problems with inexact oracle,” CORE Discussion Paper 2013/17 (2013).
Yu. Nesterov, “Universal gradient methods for convex optimization problems,” Math. Prog. 152 (1–2), 381–404 (2015).
O. Devolder, F. Glineur, and Yu. Nesterov “First order methods with inexact oracle: The smooth strongly convex case,” CORE Discussion Paper 2013/16 (2013).
A. V. Gasnikov and P. E. Dvurechensky, “Stochastic intermediate gradient method for convex optimization problems,” Dokl. Math. 93 (2), 148–151 (2016). arXiv:1411.2876.
A. V. Gasnikov, P. E. Dvurechensky, and Yu. E. Nesterov, “Stochastic gradient methods with an inexact oracle,” Tr. Mosk. Fiz.-Tekh. Inst. 8 (1), 41–91 (2016). arXiv:1411.4218.
A. V. Gasnikov, D. I. Kamzolov, and M. A. Mendel’, “Basic constructions over convex optimization algorithms and applications to the derivation of new estimates for strongly convex problems,” Tr. Mosk. Fiz.-Tekh. Inst. 8 (3), 25–42 (2016). arXiv:1603.07701.
A. Nemirovski, Lectures on Modern Convex Optimization Analysis, Algorithms, and Engineering Applications (SIAM, Philadelphia, 2013).
A. S. Nemirovski and D. B. Yudin, Complexity of Problems and Efficiency of Optimization Methods (Nauka, Moscow, 1979) [in Russian].
Yu. Nesterov, “Primal-dual subgradient methods for convex problems,” Math. Prog. Ser. B 120 (1), 261–283 (2009).
A. S. Anikin, A. V. Gasnikov, P. E. Dvurechensky, A. I. Tyurin, and A. V. Chernov, “Dual approaches to the minimization of strongly convex functionals with a simple structure under affine constraints,” Comput. Math. Math. Phys. 57 (8), 1262–1276 (2017).
A. Juditsky and Yu. Nesterov, “Deterministic and stochastic primal-dual subgradient algorithms for uniformly convex minimization,” Stoch. Syst. 4 (1), 44–80 (2014).
A. V. Gasnikov, E. V. Gasnikova, Yu. E. Nesterov, and A. V. Chernov, “Efficient numerical methods for entropy-linear programming problems,” Comput. Math. Math. Phys. 56 (4), 514–524 (2016). arXiv:1410.7719.
A. S. Anikin, A. V. Gasnikov, and A. Yu. Gornov, “Efficient nonaccelerated methods for solving sparse quadratic optimization problems,” Tr. Mosk. Fiz.-Tekh. Inst. 8 (2), 44–59 (2016). arXiv:1602.01124.
Yu. E. Nesterov, Introduction to Convex Optimization (MTsNMO, Moscow, 2010) [in Russian].
B. O’Donoghue and E. Candes, “Adaptive restart for accelerated gradient schemes,” Foundations Comput. Math. 15, 715–732 (2015).
C. Guzman and A. Nemirovski, “On lower complexity bounds for large-scale smooth convex optimization,” J. Complexity 31, 1–14 (2015). arXiv:1307.5001.
J. Nocedal and S. Wright, Numerical Optimization (Springer, Berlin, 2006).
A. Tyurin, “Mirror version of similar triangles method for constrained optimization problems,” e-print (2017). arXiv:1705.09809.
A. V. Gasnikov, P. E. Dvurechensky, D. I. Kamzolov, Yu. E. Nesterov, V. G. Spokoinyi, P. I. Stetsyuk, A. L. Suvorikova, and A. V. Chernov, “Search for equilibria in multistage traffic flow models,” Tr. Mosk. Fiz.- Tekh. Inst. 7 (4), 143–155 (2015).
A. V. Gasnikov, P. E. Dvurechensky, V. G. Spokoinyi, P. I. Stetsyuk, and A. L. Suvorikova, “Superposition of the balancing and universal gradient methods for finding an entropy-smoothed Wasserstein barycenter and equilibria in multistage traffic flow models,” Tr. Mosk. Fiz.-Tekh. Inst. 8 (3), 5–24 (2016). arXiv:1506.00292.
V. Guiges, A. Juditsky, and A. Nemirovski“Nonasymptotic confidence bounds for the optimal value of a stochastic program,” e-print (2016). arXiv:1601.07592.
Author information
Authors and Affiliations
Corresponding author
Additional information
Original Russian Text © A.V. Gasnikov, Yu.E. Nesterov, 2018, published in Zhurnal Vychislitel’noi Matematiki i Matematicheskoi Fiziki, 2018, Vol. 58, No. 1, pp. 52–69.
Rights and permissions
About this article
Cite this article
Gasnikov, A.V., Nesterov, Y.E. Universal Method for Stochastic Composite Optimization Problems. Comput. Math. and Math. Phys. 58, 48–64 (2018). https://doi.org/10.1134/S0965542518010050
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1134/S0965542518010050