Oracle complexity of second-order methods for smooth convex optimization
- 121 Downloads
Second-order methods, which utilize gradients as well as Hessians to optimize a given function, are of major importance in mathematical optimization. In this work, we prove tight bounds on the oracle complexity of such methods for smooth convex functions, or equivalently, the worst-case number of iterations required to optimize such functions to a given accuracy. In particular, these bounds indicate when such methods can or cannot improve on gradient-based methods, whose oracle complexity is much better understood. We also provide generalizations of our results to higher-order methods.
KeywordsSmooth convex optimization Oracle complexity
Mathematics Subject Classification90C25 65K05 49M37
We thank Yurii Nesterov for several helpful comments on a preliminary version of this paper, as well as Naman Agarwal, Elad Hazan and Zeyuan Allen-Zhu for informing us about the A-NPE algorithm of .
- 1.Agarwal, N., Hazan, E.: Lower bounds for higher-order optimization. Working draft (2017)Google Scholar
- 2.Allen-Zhu, Z., Hazan, E.: Optimal black-box reductions between optimization objectives. In: Advances in Neural Information Processing Systems, pp. 1614–1622 (2016)Google Scholar
- 3.Arjevani, Y., Shamir, O.: On the iteration complexity of oblivious first-order optimization algorithms. In: International Conference on Machine Learning, pp. 908–916 (2016)Google Scholar
- 4.Arjevani, Y., Shamir, O.: Oracle complexity of second-order methods for finite-sum problems. arXiv preprint arXiv:1611.04982 (2016)
- 5.Baes, M.: Estimate Sequence Methods: Extensions and Approximations. Institute for Operations Research, ETH, Zürich (2009)Google Scholar
- 12.Nemirovski, A.: Efficient methods in convex programming—lecture notes (2005)Google Scholar
- 13.Nemirovsky, A., Yudin, D.: Problem Complexity and Method Efficiency in Optimization. Wiley, New York (1983)Google Scholar
- 20.Woodworth, B., Srebro, N.: Lower bound for randomized first order convex optimization. arXiv preprint arXiv:1709.03594 (2017)