# Oracle complexity of second-order methods for smooth convex optimization

- 121 Downloads

## Abstract

Second-order methods, which utilize gradients as well as Hessians to optimize a given function, are of major importance in mathematical optimization. In this work, we prove tight bounds on the oracle complexity of such methods for smooth convex functions, or equivalently, the worst-case number of iterations required to optimize such functions to a given accuracy. In particular, these bounds indicate when such methods can or cannot improve on gradient-based methods, whose oracle complexity is much better understood. We also provide generalizations of our results to higher-order methods.

## Keywords

Smooth convex optimization Oracle complexity## Mathematics Subject Classification

90C25 65K05 49M37## Notes

### Acknowledgements

We thank Yurii Nesterov for several helpful comments on a preliminary version of this paper, as well as Naman Agarwal, Elad Hazan and Zeyuan Allen-Zhu for informing us about the A-NPE algorithm of [10].

## References

- 1.Agarwal, N., Hazan, E.: Lower bounds for higher-order optimization. Working draft (2017)Google Scholar
- 2.Allen-Zhu, Z., Hazan, E.: Optimal black-box reductions between optimization objectives. In: Advances in Neural Information Processing Systems, pp. 1614–1622 (2016)Google Scholar
- 3.Arjevani, Y., Shamir, O.: On the iteration complexity of oblivious first-order optimization algorithms. In: International Conference on Machine Learning, pp. 908–916 (2016)Google Scholar
- 4.Arjevani, Y., Shamir, O.: Oracle complexity of second-order methods for finite-sum problems. arXiv preprint arXiv:1611.04982 (2016)
- 5.Baes, M.: Estimate Sequence Methods: Extensions and Approximations. Institute for Operations Research, ETH, Zürich (2009)Google Scholar
- 6.Boyd, S., Vandenberghe, L.: Convex Optimization. Cambridge University Press, Cambridge (2004)CrossRefzbMATHGoogle Scholar
- 7.Cartis, C., Gould, N.I., Toint, P.L.: On the complexity of steepest descent, newton’s and regularized newton’s methods for nonconvex unconstrained optimization problems. SIAM J. Optim.
**20**(6), 2833–2852 (2010)MathSciNetCrossRefzbMATHGoogle Scholar - 8.Cartis, C., Gould, N.I., Toint, P.L.: Evaluation complexity of adaptive cubic regularization methods for convex unconstrained optimization. Optim Methods Softw.
**27**(2), 197–219 (2012)MathSciNetCrossRefzbMATHGoogle Scholar - 9.Kantorovich, L.V.: Functional analysis and applied mathematics. Uspekhi Matematicheskikh Nauk
**3**(6), 89–185 (1948)MathSciNetzbMATHGoogle Scholar - 10.Monteiro, R.D., Svaiter, B.F.: An accelerated hybrid proximal extragradient method for convex optimization and its implications to second-order methods. SIAM J. Optim.
**23**(2), 1092–1125 (2013)MathSciNetCrossRefzbMATHGoogle Scholar - 11.Mu, C., Hsu, D., Goldfarb, D.: Successive rank-one approximations for nearly orthogonally decomposable symmetric tensors. SIAM J. Matrix Anal. Appl.
**36**(4), 1638–1659 (2015)MathSciNetCrossRefzbMATHGoogle Scholar - 12.Nemirovski, A.: Efficient methods in convex programming—lecture notes (2005)Google Scholar
- 13.Nemirovsky, A., Yudin, D.: Problem Complexity and Method Efficiency in Optimization. Wiley, New York (1983)Google Scholar
- 14.Nesterov, Y.: A method of solving a convex programming problem with convergence rate \(O(1/k^2)\). Sov. Math. Dokl.
**27**(2), 372–376 (1983)zbMATHGoogle Scholar - 15.Nesterov, Y.: Introductory Lectures on Convex Optimization: A Basic Course, vol. 87. Springer, Berlin (2004)zbMATHGoogle Scholar
- 16.Nesterov, Y.: Accelerating the cubic regularization of newton method on convex problems. Math. Program.
**112**(1), 159–181 (2008)MathSciNetCrossRefzbMATHGoogle Scholar - 17.Nesterov, Y., Nemirovskii, A.: Interior-Point Polynomial Algorithms in Convex Programming. SIAM, Philadelphia (1994)CrossRefzbMATHGoogle Scholar
- 18.Nesterov, Y., Polyak, B.T.: Cubic regularization of newton method and its global performance. Math. Program.
**108**(1), 177–205 (2006)MathSciNetCrossRefzbMATHGoogle Scholar - 19.Vladimirov, A., Nesterov, Y.E., Chekanov, Y.N.: On uniformly convex functionals. Vestnik Moskov. Univ. Ser. XV Vychisl. Mat. Kibernet
**3**, 12–23 (1978)MathSciNetzbMATHGoogle Scholar - 20.Woodworth, B., Srebro, N.: Lower bound for randomized first order convex optimization. arXiv preprint arXiv:1709.03594 (2017)