Advertisement

Oracle complexity of second-order methods for smooth convex optimization

  • Yossi Arjevani
  • Ohad Shamir
  • Ron ShiffEmail author
Full Length Paper Series A
  • 121 Downloads

Abstract

Second-order methods, which utilize gradients as well as Hessians to optimize a given function, are of major importance in mathematical optimization. In this work, we prove tight bounds on the oracle complexity of such methods for smooth convex functions, or equivalently, the worst-case number of iterations required to optimize such functions to a given accuracy. In particular, these bounds indicate when such methods can or cannot improve on gradient-based methods, whose oracle complexity is much better understood. We also provide generalizations of our results to higher-order methods.

Keywords

Smooth convex optimization Oracle complexity 

Mathematics Subject Classification

90C25 65K05 49M37 

Notes

Acknowledgements

We thank Yurii Nesterov for several helpful comments on a preliminary version of this paper, as well as Naman Agarwal, Elad Hazan and Zeyuan Allen-Zhu for informing us about the A-NPE algorithm of [10].

References

  1. 1.
    Agarwal, N., Hazan, E.: Lower bounds for higher-order optimization. Working draft (2017)Google Scholar
  2. 2.
    Allen-Zhu, Z., Hazan, E.: Optimal black-box reductions between optimization objectives. In: Advances in Neural Information Processing Systems, pp. 1614–1622 (2016)Google Scholar
  3. 3.
    Arjevani, Y., Shamir, O.: On the iteration complexity of oblivious first-order optimization algorithms. In: International Conference on Machine Learning, pp. 908–916 (2016)Google Scholar
  4. 4.
    Arjevani, Y., Shamir, O.: Oracle complexity of second-order methods for finite-sum problems. arXiv preprint arXiv:1611.04982 (2016)
  5. 5.
    Baes, M.: Estimate Sequence Methods: Extensions and Approximations. Institute for Operations Research, ETH, Zürich (2009)Google Scholar
  6. 6.
    Boyd, S., Vandenberghe, L.: Convex Optimization. Cambridge University Press, Cambridge (2004)CrossRefzbMATHGoogle Scholar
  7. 7.
    Cartis, C., Gould, N.I., Toint, P.L.: On the complexity of steepest descent, newton’s and regularized newton’s methods for nonconvex unconstrained optimization problems. SIAM J. Optim. 20(6), 2833–2852 (2010)MathSciNetCrossRefzbMATHGoogle Scholar
  8. 8.
    Cartis, C., Gould, N.I., Toint, P.L.: Evaluation complexity of adaptive cubic regularization methods for convex unconstrained optimization. Optim Methods Softw. 27(2), 197–219 (2012)MathSciNetCrossRefzbMATHGoogle Scholar
  9. 9.
    Kantorovich, L.V.: Functional analysis and applied mathematics. Uspekhi Matematicheskikh Nauk 3(6), 89–185 (1948)MathSciNetzbMATHGoogle Scholar
  10. 10.
    Monteiro, R.D., Svaiter, B.F.: An accelerated hybrid proximal extragradient method for convex optimization and its implications to second-order methods. SIAM J. Optim. 23(2), 1092–1125 (2013)MathSciNetCrossRefzbMATHGoogle Scholar
  11. 11.
    Mu, C., Hsu, D., Goldfarb, D.: Successive rank-one approximations for nearly orthogonally decomposable symmetric tensors. SIAM J. Matrix Anal. Appl. 36(4), 1638–1659 (2015)MathSciNetCrossRefzbMATHGoogle Scholar
  12. 12.
    Nemirovski, A.: Efficient methods in convex programming—lecture notes (2005)Google Scholar
  13. 13.
    Nemirovsky, A., Yudin, D.: Problem Complexity and Method Efficiency in Optimization. Wiley, New York (1983)Google Scholar
  14. 14.
    Nesterov, Y.: A method of solving a convex programming problem with convergence rate \(O(1/k^2)\). Sov. Math. Dokl. 27(2), 372–376 (1983)zbMATHGoogle Scholar
  15. 15.
    Nesterov, Y.: Introductory Lectures on Convex Optimization: A Basic Course, vol. 87. Springer, Berlin (2004)zbMATHGoogle Scholar
  16. 16.
    Nesterov, Y.: Accelerating the cubic regularization of newton method on convex problems. Math. Program. 112(1), 159–181 (2008)MathSciNetCrossRefzbMATHGoogle Scholar
  17. 17.
    Nesterov, Y., Nemirovskii, A.: Interior-Point Polynomial Algorithms in Convex Programming. SIAM, Philadelphia (1994)CrossRefzbMATHGoogle Scholar
  18. 18.
    Nesterov, Y., Polyak, B.T.: Cubic regularization of newton method and its global performance. Math. Program. 108(1), 177–205 (2006)MathSciNetCrossRefzbMATHGoogle Scholar
  19. 19.
    Vladimirov, A., Nesterov, Y.E., Chekanov, Y.N.: On uniformly convex functionals. Vestnik Moskov. Univ. Ser. XV Vychisl. Mat. Kibernet 3, 12–23 (1978)MathSciNetzbMATHGoogle Scholar
  20. 20.
    Woodworth, B., Srebro, N.: Lower bound for randomized first order convex optimization. arXiv preprint arXiv:1709.03594 (2017)

Copyright information

© Springer-Verlag GmbH Germany, part of Springer Nature and Mathematical Optimization Society 2018

Authors and Affiliations

  1. 1.Department of Computer ScienceWeizmann Institute of ScienceRehovotIsrael

Personalised recommendations