Advertisement

Foundations of Computational Mathematics

, Volume 17, Issue 2, pp 527–566 | Cite as

Random Gradient-Free Minimization of Convex Functions

  • Yurii NesterovEmail author
  • Vladimir Spokoiny
Article

Abstract

In this paper, we prove new complexity bounds for methods of convex optimization based only on computation of the function value. The search directions of our schemes are normally distributed random Gaussian vectors. It appears that such methods usually need at most n times more iterations than the standard gradient methods, where n is the dimension of the space of variables. This conclusion is true for both nonsmooth and smooth problems. For the latter class, we present also an accelerated scheme with the expected rate of convergence \(O\Big ({n^2 \over k^2}\Big )\), where k is the iteration counter. For stochastic optimization, we propose a zero-order scheme and justify its expected rate of convergence \(O\Big ({n \over k^{1/2}}\Big )\). We give also some bounds for the rate of convergence of the random gradient-free methods to stationary points of nonconvex functions, for both smooth and nonsmooth cases. Our theoretical results are supported by preliminary computational experiments.

Keywords

Convex optimization Stochastic optimization Derivative-free methods Random methods Complexity bounds 

Mathematics Subject Classification

90C25 0C47 68Q25 

Notes

Acknowledgments

The authors would like to thank two anonymous referees for enormously careful and helpful comments. Pavel Dvurechensky proposed a better proof of inequality (37), which we use in this paper. Research activity of the first author for this paper was partially supported by the grant “Action de recherche concertè ARC 04/09-315” from the “Direction de la recherche scientifique - Communautè française de Belgique,” and RFBR research projects 13-01-12007 ofi_m. The second author was supported by Laboratory of Structural Methods of Data Analysis in Predictive Modeling, MIPT, through RF government grant, ag.11.G34.31.0073.

References

  1. 1.
    A. Agarwal, O. Dekel, and L. Xiao, Optimal algorithms for online convex optimization with multi-point bandit feedback, in Proceedings of the 23rd Annual Conference on Learning, 2010, pp. 2840.Google Scholar
  2. 2.
    A. Agarwal, D. Foster, D. Hsu, S. Kakade, and A. Rakhlin, Stochastic convex optimization with bandit feedback,. SIAM J. on Optimization, 23 (2013), pp. 213-240.MathSciNetCrossRefzbMATHGoogle Scholar
  3. 3.
    D. Bertsimas and S. Vempala, Solving convex programs by random walks, J. of the ACM, 51 (2004), pp. 540-556.MathSciNetCrossRefzbMATHGoogle Scholar
  4. 4.
    F. Clarke, Optimization and nonsmooth analysis, Wliley, New York, 1983.zbMATHGoogle Scholar
  5. 5.
    A. Conn, K. Scheinberg, and L. Vicente , Introduction to derivative-free optimization. MPS-SIAM series on optimization, SIAM, Philadelphia, 2009.CrossRefzbMATHGoogle Scholar
  6. 6.
    C. Dorea, Expected number of steps of a random optimization method, JOTA, 39 (1983), pp. 165-171.MathSciNetCrossRefzbMATHGoogle Scholar
  7. 7.
    J. Duchi, M.I. Jordan, M.J. Wainwright, and A. Wibisono, Finite sample convergence rate of zero-order stochastic optimization methods, in NIPS, 2012, pp. 1448-1456.Google Scholar
  8. 8.
    A. D. Flaxman, A.T. Kalai, and B.H. Mcmahan, Online convex optimization in the bandit setting: gradient descent without a gradient, in Proceedings of the 16th annual ACM-SIAM symposium on Discrete Algorithms, 2005, pp. 385-394 .Google Scholar
  9. 9.
    R. Kleinberg, A. Slivkins, and E. Upfal, Multi-armed bandits in metric spaces, in Proceedings of the 40th annual ACM symposium on Theory of Computing, 2008, pp. 681-690.Google Scholar
  10. 10.
    J. C. Lagarias, J. A. Reeds, M. H. Wright, and P. E. Wright, Convergence properties of the Nelder-Mead Simplex Algorithm in low dimensions, SIAM J. Optimization, 9 (1998), pp. 112-147.CrossRefzbMATHGoogle Scholar
  11. 11.
    J. C. Lagarias, B. Poonen, and M. H. Wright, Convergence of the restricted Nelder-Mead algorithm in two dimensions, SIAM J. Optimization, 22 (2012), pp. 501-532.MathSciNetCrossRefzbMATHGoogle Scholar
  12. 12.
    J. Matyas, Random optimization. Automation and Remote Control, 26 (1965), pp. 246-253.MathSciNetzbMATHGoogle Scholar
  13. 13.
    J. A. Nelder and R. Mead, A simplex method for function minimization, Computer Journal, 7 (1965), pp. 308–3013MathSciNetCrossRefzbMATHGoogle Scholar
  14. 14.
    A. Nemirovski, A. Juditsky, G. Lan, and A.Shapiro, Robust Stochastic Approximation approach to Stochastic Programming, SIAM J. on Optimization, 19 (2009), pp. 1574-1609.MathSciNetCrossRefzbMATHGoogle Scholar
  15. 15.
    A. Nemirovsky and D.Yudin, Problem complexity and method efficiency in optimization, John Wiley and Sons, New York, 1983.Google Scholar
  16. 16.
    Yu. Nesterov, Introductory Lectures on Convex Optimization, Kluwer, Boston, 2004.CrossRefzbMATHGoogle Scholar
  17. 17.
    Yu. Nesterov, Lexicographic differentiation of nonsmooth functions’, Mathematical Programming, 104 (2005), pp. 669-700.MathSciNetCrossRefzbMATHGoogle Scholar
  18. 18.
    Yu. Nesterov, Random gradient-free minimization of convex functions, CORE Discussion Paper # 2011/1, (2011).Google Scholar
  19. 19.
    Yu. Nesterov, Efficiency of coordinate descent methods on huge-scale optimization problems. SIAM J. on Optimization, 22 (2012), pp. 341-362.MathSciNetCrossRefzbMATHGoogle Scholar
  20. 20.
    B. Polyak, Introduction to Optimization. Optimization Software - Inc., Publications Division, New York, 1987.zbMATHGoogle Scholar
  21. 21.
    V. Protasov, Algorithms for approximate calculation of the minimum of a convex function from its values, Mathematical Notes, 59 (1996), pp. 69-74.MathSciNetCrossRefzbMATHGoogle Scholar
  22. 22.
    M. Sarma, On the convergence of the Baba and Dorea random optimization methods, JOTA, 66 (1990), pp. 337-343.MathSciNetCrossRefzbMATHGoogle Scholar

Copyright information

© SFoCM 2015

Authors and Affiliations

  1. 1.Center for Operations Research and Econometrics (CORE)Catholic University of Louvain (UCL)LeuvenBelgium
  2. 2.Weierstrass Institute for Applied Analysis and Stochastics (WIAS)Humboldt University of BerlinBerlinGermany

Personalised recommendations