On the optimal order of worst case complexity of direct search

Abstract

The worst case complexity of direct-search methods has been recently analyzed when they use positive spanning sets and impose a sufficient decrease condition to accept new iterates. For smooth unconstrained optimization, it is now known that such methods require at most \(\mathcal {O}(n^2\epsilon ^{-2})\) function evaluations to compute a gradient of norm below \(\epsilon \in (0,1)\), where n is the dimension of the problem. Such a maximal effort is reduced to \(\mathcal {O}(n^2\epsilon ^{-1})\) if the function is convex. The factor \(n^2\) has been derived using the positive spanning set formed by the coordinate vectors and their negatives at all iterations. In this paper, we prove that such a factor of \(n^2\) is optimal in these worst case complexity bounds, in the sense that no other positive spanning set will yield a better order of n. The proof is based on an observation that reveals the connection between cosine measure in positive spanning and sphere covering.

This is a preview of subscription content, access via your institution.

Fig. 1
Fig. 2

Notes

  1. 1.

    We are grateful to Professor Károly Böröczky, Jr. for drawing our attention to these references.

References

  1. 1.

    Bárány, I., Füredi, Z.: Approximation of the sphere by polytopes having few vertices. Proc. Am. Math. Soc. 102, 651–659 (1988)

    MathSciNet  Article  MATH  Google Scholar 

  2. 2.

    Böröczky Jr, K.: Finite Packing and Covering. Cambridge University Press, New York (2004)

    Google Scholar 

  3. 3.

    Böröczky Jr, K.: April 25, 2014 (private communication)

  4. 4.

    Böröczky Jr, K., Wintsche, G.: Covering the sphere by equal spherical balls. In: Aronov, B., Basu, S., Pach, J., Sharir, M. (eds.) Discrete and Computational Geometry. Algorithms and Combinatorics, vol. 25, pp. 235–251. Springer, Berlin (2003)

    Google Scholar 

  5. 5.

    Bourgain, J., Lindenstrauss, J., Milman, V.: Approximation of zonoids by zonotopes. Acta Math. 162, 73–141 (1989)

    MathSciNet  Article  MATH  Google Scholar 

  6. 6.

    Carl, B., Pajor, A.: Gelfand numbers of operators with values in a Hilbert space. Invent. Math. 94, 479–504 (1988)

    MathSciNet  Article  MATH  Google Scholar 

  7. 7.

    Conn, A.R., Scheinberg, K., Vicente, L.N.: Introduction to Derivative-Free Optimization. MPS-SIAM Series on Optimization. SIAM, Philadelphia (2009)

    Google Scholar 

  8. 8.

    Dalla, L., Larman, D.G., Mani-Levitska, P., Zong, C.: The blocking numbers of convex bodies. Discret. Comput. Geom. 24, 267–278 (2000)

    MathSciNet  Article  MATH  Google Scholar 

  9. 9.

    Dodangeh, M., Vicente, L.N.: Worst case complexity of direct search under convexity. Math. Program. doi:10.1007/s10107-014-0847-0

  10. 10.

    Dolan, E.D., Lewis, R.M., Torczon, V.: On the local convergence of pattern search. SIAM J. Optim. 14, 567–583 (2003)

    MathSciNet  Article  MATH  Google Scholar 

  11. 11.

    Fejes Tóth, L.: Regular Figures. Pergman Press, London (1964)

    Google Scholar 

  12. 12.

    Fercoq, O., Richtárik, P.: Accelerated, parallel and proximal coordinate descent. Technical report, School of Mathematics, University of Edinburgh, December 2013. arXiv:1312.5799v2

  13. 13.

    Gluskin, E.D.: Extremal properties of orthogonal parallelepipeds and their applications to the geometry of Banach spaces. Math. USSR-Sbornik 64, 85–96 (1989)

    MathSciNet  Article  MATH  Google Scholar 

  14. 14.

    Gratton, S., Royer, C.W., Vicente, L.N., Zhang, Z.: Direct search based on probabilistic descent. SIAM J. Optim. (to appear)

  15. 15.

    Kolda, T.G., Lewis, R.M., Torczon, V.: Optimization by direct search: new perspectives on some classical and modern methods. SIAM Rev. 45, 385–482 (2003)

    MathSciNet  Article  MATH  Google Scholar 

  16. 16.

    Konečný, J., Richtárik, P.: Simple complexity analysis of simplified direct search. Technical report, School of Mathematics, University of Edinburgh, November 2014. arXiv:1410.0390v2

  17. 17.

    Nesterov, Y.: Efficiency of coordinate descent methods on huge-scale optimization problems. SIAM J. Optim. 22, 341–362 (2012)

    MathSciNet  Article  MATH  Google Scholar 

  18. 18.

    Tikhomirov, K.E.: On the distance of polytopes with few vertices to the Euclidean ball. Discret. Comput. Geom. 53, 173–181 (2015)

    MathSciNet  Article  MATH  Google Scholar 

  19. 19.

    Torczon, V.: On the convergence of pattern search algorithms. SIAM J. Optim. 7, 1–25 (1997)

    MathSciNet  Article  MATH  Google Scholar 

  20. 20.

    Vicente, L.N.: Worst case complexity of direct search. Euro J. Comput. Optim. 1, 143–153 (2013)

    Article  MATH  Google Scholar 

Download references

Acknowledgments

We would like to thank Professors Károly Böröczky, Jr., Ilya Dumer, Gabor Fejes Tóth, and Tibor Tarnai, with whom we had helpful discussions on sphere covering.

Author information

Affiliations

Authors

Corresponding author

Correspondence to Z. Zhang.

Additional information

M. Dodangeh: support for this author was provided by FCT under the scholarship SFRH/BD/51168/2010. L. N. Viente: support for this author’s research was provided by FCT under Grants PTDC MAT/116736/2010 and PEst-C/MAT/UI0324/2011. Z. Zhang: this author works within the framework of the Project FILAOS funded by RTRA STAE.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Dodangeh, M., Vicente, L.N. & Zhang, Z. On the optimal order of worst case complexity of direct search. Optim Lett 10, 699–708 (2016). https://doi.org/10.1007/s11590-015-0908-1

Download citation

Keywords

  • Direct search
  • Worst case complexity
  • Optimal order
  • Sphere covering
  • Positive spanning set
  • Cosine measure