On the Ultimate Convergence Rates for Isotropic Algorithms and the Best Choices Among Various Forms of Isotropy

  • Olivier Teytaud
  • Sylvain Gelly
  • Jérémie Mary
Part of the Lecture Notes in Computer Science book series (LNCS, volume 4193)


In this paper, we show universal lower bounds for isotropic algorithms, that hold for any algorithm such that each new point is the sum of one already visited point plus one random isotropic direction multiplied by any step size (whenever the step size is chosen by an oracle with arbitrarily high computational power). The bound is 1–O(1/d) for the constant in the linear convergence (i.e. the constant C such that the distance to the optimum after n steps is upper bounded by C n ), as already seen for some families of evolution strategies in [19,12], in contrast with 1–O(1) for the reverse case of a random step size and a direction chosen by an oracle with arbitrary high computational power. We then recall that isotropy does not uniquely determine the distribution of a sample on the sphere and show that the convergence rate in isotropic algorithms is improved by using stratified or antithetic isotropy instead of naive isotropy. We show at the end of the paper that beyond the mathematical proof, the result holds on experiments. We conclude that one should use antithetic-isotropy or stratified-isotropy, and never standard-isotropy.


Convergence Rate Unit Sphere Sphere Function Linear Convergence Random Independent Sampling 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Auger, A.: Convergence results for (1,λ)-SA-ES using the theory of ϕ-irreducible markov chains. Theoretical Computer Science (in press, 2005)Google Scholar
  2. 2.
    Auger, A., Jebalia, M., Teytaud, O.: Xse: quasi-random mutations for evolution strategies. In: Proceedings of Evolutionary Algorithms, pages 12 (2005)Google Scholar
  3. 3.
    Beyer, H.-G.: The Theory of Evolutions Strategies. Springer, Heidelberg (2001)Google Scholar
  4. 4.
    Broyden, C.G.: The convergence of a class of double-rank minimization algorithms 2, the new algorithm. j. of the inst. for math. and applications 6, 222–231 (1970)MATHCrossRefMathSciNetGoogle Scholar
  5. 5.
    Chazelle, B.: The discrepancy method: randomness and complexity. Cambridge University Press, New York (2000)MATHGoogle Scholar
  6. 6.
    Conway, J.H., Sloane, N.J.: Sphere packings, lattices and groups (1998)Google Scholar
  7. 7.
    Fletcher, R.: A new approach to variable-metric algorithms. computer journal 13, 317–322 (1970)MATHCrossRefGoogle Scholar
  8. 8.
    Frahm, G., Junker, M.: Generalized elliptical distributions: Models and estimation. Technical Report 0037 (2003)Google Scholar
  9. 9.
    Goldfarb, D.: A family of variable-metric algorithms derived by variational means. mathematics of computation 24, 23–26 (1970)MATHCrossRefMathSciNetGoogle Scholar
  10. 10.
    Haagerup, U.: The best constants in the khintchine inequality. Studia Math. 70, 231–283 (1982)MATHMathSciNetGoogle Scholar
  11. 11.
    Hansen, N., Ostermeier, A.: Completely derandomized self-adaptation in evolution strategies. Evolutionary Computation 11(1) (2003)Google Scholar
  12. 12.
    Jagerskupper, J.: In between progress rate and stochastic convergence. Dagstuhl’s seminar (2006)Google Scholar
  13. 13.
    Literka: A remarkable monotonic property of the gamma function. Technical report (2005)Google Scholar
  14. 14.
    Salomon, R.: Resampling and its avoidance in genetic algorithms. In: Porto, V.W., Saravanan, N., Waagen, D., Eiben, A.E. (eds.) Evolutionary Programming VII, pp. 335–344. Springer, Berlin (1998)CrossRefGoogle Scholar
  15. 15.
    Salomon, R.: The deterministic genetic algorithm: Implementation details and some results (1999)Google Scholar
  16. 16.
    Shanno, D.F.: Conditioning of quasi-newton methods for function minimization. mathematics of computation 24, 647–656 (1970)CrossRefMathSciNetGoogle Scholar
  17. 17.
    Teytaud, O., Gelly, S.: General lower bounds for evolutionary algorithms, ppsn (2006)Google Scholar
  18. 18.
    Wang, Z., Droegemeier, K., White, L., Navon, I.M.: Application of a new adjoint newton algorithm to the 3-d arps storm scale model using simulated data. Monthly Weather Review 125(10), 2460–2478 (1997)CrossRefGoogle Scholar
  19. 19.
    Witt, C., Jägersküpper, J.: Rigorous runtime analysis of a (mu+1) es for the sphere function. In: Proceedings of the 2005 Conference on Genetic and Evolutionary Computation, pp. 849–856 (2005)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2006

Authors and Affiliations

  • Olivier Teytaud
    • 1
  • Sylvain Gelly
    • 1
  • Jérémie Mary
    • 1
  1. 1.TAO (Inria), LRI, UMR 8623(CNRS – Univ. Paris-Sud)Univ. Paris-SudOrsayFrance

Personalised recommendations