Methods to compare expensive stochastic optimization algorithms with random restarts

  • Warren Hare
  • Jason Loeppky
  • Shangwei Xie


We consider the challenge of numerically comparing optimization algorithms that employ random-restarts under the assumption that only limited test data is available. We develop a bootstrapping technique to estimate the incumbent solution of the optimization problem over time as a stochastic process. The asymptotic properties of the estimator are examined and the approach is validated by an out-of-sample test. Finally, three methods for comparing the performance of different algorithms based on the estimator are proposed and demonstrated with data from a real-world optimization problem.


Random restarts Stochastic optimization Benchmarking Nonconvex optimization 



This work was supported by Natural Sciences and Engineering Research Council of Canada (NSERC) under Collaborative Research and Development (CRD) Grant #CRDPJ 411318-15. The Grant was sponsored by Softree Technical Systems Inc. This work was supported by Discovery Grants #355571-2013 (Hare) and #2015-03895 (Loeppky) from NSERC. Part of the research was performed in the Computer-Aided Convex Analysis (CA2) laboratory funded by a Leaders Opportunity Fund (LOF) from the Canada Foundation for Innovation (CFI) and by a British Columbia Knowledge Development Fund (BCKDF).


  1. 1.
    Abramson, M.A., Audet, C., Couture, G., Dennis Jr., J.E., Le Digabel, S., Tribes, C.: The NOMAD project. Software available at
  2. 2.
    Audet, C., Dennis Jr., J.: Mesh adaptive direct search algorithms for constrained optimization. SIAM J. Optim. 17(1), 188–217 (2006)MathSciNetCrossRefzbMATHGoogle Scholar
  3. 3.
    Beiranvand, V., Hare, W., Lucet, Y.: Best practices for comparing optimization algorithms. Optim. Eng. 18(4), 815–848 (2017)MathSciNetCrossRefGoogle Scholar
  4. 4.
    Bickel, P.J., Freedman, D.A.: Some asymptotic theory for the bootstrap. Ann. Stat. 9(6), 1196–1217 (1981)MathSciNetCrossRefzbMATHGoogle Scholar
  5. 5.
    Butler, A., Haynes, R.D., Humphries, T.D., Ranjan, P.: Efficient optimization of the likelihood function in Gaussian process modelling. Comput. Stat. Data Anal. 73, 40–52 (2014)MathSciNetCrossRefGoogle Scholar
  6. 6.
    Currie, J., Wilson, D.: OPTI: lowering the barrier between open source optimizers and the industrial MATLAB user. In: Sahinidis, N., Pinto, J. (eds.) Foundations of Computer-Aided Process Operations. Savannah, Georgia (2012)Google Scholar
  7. 7.
    Dolan, E.D., Moré, J.J.: Benchmarking optimization software with performance profiles. Math. Program. 91(2), 201–213 (2002)MathSciNetCrossRefzbMATHGoogle Scholar
  8. 8.
    Efron, B., Tibshirani, R.J.: An Introduction to the Bootstrap. Number 57 in Monographs on Statistics and Applied Probability. Chapman & Hall/CRC, Boca Raton (1993)CrossRefzbMATHGoogle Scholar
  9. 9.
    Feo, T.A., Resende, M.G.C., Smith, S.H.: A greedy randomized adaptive search procedure for maximum independent set. Oper. Res. 42(5), 860–878 (1994)CrossRefzbMATHGoogle Scholar
  10. 10.
    Fowler, K.R., Reese, J.P., Kees, C.E., Dennis Jr., J.E., Kelley, C.T., Miller, C.T., Audet, C., Booker, A.J., Couture, G., Darwin, R.W., Farthing, M.W., Finkel, D.E., Gablonsky, J.M., Gray, G., Kolda, T.G.: Comparison of derivative-free optimization methods for groundwater supply and hydraulic capture community problems. Adv. Water Resour. 31(5), 743–757 (2008)CrossRefGoogle Scholar
  11. 11.
    Glover, F.: A Template for Scatter Search and Path Relinking, pp. 1–51. Springer, Berlin (1998)Google Scholar
  12. 12.
    Grishagin, V.A.: Operating characteristics of some global search algorithms. Probl. Stoch. Search 7, 198–206 (1978)zbMATHGoogle Scholar
  13. 13.
    Hoos, H.H., Stützle, T.: Evaluating Las Vegas algorithms: pitfalls and remedies. In: Proceedings of the Fourteenth Conference on Uncertainty in Artificial Intelligence, UAI’98, pp. 238–245, San Francisco, CA, USA, 1998. Morgan Kaufmann Publishers IncGoogle Scholar
  14. 14.
    Houck, C.R., Joines, J.A., Kay, M.G.: Comparison of genetic algorithms, random restart and two-opt switching for solving large location–allocation problems. Comput. Oper. Res. 23(6), 587–596 (1996)CrossRefzbMATHGoogle Scholar
  15. 15.
    Kocsis, L., György, A.: Efficient Multi-start Strategies for Local Search Algorithms, pp. 705–720. Springer, Berlin (2009)zbMATHGoogle Scholar
  16. 16.
    Kuindersma, S.R., Grupen, R.A., Barto, A.G.: Variable risk control via stochastic optimization. Int. J. Robot. Res. 32(7), 806–825 (2013)CrossRefGoogle Scholar
  17. 17.
    Mondal, S., Lucet, Y., Hare, W.: Optimizing horizontal alignment of roads in a specified corridor. Comput. Oper. Res. 64, 130–138 (2015)MathSciNetCrossRefzbMATHGoogle Scholar
  18. 18.
    Moré, J., Wild, S.: Benchmarking derivative-free optimization algorithms. SIAM J. Optim. 20(1), 172–191 (2009)MathSciNetCrossRefzbMATHGoogle Scholar
  19. 19.
  20. 20.
    Sergeyev, Y.D., Kvasov, D.E., Mukhametzhanov, M.S.: Operational zones for comparing metaheuristic and deterministic one-dimensional global optimization algorithms. Math. Comput. Simul. 141, 96–109 (2016)MathSciNetCrossRefGoogle Scholar
  21. 21.
    Sergeyev, Y.D., Kvasov, D.E., Mukhametzhanov, M.S.: On the efficiency of nature-inspired metaheuristics in expensive global optimization with limited budget. Sci. Rep. (2018). Google Scholar
  22. 22.
    Spall, J.C.: Introduction to Stochastic Search and Optimization: Estimation, Simulation, and Control. Wiley-Interscience, Hoboken (2003)CrossRefzbMATHGoogle Scholar
  23. 23.
    Stich, S.U., Müller, C.L., Gärtner, B.: Optimization of convex functions with random pursuit. SIAM J. Optim. 23(2), 1284–1309 (2013)MathSciNetCrossRefzbMATHGoogle Scholar
  24. 24.
    Vasant, P., Weber, G., Dieu, V.: Handbook of Research on Modern Optimization Algorithms and Applications in Engineering and Economics. IGI Global, Hershey (2016)CrossRefzbMATHGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2018

Authors and Affiliations

  1. 1.MathematicsUniversity of British ColumbiaKelownaCanada
  2. 2.StatisticsUniversity of British ColumbiaKelownaCanada
  3. 3.Computer ScienceUniversity of British ColumbiaKelownaCanada

Personalised recommendations