Methods to compare expensive stochastic optimization algorithms with random restarts
We consider the challenge of numerically comparing optimization algorithms that employ random-restarts under the assumption that only limited test data is available. We develop a bootstrapping technique to estimate the incumbent solution of the optimization problem over time as a stochastic process. The asymptotic properties of the estimator are examined and the approach is validated by an out-of-sample test. Finally, three methods for comparing the performance of different algorithms based on the estimator are proposed and demonstrated with data from a real-world optimization problem.
KeywordsRandom restarts Stochastic optimization Benchmarking Nonconvex optimization
This work was supported by Natural Sciences and Engineering Research Council of Canada (NSERC) under Collaborative Research and Development (CRD) Grant #CRDPJ 411318-15. The Grant was sponsored by Softree Technical Systems Inc. This work was supported by Discovery Grants #355571-2013 (Hare) and #2015-03895 (Loeppky) from NSERC. Part of the research was performed in the Computer-Aided Convex Analysis (CA2) laboratory funded by a Leaders Opportunity Fund (LOF) from the Canada Foundation for Innovation (CFI) and by a British Columbia Knowledge Development Fund (BCKDF).
- 1.Abramson, M.A., Audet, C., Couture, G., Dennis Jr., J.E., Le Digabel, S., Tribes, C.: The NOMAD project. Software available at https://www.gerad.ca/nomad/
- 6.Currie, J., Wilson, D.: OPTI: lowering the barrier between open source optimizers and the industrial MATLAB user. In: Sahinidis, N., Pinto, J. (eds.) Foundations of Computer-Aided Process Operations. Savannah, Georgia (2012)Google Scholar
- 10.Fowler, K.R., Reese, J.P., Kees, C.E., Dennis Jr., J.E., Kelley, C.T., Miller, C.T., Audet, C., Booker, A.J., Couture, G., Darwin, R.W., Farthing, M.W., Finkel, D.E., Gablonsky, J.M., Gray, G., Kolda, T.G.: Comparison of derivative-free optimization methods for groundwater supply and hydraulic capture community problems. Adv. Water Resour. 31(5), 743–757 (2008)CrossRefGoogle Scholar
- 11.Glover, F.: A Template for Scatter Search and Path Relinking, pp. 1–51. Springer, Berlin (1998)Google Scholar
- 13.Hoos, H.H., Stützle, T.: Evaluating Las Vegas algorithms: pitfalls and remedies. In: Proceedings of the Fourteenth Conference on Uncertainty in Artificial Intelligence, UAI’98, pp. 238–245, San Francisco, CA, USA, 1998. Morgan Kaufmann Publishers IncGoogle Scholar
- 19.G.A. Ortiz: Evolution strategies, May 2012. http://www.mathworks.com/matlabcentral/fileexchange/35801-evolution-strategies--es-