Abstract
This article uses a sequentialized experimental design to select simulation input combinations for global optimization, based on Kriging (also called Gaussian process or spatial correlation modeling); this Kriging is used to analyze the input/output data of the simulation model (computer code). This design and analysis adapt the classic “expected improvement” (EI) in “efficient global optimization” (EGO) through the introduction of an improved estimator of the Kriging predictor variance; this estimator uses parametric bootstrapping. Classic EI and bootstrapped EI are compared through various test functions, including the six-hump camel-back and several Hartmann functions. These empirical results demonstrate that in some applications bootstrapped EI finds the global optimum faster than classic EI does; in general, however, the classic EI may be considered to be a robust global optimizer.
Article PDF
Similar content being viewed by others
Avoid common mistakes on your manuscript.
References
Abt M.: Estimating the prediction mean squared error in Gaussian stochastic processes with exponential correlation structure. Scand. J. Stat. 26(4), 563–578 (1999)
Ankenman B., Nelson B., Staum J.: Stochastic kriging for simulation metamodeling. Oper. Res. 58(2), 371–382 (2010)
Bull, A.D.: Convergence Rates of Efficient Global Optimization algorithms. Arxiv (2011)
Cressie N.A.C.: Statistics for Spatial Data: Revised Edition. Wiley, New York (1993)
del Castillo, E., Santiago, E.: A Matrix-T approach to the sequential design of optimization experiments. IIE Trans. (accepted 2010)
Den Hertog D., Kleijnen Siem A.Y.D.: The correct kriging variance estimated by bootstrapping. J. Oper. Res. Soc. 57, 400–409 (2006)
Efron B., Tibshirani R.J.: An Introduction to the Bootstrap. Chapman & Hall, New York (1993)
Forrester A.I.J., Keane A.J.: Recent advances in surrogate-based optimization. Prog. Aearosp. Sci. 45(1–3), 50–79 (2009)
Forrester A., Sóbester A., Keane A.: Engineering Design via Surrogate Modelling: a Practical Guide. Wiley, Chichester (2008)
Frazier, P.I . : Learning with Dynamic Programming. In: Cochran, J.J., Cox, L.A., Keskinocak, P., Kharoufeh, J.P., Smith, J.C. (eds.)Wiley Encyclopedia of Operations Research and Management Science, Wiley, New York (2011)
Frazier, P., Powell, W., Dayanik, S.: The knowledge-gradient policy for correlated normal beliefs. INFORMS J. Comput. (2009)
Fu M.C.: Are we there yet? The marriage between simulation & optimization. OR/MS Today 34, 16–17 (2007)
Gorissen, D.: Grid-enabled Adaptive Surrogate Modeling for Computer Aided Engineering. Ph.D. dissertation, Ghent University, Ghent, Belgium (2010)
Hardle W., Simar L.: Applied Multivariate Statistical Analysis. Springer, New York (2003)
Jones D.R., Schonlau M., Welch W.J.: Efficient global optimization of expensive black-box functions. J. Global Optim. 13, 455–492 (1998)
Kleijnen, J.P.C.: Design and Analysis of Simulation Experiments.Springer (Chinese translation: published by Publishing House of Electronics Industry, Beijing, 2010) (2008)
Kleijnen J.P.C., Van Beers W., Van Nieuwenhuyse I.: Constrained optimization in simulation: a novel approach. EJOR 202, 164–174 (2010)
Loeppky J. L., Sacks J., Welch W.: Choosing the sample size of a computer experiment: a practical guide. Technometrics 51, 366–376 (2009)
Lophaven S.N., Nielsen H.B., Sondergaard J.: DACE: a Matlab Kriging Toolbox, Version 2.0. IMM Technical University of Denmark, Lyngby (2002)
Martin J.D., Simpson T.W.: On the use of kriging models to approximate deterministic computer models. AIAA J. 43(4), 853–863 (2005)
Müller, W. G., Pronzato, L.: Towards an optimal design equivalence theorem for random fields? IFAS Research Paper Series No. 2009-45, Department of Applied Statistics, Johannes Kepler University Linz, Linz, Austria (2009)
Nakayama H., Yun Y., Yoon M.: Sequential Approximate Multiobjective Optimization Using Computational Intelligence. Springer, Berlin (2009)
Picheny, V., Ginsbourger, D., Richet, Y.: Noisy Expected Improvement and on-line computation time allocation for the optimization of simulators with tunable fidelity. In: 2nd International Conference on Engineering Optimization, September 6–9, 2010, Lisbon, Portugal (2010)
Regis R.G.: Stochastic radial basis function algorithms for large-scale optimization involving expensive black-box objective and constraint functions. Comput. & OR 38, 837–853 (2011)
Sacks J., Welch W.J., Mitchell T.J., Wynn H.P.: Design and analysis of computer experiments (includes comments and rejoinder). Stat. Sci. 4(4), 409–435 (1989)
Santner T.J., Williams B.J., Notz W.I.: The Design and Analysis of Computer Experiments. Springer, New York (2003)
Sasena M.J, Papalambros P., Goovaerts P.: Exploration of metamodeling sampling criteria for constrained global optimization. Eng. Optimiz. 34(3), 263–278 (2002)
Schonlau, M.: Computer Experiments and Global Optimization, Ph.D. thesis, University of Waterloo, Waterloo, Canada (1997)
Simpson T.W., Booker A.J., Ghosh D., Giunta A.A., Koch P.N., Yang R.-J.: Approximation methods in multidisciplinary analysis and optimization: a panel discussion. Struct. Multidiscip. O. 27(5), 302–313 (2004)
Törn A., Z̆ilinkas A.: Global Optimization. Springer, Berlin (1989)
Villemonteix J., Vazquez E., Sidorkiewicz M., Walter E.: Global optimization of expensive-to-evaluate functions: an empirical comparison of two sampling criteria. J. Global Optim 43(2–3), 373–389 (2009)
Yin, J., Ng, S.H., Ng, K.M.: A Bayesian metamodeling approach for stochastic simulations. In: Johansson, B., Jain, S., Montoya-Torres, J., Hugan, J., Yücesan, E. (eds.), Pres. 2010 WSC, pp. 1055–1066 (2010)
Acknowledgments
We thank the anonymous referee for a very detailed report that lead to the additional experiments in Table 1 aimed at investigating the effects of different initial sample sizes, the example in the Appendix, a simpler notation, and many other minor improvements. We also thank Emmanuel Vazquez (SUPÉLEC) for bringing Abt (1999) and Müller and Pronzato (2009) to our attention.
Open Access
This article is distributed under the terms of the Creative Commons Attribution Noncommercial License which permits any noncommercial use, distribution, and reproduction in any medium, provided the original author(s) and source are credited.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
Open Access This is an open access article distributed under the terms of the Creative Commons Attribution Noncommercial License (https://creativecommons.org/licenses/by-nc/2.0), which permits any noncommercial use, distribution, and reproduction in any medium, provided the original author(s) and source are credited.
About this article
Cite this article
Kleijnen, J.P.C., van Beers, W. & van Nieuwenhuyse, I. Expected improvement in efficient global optimization through bootstrapped kriging. J Glob Optim 54, 59–73 (2012). https://doi.org/10.1007/s10898-011-9741-y
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10898-011-9741-y