Abstract
Efficient global optimization (EGO) is one of the most popular sequential adaptive design (SAD) methods for expensive black-box optimization problems. A well-recognized weakness of the original EGO in complex computer experiments is that it is serial, and hence the modern parallel computing techniques cannot be utilized to speed up the running of simulator experiments. For those multiple points EGO methods, the heavy computation and points clustering are the obstacles. In this work, a novel batch SAD method, named “Accelerated EGO”, is forwarded by using a refined sampling/importance resampling (SIR) method to search the points with large expected improvement (EI) values. The computation burden of the new method is much lighter, and the points clustering is also avoided. The efficiency of the proposed batch SAD is validated by nine classic test functions with dimension from 2 to 12. The empirical results show that the proposed algorithm indeed can parallelize original EGO, and gain much improvement compared against the other parallel EGO algorithm especially under high-dimensional case. Additionally, the new method is applied to the hyper-parameter tuning of support vector machine (SVM) and XGBoost models in machine learning. Accelerated EGO obtains comparable cross validation accuracy with other methods and the CPU time can be reduced a lot due to the parallel computation and sampling method.
Similar content being viewed by others
References
Chang, C. C., & Lin, C. J. (2011). Libsvm: A library for support vector machines. ACM Transactions on Intelligent Systems and Technology, 27(2), 1–27. http://www.csie.ntu.edu.tw/~cjlin/libsvm
Chen, T., & Guestrin, C. (2016). XGBoost: A scalable tree boosting system. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining (pp. 785–794). New York: ACM.
Chevalier, C., & Ginsbourger, D. (2013). Fast computation of the multi-points expected improvement with applications in batch selection. In International conference on learning and intelligent optimization (pp. 59–69). New York: Springer.
Christophe, D., & Petr, S. (2019). randtoolbox: Generating and Testing Random Numbers. R package version 1.30.0. https://CRAN.R-project.org/package=randtoolbox
Cristianini, N., & John, S. T. (2000). An introduction to support vector machines and other kernel-based learning methods. Cambridge: Cambridge University Press.
Fang, K. T., Li, R. Z., & Sudjianto, A. (2006). Design and modeling for computer experiments. London: Chapman & Hall/CRC.
Fang, K. T., Liu, M. Q., Qin, H., et al. (2018). Theory and application of uniform experimental designs. New York: Springer.
Ginsbourger, D., Riche, R. L., & Carraro, L., et al. (2010). Kriging is well-suited to parallelize optimization. In L. M. Hiot, Y. S. Ong, & Y. Tenne (Eds.), Computational intelligence in expensive optimization problems (pp. 131–162). Berlin: Springer.
He, X. (2019). Sliced rotated sphere packing designs. Technometrics, 61(1), 66–76.
He, X., Tuo, R., & Wu, C. F. J. (2017). Optimization of multi-fidelity computer experiments via the EQIE criterion. Technometrics, 59(1), 58–68.
Huang, D., Allen, T., Notz, W., et al. (2006). Global optimization of stochastic black-box systems via sequential kriging meta-models. Journal of Global Optimization, 32, 441–466.
Hutter, F., Kotthoff, L., & Vanschoren, J. (2019). Automated machine learning-methods, systems, challenges. New York: Springer.
Jones, D. R. (2001). A taxonomy of global optimization methods based on response surfaces. Journal of Global Optimization, 21(4), 345–383.
Jones, D. R., Schonlau, M., & Welch, W. J. (1998). Efficient global optimization of expensive black-box functions. Journal of Global Optimization, 13(4), 455–492.
Krige, D. G. (1951). A statistical approach to some mine valuations and allied problems at the witwatersrand. Master’s thesis, University of Witwatersrand.
Lam, C., & Notz, W. (2008). Sequential adaptive designs in computer experiments for response surface model fit. Statistics and Applications, 6(2), 207–233.
L’Ecuyer, P., & Lemieux, C. (2002). Recent advances in randomized quasi-monte carlo methods. In M. Dror, P. L’Ecuyer, & F. Szidarovszky (Eds.), Modeling uncertainty: An examination of stochastic theory, methods, and applications (pp. 419–474). Boston: Kluwer Academic.
Locatelli, M. (1997). Bayesian algorithms for one-dimensional global optimization. Journal of Global Optimization, 10, 57–76.
Loeppky, J. L., Moore, L. M., & Williams, B. J. (2010). Batch sequential designs for computer experiments. Journal of Statistical Planning and Inference, 140(6), 1452–1464.
Loeppky, J. L., Sacks, J., & Welch, W. J. (2009). Choosing the sample size of a computer experiment: A practical guide. Technometrics, 51(4), 366–376.
Mak, S., & Joseph, V. R. (2017). Support points. Annals of Statistics, 46(6), 2562–2592.
Mehdad, E., & Kleijnen, J. P. (2018). Efficient global optimisation for black-box simulation via sequential intrinsic kriging. Journal of the Operational Research Society, 69(11), 1725–37.
Niederreiter, H. (1992). Random number generation and quasi-Monte Carlo methods. Philadelphia: Society for Industrial and Applied Mathematics.
Ooi, H., & Microsoft, Weston, S. (2019b). foreach: Provides foreach looping construct. R package version 1.4.7. https://CRAN.R-project.org/package=foreach
Ooi, H., Corporation, M., & Weston, S., et al. (2019a). doParallel: Foreach parallel adaptor for the ‘treparallel’ Package. R package version 1.0.15. https://CRAN.R-project.org/package=doParallel
Ponweiser, W., Wagner, T., & Vincze, M. (2008). Clustered multiple generalized expected improvement: A novel infill sampling criterion for surrogate models. IEEE Congress on Evolutionary Computation, 1, 3514–3521.
Quan, A. (2014). Batch sequencing methods for computer experiments. Doctor of Philosophy. Ohio State University, Ohio.
Roustant, O., Ginsbourger, D., & Deville, Y. (2012). DiceKriging, DiceOptim: Two R packages for the analysis of computer experiments by kriging-based metamodeling and optimization. Journal of Statistical Software, 51(1), 1–55.
Sacks, J., Welch, W. J., Mitchell, T. J., et al. (1989). Design and analysis of computer experiments (with discussion). Statistical Science, 4(4), 409–435.
Santner, T., Williams, B., & Notz, W. (2018). The design and analysis of computer experiments (2nd ed.). New York: Springer.
Schonlau, M. (1997). Computer experiments and global optimization. PhD thesis, University of Waterloo, Waterloo.
Surjanovic, S., & Bingham, D. (2013). Virtual library of simulation experiments: Test functions and datasets. Retrieved August 30, 2021, from http://www.sfu.ca/~ssurjano
Viana, F. A. C., Haftka, R. T., & Watson, L. T. (2013). Efficient global optimization algorithm assisted by multiple surrogate techniques. Journal of Global Optimization, 56(2), 669–689.
Zhang, A., Li, H., & Quan, S., et al. (2018). UniDOE: Uniform Design of Experiments. R package version 1.0.2. https://CRAN.R-project.org/package=UniDOE
Acknowledgements
This study is partially supported by the National Natural Science Foundation of China (Nos. 11571133, 11871237 and 12001540).
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Supplementary Information
Below is the link to the electronic supplementary material.
Appendix
Appendix
The constant Liar (CL), one of the parallel EGO algorithms, is a sequential strategy for targeting the multiple points at each cycle, in which the surrogate model is updated (still without hyper-parameter re-estimation) at each iteration with a value \(L\in R\) exogenously fixed by the user, called a “lie”. Three values, \(\min \{\varvec{y}\}\), \(\max \{\varvec{y}\}\) and \(\text {mean}\{\varvec{y}\}\) were considered in Ginsbourger et al. (2010). In our simulations, the lie “L” is taken as \(\min \{\varvec{y}\}\).
Rights and permissions
About this article
Cite this article
Xiao, Y., Ning, J., Xiong, Z. et al. Batch sequential adaptive designs for global optimization. J. Korean Stat. Soc. 51, 780–802 (2022). https://doi.org/10.1007/s42952-022-00161-9
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s42952-022-00161-9