Abstract
This paper discusses a new sequential adaptive design of experiments (DoE) approach for global Kriging metamodeling applications. The sequential implementation is established by using the current metamodel, formulated based on the existing experiments, to guide the selection of the optimal new experiment(s). The score function, defining the DoE objective, combines two components: (1) the metamodel prediction variability, expressed through the predictive variance, and (2) the metamodel bias, approximated through the leave-one-out cross validation (LOOCV) error. The latter is used as a weighting factor to extend traditional DoE approaches that focus solely on the metamodel prediction variability. Two such approaches are considered here, adopting either the integrated mean squared error or the maximum mean squared error as the basic component of the score function. The incorporation of bias information as weighting within these well-established approaches facilitates a direct extension of their respective computational workflows, making the proposed implementation attractive from computational perspective. An efficient optimization scheme for identification of the next experiment, as well as the balancing of exploration and exploitation between the two components of the score function, are also discussed. The incorporation of LOOCV weightings is shown to be highly beneficial in a total of six analytical and engineering examples. Furthermore, these examples demonstrate that for DoE approaches which use LOOCV information as weights, it is preferable to update the predictive variance to explicitly consider the impact of the new experiment, rather than relying strictly on the current metamodel variance.
Similar content being viewed by others
References
Aute V, Saleh K, Abdelaziz O, Azarm S, Radermacher R (2013) Cross-validation based single response adaptive design of experiments for Kriging metamodeling of deterministic computer simulations. Struct Multidiscip Optim 48(3):581–605
Bect J, Li L, Vazquez E (2017) Bayesian subset simulation. SIAM/ASA Journal on Uncertainty Quantification 5(1):762–786
Chen T, Morris J, Martin E (2007) Gaussian process regression for multivariate spectroscopic calibration. Chemom Intell Lab Syst 87(1):59–71
Dixon LCW, Szegö GP (1978) Towards global optimisation, vol 2. North-Holland Amsterdam,
Dubourg V, Sudret B, Bourinet J-M (2011) Reliability-based design optimization using kriging surrogates and subset simulation. Struct Multidiscip Optim 44(5):673–690
Dubrule O (1983) Cross validation of kriging in a unique neighborhood. J Int Assoc Math Geol 15(6):687–699
Falnes J (2002) Ocean waves and oscillating systems: linear interactions including wave-energy extraction. Cambridge university press,
Forrester AI, Keane AJ (2009) Recent advances in surrogate-based optimization. Prog Aerosp Sci 45(1–3):50–79
Hartigan JA, Wong MA (1979) Algorithm AS 136: a K-means clustering algorithm. J R Stat Soc: Ser C: Appl Stat 28(1):100–108
Hu Z, Mahadevan S (2016) A single-loop kriging surrogate modeling for time-dependent reliability analysis. J Mech Design 138(6):061406
Jia G, Taflanidis AA (2013) Kriging metamodeling for approximation of high-dimensional wave and surge responses in real-time storm/hurricane risk assessment. Comput Methods Appl Mech Eng 261:24–38
Jia G, Taflanidis AA (2014) Sample-based evaluation of global probabilistic sensitivity measures. Comput Struct 144:103–118
Jiang P, Shu L, Zhou Q, Zhou H, Shao X, Xu J (2015) A novel sequential exploration-exploitation sampling strategy for global metamodeling. IFAC-PapersOnLine 48(28):532–537
Jin R, Chen W, Sudjianto A (2002) On sequential sampling for global metamodeling in engineering design. In: ASME 2002 international design engineering technical conferences and computers and information in engineering conference. American Society of Mechanical Engineers, pp 539-548
Johnson ME, Moore LM, Ylvisaker D (1990) Minimax and maximin distance designs. J Stat Plan Infer 26(2):131–148
Jones DR, Schonlau M, Welch WJ (1998) Efficient global optimization of expensive black-box functions. J Glob Optim 13(4):455–492
Kennedy MC, O'Hagan A (2001) Bayesian calibration of computer models. J R Stat Soc Series B Stat Methodol 63(3):425–464
Kianifar MR, Campean F (2019) Performance evaluation of metamodelling methods for engineering problems: towards a practitioner guide. Struct Multidiscip Optim:1–28
Kleijnen JP (2009) Kriging metamodeling in simulation: a review. Eur J Oper Res 192(3):707–716
Kleijnen JP (2015) Design and analysis of simulation experiments. International Workshop on Simulation. Springer, In, pp 3–22
Kleijnen JP, Beers W (2004) Application-driven sequential designs for simulation experiments: Kriging metamodelling. J Oper Res Soc 55(8):876–883
Koehler J, Owen A (1996) Computer experiments. Handbook of statistics. (S. Ghosh and CR Rao, eds). Elsevier science, New York
Le Gratiet L, Cannamela C (2015) CoKriging-based sequential design strategies using fast cross-validation techniques for multi-fidelity computer codes. Technometrics 57(3):418–427
Liu H, Ong Y-S, Cai J (2018) A survey of adaptive sampling for global metamodeling in support of simulation-based complex engineering design. Struct Multidiscip Optim 57(1):393–416
Liu H, Xu S, Ma Y, Chen X, Wang X (2016) An adaptive Bayesian sequential sampling approach for global metamodeling. J Mech Design 138(1):011404
Lophaven SN, Nielsen HB, Søndergaard J (2002) DACE-A Matlab Kriging toolbox, version 2.0
Mavroeidis GP, Papageorgiou AS (2003) A mathematical representation of near-fault ground motions. Bull Seismol Soc Am 93(3):1099–1131
McKay MD, Beckman RJ, Conover WJ (1979) Comparison of three methods for selecting values of input variables in the analysis of output from a computer code. Technometrics 21(2):239–245
Mehmani A, Chowdhury S, Meinrenken C, Messac A (2018) Concurrent surrogate model selection (COSMOS): optimizing model type, kernel function, and hyper-parameters. Struct Multidiscip Optim 57(3):1093–1114
Mishra SK (2006) Global optimization by differential evolution and particle swarm methods: evaluation on some benchmark functions. Available at SSRN 933827
Picheny V, Ginsbourger D, Roustant O, Haftka RT, Kim N-H (2010) Adaptive designs of experiments for accurate approximation of a target region. J Mech Design 132(7):071008
Picheny V, Wagner T, Ginsbourger D (2013) A benchmark of kriging-based infill criteria for noisy optimization. Struct Multidiscip Optim 48(3):607–626
Queipo NV, Haftka RT, Shyy W, Goel T, Vaidyanathan R, Tucker PK (2005) Surrogate-based analysis and optimization. Prog Aerosp Sci 41(1):1–28
Sacks J, Welch WJ, Mitchell TJ, Wynn HP (1989) Design and analysis of computer experiments. Stat Sci 4(4):409–435
Santner TJ, Williams BJ, Notz W, Williams BJ (2003) The design and analysis of computer experiments, vol 1. Springer
Schöbi R, Sudret B, Marelli S (2016) Rare event estimation using polynomial-chaos kriging. ASCE-ASME J. Risk Uncertain. Eng. Syst., part a. Civ Eng 3(2):D4016002
Shewry MC, Wynn HP (1987) Maximum entropy sampling. J Appl Stat 14(2):165–170
Simpson TW, Mauery TM, Korte JJ, Mistree F (2001a) Kriging models for global approximation in simulation-based multidisciplinary design optimization. AIAA J 39(12):2233–2241
Simpson TW, Poplinski J, Koch PN, Allen JK (2001b) Metamodels for computer-based engineering design: survey and recommendations. Eng Comput 17(2):129–150
Spall JC (2005) Introduction to stochastic search and optimization: estimation, simulation, and control, vol 65. John Wiley & Sons
Sundararajan S, Keerthi SS (2001) Predictive approaches for choosing hyperparameters in Gaussian processes. Neural Comput 13(5):1103–1118
Van Beers WC, Kleijnen JP (2008) Customized sequential designs for random simulation experiments: Kriging metamodeling and bootstrapping. Eur J Oper Res 186(3):1099–1113
Williams CK, Rasmussen CE (2006) Gaussian processes for machine learning, vol 2. vol 3. MIT Press Cambridge, MA
Zhang J, Taflanidis AA, Nadal-Caraballo NC, Melby JA, Diop F (2018) Advances in surrogate modeling for storm surge prediction: storm selection and addressing characteristics related to climate change. Nat Hazards 94(3):1225–1253
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
The authors declare that they have no conflict of interest..
Replication of results
Numerical simulations were performed in MATLAB 2017b. Calibration of the surrogate model hyper-parameters was performed utilizing DACE toolbox (Lophaven et al. 2002). Numerical codes for the DoE optimization, described in detail in Section 3.2 of the manuscript, can be obtained from the corresponding author with reasonable request.
Additional information
Responsible Editor: Shapour Azarm
Publisher’s note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Kyprioti, A.P., Zhang, J. & Taflanidis, A.A. Adaptive design of experiments for global Kriging metamodeling through cross-validation information. Struct Multidisc Optim 62, 1135–1157 (2020). https://doi.org/10.1007/s00158-020-02543-1
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00158-020-02543-1