Abstract
Metamodels (or emulators) are statistical tools for the analysis of large complex simulation models. They consist of a Gaussian (or second order) process (kriging) fitted to a designed set of simulator runs. Once an emulator has been built, it is important that it is validated against some independent runs of the simulator. This paper considers the design of experiments for the validation of the fitted metamodel. All the proposed designs are based on maximin Latin hypercubes and add an extra criterion to be optimised based on the distances between the points in the validation and original designs. Simulation experiments are carried out to determine how well each design performs against the alternative criteria.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Andrianakis I and Challenor PG (2012). The effect of the nugget on Gaussian process emulators of computer models. Computational Statistics & Data Analysis 56 (12): 4215–4228.
Ankenman B, Nelson BL and Staum J (2010). Stochastic kriging for simulation metamodeling. Operations Research 58 (2): 371–382.
Bastos LS and O’Hagan A (2009). Diagnostics for Gaussian process emulators. Technometrics 51 (4): 425–438.
Bayarri MJ et al (2007). A framework for validation of computer models. Technometrics 49 (2): 138–154.
Challenor P, McNeall D and Gattiker J (2010). Assessing the probability of rare climate events. In: O’Hagan A and West M (eds.) The Oxford Handbook of Applied Bayesian Analysis, Chapter 16, Oxford University Press, Oxford.
Cioppa TM and Lucas TW (2007). Efficient nearly orthogonal and space-filling Latin hypercubes. Technometrics 49 (1): 45–55.
Craig PS, Goldstein M, Rougier JC and Seheult AH (2001). Bayesian forecasting for complex systems using computer simulators. Journal of the American Statistical Association 96 (454): 717–729.
Draper D (1995). Assessment and propagation of model uncertainty. Journal of the Royal Statistical Society: Series B (Methodological) 57 (1): 45–97.
Forrester A, Sobester A and Keane A (2008). Engineering Design Via Surrogate Modelling: A Practical Guide. Wiley, Chichester.
Gramacy RB and Lee HKH (2010). Cases for the nugget in modeling computer experiments. Statistics and Computing 22 (3): 713–722.
Helton JC and Davis FJ (2003). Latin hypercube sampling and the propagation of uncertainty in analyses of complex systems. Reliability Engineering and System Safety 81 (1): 23–69.
Kennedy MC and O’Hagan A (2001). Bayesian calibration of computer models. Journal of the Royal Statistical Society: B 63 (3): 425–464.
Kleijnen JP (1992). Regression metamodels for simulation with common random numbers: Comparison of validation tests and confidence intervals. Management Science 38 (8): 1164–1185.
Kleijnen J and Sargent R (2000). A methodology for fitting and validating metamodels in simulation. European Journal of Operational Research 120 (1): 14–29.
Loeppky JL, Sacks J and Welch WJ (2009). Choosing the sample size of a computer experiment: A practical guide. Technometrics 51 (4): 366–376.
McKay MD, Beckman RJ and Conover WJ (1979). A comparison of three methods for selecting values of input variables in the analysis of output from a computer code. Technometrics 21 (2): 239–245.
Oakley JE and O’Hagan A (2002). Bayesian inference for the uncertainty distribution of computer model outputs. Biometrika 89 (4): 769–784.
Oakley JE and O’Hagan A (2004). Probabilistic sensitivity analysis of complex models: A Bayesian approach. Journal of the Royal Statistical Society Series B 66 (3): 751–769.
O’Hagan A (2006). Bayesian analysis of computer code output: A tutorial. Reliability Engineering and System Safety 91 (10–11): 1290–1300.
Rasmussen CE and Williams CKI (2006). Gaussian Processes for Machine Learning. MIT Press, Cambridge.
Sacks J, Welch WJ, Mitchell TJ and Wynn HP (1989). Design and analysis of computer experiments. Statistical Science: A Review Journal of the Institute of Mathematical Statistics 4 (4): 409–435.
Santner TJ, Williams BJ and Notz WI (2003). The Design and Analysis of Computer Experiments. Springer, New York.
Sobol L (1967). On the distribution of points in a cube and the approximate evaluation of on the distribution of points in a cube and the approximate evaluation of integrals. USSR Computational Mathematics and Mathematical Physics 7 (4): 86–112.
Tako AA and Robinson S (2008). Comparing discrete-event simulation and system dynamics: Users’ perceptions. Journal of the Operational Research Society 60 (3): 296–312.
Tang B (1993). Orthogonal array-based Latin hypercubes. Journal of the American Statistical Association 88 (424): 1392–1397.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Challenor, P. Experimental design for the validation of kriging metamodels in computer experiments. J Simulation 7, 290–296 (2013). https://doi.org/10.1057/jos.2013.17
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1057/jos.2013.17