AStA Advances in Statistical Analysis

, Volume 94, Issue 4, pp 311–324 | Cite as

Computer experiments: a review

  • Sigal LevyEmail author
  • David M. Steinberg
Original Paper


In this paper we provide a broad introduction to the topic of computer experiments. We begin by briefly presenting a number of applications with different types of output or different goals. We then review modelling strategies, including the popular Gaussian process approach, as well as variations and modifications. Other strategies that are reviewed are based on polynomial regression, non-parametric regression and smoothing spline ANOVA. The issue of multi-level models, which combine simulators of different resolution in the same experiment, is also addressed. Special attention is given to modelling techniques that are suitable for functional data. To conclude the modelling section, we discuss calibration, validation and verification. We then review design strategies including Latin hypercube designs and space-filling designs and their adaptation to computer experiments. We comment on a number of special issues, such as designs for multi-level simulators, nested factors and determination of experiment size.


Gaussian process Latin hypercube designs Deterministic output Functional data Space filling 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. Allen, T., Bernshtyn, M.A., Kabiri-Bamoradian, K.: Constructing metamodels for computer experiments. J. Qual. Technol. 35(3), 264–274 (2003) Google Scholar
  2. Bastos, L.S., O’Hagan, A.: Diagnostics for Gaussian process emulators. Technometrics 51, 425–438 (2009) CrossRefGoogle Scholar
  3. Bates, R.A., Buck, R.J., Riccomagno, E., Wynn, H.P.: Experimental design and observation for large systems. J. R. Stat. Soc. B 58, 77–111 (1996) zbMATHMathSciNetGoogle Scholar
  4. Bayarri, M.J., Berger, J.O., Cafeo, J., Garcia-Donato, G., Liu, F., Palomo, Parthasarathy, R.J., Paulo, R., Sacks, J., Walsh, D.: Computer model validation with functional output. Ann. Stat. 35(5), 1874–1906 (2007a) zbMATHCrossRefMathSciNetGoogle Scholar
  5. Bayarri, M.J., Berger, J.O., Paulo, R., Sacks, J., Cafeo, J.A., Cavendish, J., Lin, C.H., Tu, J.: A framework for validation of computer models. Technometrics 49(2), 138–154 (2007b) CrossRefMathSciNetGoogle Scholar
  6. Bayarri, M.J., Berger, J.O., Calder, E.S., Dalbey, K., Lunagomes, S., Patra, A.K., Pitman, E.B., Spiller, E.T., Wolpert, R.L.: Using statistical and computer models to quantify volcanic hazards. Technometrics 51, 402–413 (2009a) CrossRefGoogle Scholar
  7. Bayarri, M.J., Berger, J.O., Kennedy, M.C., Kottas, A., Paulo, R., Sacks, J., Cafeo, J.A., Lin, C.H., Tu, J.: Predicting vehicle crashworthiness: validation of computer models for functional and hierarchical data. J. Am. Stat. Assoc. 104, 929–943 (2009b) CrossRefGoogle Scholar
  8. Bingham, D., Sitter, R.R., Tang, B.: Orthogonal and nearly orthogonal designs for computer experiments. Biometrika 96, 51–65 (2009) zbMATHCrossRefMathSciNetGoogle Scholar
  9. Bursztyn, D., Steinberg, D.M.: Comparison of designs for computer experiments. J. Stat. Plann. Inference 136, 1103–1119 (2006) zbMATHCrossRefMathSciNetGoogle Scholar
  10. Chen, V.C.P., Tsui, K.L., Barton, R.R., Meckesheimer, M.: A review on design modeling and applications of computer experiments. IIE Trans. 38(4), 273–291 (2006) CrossRefGoogle Scholar
  11. Cheng, R.C.H., Kleijnen, J.P.C.: Improved design of queueing simulation experiments with highly heteroscedastic responses. Oper. Res. 47, 762–777 (1999) zbMATHCrossRefGoogle Scholar
  12. Craig, P.S., Goldstein, M., Rougier, J.C., Seheult, A.H.: Bayesian forecasting for complex systems using computer simulators. J. Am. Stat. Assoc. 96, 717–729 (2001) zbMATHCrossRefMathSciNetGoogle Scholar
  13. Cressie, N.: Statistics for Spatial Data. Wiley, New York (1993) Google Scholar
  14. Cumming, J.A., Goldstein, M.: Small sample Bayesian designs for complex high-dimensional models based on information gained using fast approximations. Technometrics 51, 377–388 (2009) CrossRefGoogle Scholar
  15. Currin, C., Mitchell, T., Morris, M., Ylvisaker, D.: Bayesian prediction of deterministic functions, with application to the design and analysis of computer experiments. J. Am. Stat. Assoc. 86, 953–963 (1991) CrossRefMathSciNetGoogle Scholar
  16. Da Veiga, S., Wahl, F., Gamboa, F.: Local polynomial estimation for sensitivity analysis on models with correlated inputs. Technometrics 51, 452–463 (2009) CrossRefGoogle Scholar
  17. Drignei, D.: Empirical Bayesian analysis for high-dimensional computer output. Technometrics 48(2), 230–240 (2006) CrossRefMathSciNetGoogle Scholar
  18. Fang, K.T., Hickernell, F.J.: Uniform experimental designs. In: Ruggeri, F., Kenett, R., Faltin, F.W. (eds.) Encyclopedia of Quality and Reliability, vol. 3, pp. 2037–2040. Wiley, New York (2007) Google Scholar
  19. Fang, K.T., Li, R., Sudjianto, A.: Design and Modeling for Computer Experiments. Chapman & Hall/CRC, Boca Raton (2006) zbMATHGoogle Scholar
  20. Fang, K.T., Lin, D.K.J., Winker, P., Zhang, Y.: Uniform design: theory and application. Technometrics 42(3), 237–248 (2000) zbMATHCrossRefMathSciNetGoogle Scholar
  21. Faraway, J.J.: Regression analysis for a functional response. Technometrics 39(3), 254–261 (1997) zbMATHCrossRefMathSciNetGoogle Scholar
  22. Friedman, J.H.: Multivariate adaptive regression splines. Ann. Stat. 19, 1–141 (1991) zbMATHCrossRefGoogle Scholar
  23. Friedman, J.H., Stuetzle, W.: Projection pursuit regression. J. Am. Stat. Assoc. 76, 817–823 (1981) CrossRefMathSciNetGoogle Scholar
  24. Goldstein, M., Rougier, J.C.: Bayes linear calibrated prediction for complex systems. J. Am. Stat. Assoc. 101, 1132–1143 (2006) zbMATHCrossRefMathSciNetGoogle Scholar
  25. Gramacy, R.B., Lee, H.K.H.: Bayesian treed Gaussian process models with an application to computer modeling. J. Am. Stat. Assoc. 103, 1119–1130 (2008) CrossRefMathSciNetGoogle Scholar
  26. Halton, J.H.: On the efficiency of certain quasi-random sequences of points in evaluating multi-dimensional integrals. Numer. Math. 2, 84–90 (1960) CrossRefMathSciNetGoogle Scholar
  27. Han, G., Santner, T.J., Notz, W.I., Bartel, D.L.: Prediction for computer experiments having quantitative and qualitative input variables. Technometrics 51, 278–288 (2009a) CrossRefGoogle Scholar
  28. Han, G., Santner, T.J., Rawlinson, J.J.: Simultaneous determination of tuning and calibration parameters for computer experiments. Technometrics 51, 464–474 (2009b) CrossRefGoogle Scholar
  29. Higdon, D., Gattiker, J., Williams, B., Rightley, M.: Computer model calibration using high dimensional output. Technical report, Los Alamos National Laboratory (2007) Google Scholar
  30. Hung, Y., Joseph, V.R., Melkote, S.N.: Design and analysis of computer experiments with branching and nested factors. Technometrics 51, 354–365 (2009) CrossRefMathSciNetGoogle Scholar
  31. Johnson, M.E., Moore, I.M., Ylvisaker, D.: Minimax and maximin distance designs. J. Stat. Plann. Inference 26, 131–148 (1990) CrossRefMathSciNetGoogle Scholar
  32. Jones, B., Johnson, R.T.: Design and analysis for the Gaussian process model. Qual. Reliab. Eng. Int. 25, 515–550 (2009) CrossRefGoogle Scholar
  33. Joseph, R.V., Hung, Y., Sudjianto, A.: Blind kriging: a new method for developing metamodels. J. Mech. Des. 130(3), 1–8 (2008) CrossRefGoogle Scholar
  34. Jourdan, A., Franco, J.: Optimal Latin hypercube designs for the Kullback–Leibler criterion. Adv. Stat. Anal. (2010). doi: 10.1007/s10182-010-0145-y Google Scholar
  35. Kennedy, M.C., O’Hagan, A.: Predicting the output from a complex computer code when fast approximations are available. Biometrika 87(1), 1–13 (2000) zbMATHCrossRefMathSciNetGoogle Scholar
  36. Lefebvre, S., Roblin, A., Varet, S., Durand, G.: Metamodeling of aircraft infrared signature dispersion. Adv. Stat. Anal. (2010). doi: 10.1007/s10182-010-0146-x Google Scholar
  37. Lemieux, C.: Monte Carlo and Quasi-Monte Carlo sampling. Springer, New York (2009) zbMATHGoogle Scholar
  38. Levy, S.: The analysis of time dependent computer experiments. Unpublished Ph.D. dissertation, Tel-Aviv University (2008) Google Scholar
  39. Lin, C.D., Mukerjee, R., Tang, B.: Construction of orthogonal and nearly orthogonal Latin hypercubes. Biometrika 96, 243–247 (2009) zbMATHCrossRefMathSciNetGoogle Scholar
  40. Linkletter, C., Bingham, D., Hengartner, N., Higdon, D., Ye, K.Q.: Variable selection for Gaussian process models in computer experiments. Technometrics 48, 478–490 (2006) CrossRefMathSciNetGoogle Scholar
  41. Loeppky, J.L., Sacks, J., Welch, W.J.: Choosing the sample size of a computer experiment: a practical guide. Technometrics 51, 366–376 (2009) CrossRefGoogle Scholar
  42. Matheron, G.: Principals of geostatistics. Econ. Geol. 58, 1246–1266 (1963) CrossRefGoogle Scholar
  43. McKay, M.D., Beckman, R.J., Conover, W.J.: A comparison of three methods for selecting values of input variables in the analysis of output from a computer code. Technometrics 21, 239–245 (1979) zbMATHCrossRefMathSciNetGoogle Scholar
  44. Molina, G., Bayarri, M.J., Berger, J.O.: Statistical inverse analysis for a network micosimulator. Technometrics 47(4), 388–398 (2005) CrossRefMathSciNetGoogle Scholar
  45. Neumann Ben-Ari, E., Steinberg, D.M.: Modeling data from computer experiments: an empirical comparison of kriging with MARS and projection pursuit regression. Qual. Eng. 19, 327–338 (2007) CrossRefGoogle Scholar
  46. Niederreiter, H.: Random Number Generation and Quasi-Monte Carlo Methods. Society for Industrial and Applied Mathematics, Philadelphia (1992) zbMATHGoogle Scholar
  47. O’Hagan, A.: Discussion of Sacks, Welch, Mitchell and Wynn. Stat. Sci. 4, 430–432 (1989) CrossRefGoogle Scholar
  48. Owen, A.B.: Orthogonal arrays for computer experiments, integration and visualization. Statist. Sinica 2, 439–52 (1992) zbMATHMathSciNetGoogle Scholar
  49. Petelet, M., Iooss, B., Asserin, O., Loredo, A.: Latin hypercube sampling with inequality constraints. Adv. Stat. Anal. (2010). doi: 10.1007/s10182-010-0144-z Google Scholar
  50. Pistone, G., Vicario, G.: Comparing and generating Latin hypercube designs in kriging models. Adv. Stat. Anal. (2010). doi: 10.1007/s10182-010-0142-1 Google Scholar
  51. Qian, P.: Nested Latin hypercube designs. Biometrika 96(4), 957–970 (2009) zbMATHCrossRefGoogle Scholar
  52. Qian, P., Wu, C.F.J.: Bayesian hierarchical modeling for integrating low-accuracy and high-accuracy experiments. Technometrics 50, 192–204 (2008) CrossRefMathSciNetGoogle Scholar
  53. Qian, P., Wu, H., Wu, C.F.J.: Gaussian process models for computer experiments with qualitative and quantitative factors. Technometrics 50, 383–396 (2008) CrossRefMathSciNetGoogle Scholar
  54. Ramsay, J.O., Silverman, B.W.: Functional Data Analysis, 2nd edn. Springer, New York (2005) Google Scholar
  55. Ratto, M., Pagano, A.: Using recursive algorithms for efficient identification of smoothing spline ANOVA models. Adv. Stat. Anal. (2010). doi: 10.1007/s10182-010-0148-8 zbMATHGoogle Scholar
  56. Reich, B.J., Storlie, C.B., Bondell, H.D.: Variable selection in Bayesian smoothing spline ANOVA models: application to deterministic computer codes. Technometrics 51, 110–120 (2009) CrossRefGoogle Scholar
  57. Roache, P.J.: Verification of codes and calculations. AIAA J. 36(5), 696–702 (1998) CrossRefGoogle Scholar
  58. Rougier, J.: Efficient emulators for multivariate deterministic functions. J. Comput. Graph Stat. 17, 827–843 (2007) MathSciNetGoogle Scholar
  59. Sacks, J., Welch, W.J., Mitchell, T.J., Wynn, H.P.: Design and analysis of computer experiments. Stat. Sci. 4, 409–435 (1989) zbMATHCrossRefMathSciNetGoogle Scholar
  60. Saltelli, A., Ratto, M., Andres, T., Campolongo, F., Cariboni, J., Gatelli, D., Saisana, M., Tarantola, S.: Global Sensitivity Analysis: The Primer. Wiley, New York (2008) zbMATHGoogle Scholar
  61. Santner, T.J., Williams, B.J., Notz, W.I.: The Design and Analysis of Computer Experiments. Springer, New York (2003) zbMATHGoogle Scholar
  62. Sargent, R.G.: Verification and validation of simulation models. In: Mason, S.J., Hill, R.R., Mönch, L., Rose, O., Jefferson, T., Fowler, J.W. (eds.) Proceedings of the 2008 Winter Simulation Conference, pp. 157–169 (2008) CrossRefGoogle Scholar
  63. Schlesinger, S., Crosbie, R.E., Gagné, R.E., Innis, G.S., Lalwani, C.S., Loch, J., Sylvester, R.J., Wright, R.D., Kheir, N., Bartos, D.: Terminology for model credibility. Simulation 32, 103–104 (1975) Google Scholar
  64. Shewry, M.C., Wynn, H.P.: Maximum entropy sampling. J. Appl. Stat. 14, 165–170 (1987) CrossRefGoogle Scholar
  65. Sobol’, I.M.: On the distribution of points in a cube and the approximate evaluation of integrals. USSR Comput. Math. Math. Phys. 7, 86–112 (1967) CrossRefMathSciNetGoogle Scholar
  66. Stein, M.L.: Discussion of Sacks, Welch, Mitchell and Wynn. Stat. Sci. 4, 432–433 (1989) CrossRefGoogle Scholar
  67. Steinberg, D.M., Bursztyn, D.: Data analytic tools for understanding random field regression models. Technometrics 46, 411–420 (2004) CrossRefMathSciNetGoogle Scholar
  68. Steinberg, D.M., Lin, D.K.J.: A construction method for orthogonal Latin hypercube designs. Biometrika 93, 279–288 (2006) zbMATHCrossRefMathSciNetGoogle Scholar
  69. Taddy, M.A., Lee, H.K.H., Gray, G.A., Griffin, J.D.: Bayesian guided pattern search for robust local optimization. Technometrics 51, 389–401 (2009) CrossRefGoogle Scholar
  70. Tang, B.: Orthogonal array-based Latin hypercubes. J. Am. Stat. Assoc. 88, 1392–1397 (1993) zbMATHCrossRefGoogle Scholar
  71. Wagner, T., Bröcker, C., Saba, N., Biermann, D., Matzenmiller, A., Steinhoff, K.: Modelling of a thermomechanically coupled forming process based on functional outputs from a finite element analysis and from experimental measurements. Adv. Stat. Anal. (2010). doi: 10.1007/s10182-010-0149-7 Google Scholar
  72. Wang, S., Chen, W., Tsui, K.-L.: Bayesian validation of computer models. Technometrics 51, 439–451 (2009) CrossRefGoogle Scholar
  73. Welch, W.J., Buck, R.J., Sacks, J., Wynn, H.P., Mitchell, T.J., Morris, M.D.: Screening, predicting and computer experiments. Technometrics 34, 15–25 (1992) CrossRefGoogle Scholar
  74. Young, P.C.: The identification and estimation of nonlinear stochastic systems. In: Mees, A.I. (ed.) Nonlinear Dynamics and Statistics. Birkhauser, Boston (2001) Google Scholar

Copyright information

© Springer-Verlag 2010

Authors and Affiliations

  1. 1.Department of Statistics and Operations ResearchTel Aviv UniversityTel AvivIsrael

Personalised recommendations