Statistics and Computing

, Volume 22, Issue 3, pp 681–701 | Cite as

Design of computer experiments: space filling and beyond

  • Luc PronzatoEmail author
  • Werner G. Müller


When setting up a computer experiment, it has become a standard practice to select the inputs spread out uniformly across the available space. These so-called space-filling designs are now ubiquitous in corresponding publications and conferences. The statistical folklore is that such designs have superior properties when it comes to prediction and estimation of emulator functions. In this paper we want to review the circumstances under which this superiority holds, provide some new arguments and clarify the motives to go beyond space-filling. An overview over the state of the art of space-filling is introducing and complementing these results.


Kriging Entropy Design of experiments Space-filling Sphere packing Maximin design Minimax design 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. Abt, M.: Estimating the prediction mean squared error in Gaussian stochastic processes with exponential correlation structure. Scand. J. Stat. 26(4), 563–578 (1999) MathSciNetzbMATHCrossRefGoogle Scholar
  2. Angelis, L., Senta, E.B., Moyssiadis, C.: Optimal exact experimental designs with correlated errors through a simulated annealing algorithm. Comput. Stat. Data Anal. 37(3), 275–296 (2001) zbMATHCrossRefGoogle Scholar
  3. Ash, R.: Information Theory. Wiley, New York (1965) (republished by Dover, New York, 1990) zbMATHGoogle Scholar
  4. Audze, P., Eglais, V.: New approach for planning out experiments. Probl. Dyn. Strengths 35, 104–107 (1977) Google Scholar
  5. Ball, K.: Eigenvalues of Euclidean distance matrices. J. Approx. Theory 68, 74–82 (1992) MathSciNetzbMATHCrossRefGoogle Scholar
  6. Bates, R.A., Buck, R.J., Riccomagno, E., Wynn, H.P.: Experimental design and observation for large systems. J. R. Stat. Soc. B 58(1), 77–94 (1996) MathSciNetzbMATHGoogle Scholar
  7. Beirlant, J., Dudewicz, E., Györfi, L., van der Meulen, E.: Nonparametric entropy estimation; an overview. Int. J. Math. Stat. Sci. 6(1), 17–39 (1997) MathSciNetzbMATHGoogle Scholar
  8. Bellhouse, D.R., Herzberg, A.M.: Equally spaced design points in polynomial regression: a comparison of systematic sampling methods with the optimal design of experiments. Can. J. Stat. 12(2), 77–90 (1984) MathSciNetzbMATHCrossRefGoogle Scholar
  9. Bettinger, R., Duchêne, P., Pronzato, L., Thierry, E.: Design of experiments for response diversity. In: Proc. 6th International Conference on Inverse Problems in Engineering (ICIPE). J. Phys. Conf. Ser., Dourdan (Paris) (2008) Google Scholar
  10. Bettinger, R., Duchêne, P., Pronzato, L.: A sequential design method for the inversion of an unknown system. In: Proc. 15th IFAC Symposium on System Identification, Saint-Malo, France, pp. 1298–1303 (2009) Google Scholar
  11. Bischoff, W., Miller, F.: Optimal designs which are efficient for lack of fit tests. Ann. Stat. 34(4), 2015–2025 (2006) MathSciNetCrossRefGoogle Scholar
  12. Boissonnat, J.D., Yvinec, M.: Algorithmic Geometry. Cambridge University Press, Cambridge (1998) zbMATHGoogle Scholar
  13. Bursztyn, D., Steinberg, D.: Comparison of designs for computer experiments. J. Stat. Plan. Inference 136(3), 1103–1119 (2006) MathSciNetzbMATHCrossRefGoogle Scholar
  14. Carnell, R.: lhs: latin hypercube samples. R package version 0.5 (2009) Google Scholar
  15. Chen, V.C.P., Tsui, K.L., Barton, R.R., Meckesheimer, M.: A review on design, modeling and applications of computer experiments. AIIE Trans. 38(4), 273–291 (2006) Google Scholar
  16. Cignoni, P., Montani, C., Scopigno, R.: DeWall: a fast divide and conquer Delaunay triangulation algorithm in E d. Comput. Aided Des. 30(5), 333–341 (1998) zbMATHCrossRefGoogle Scholar
  17. Cortés, J., Bullo, F.: Nonsmooth coordination and geometric optimization via distributed dynamical systems. SIAM Rev. 51(1), 163–189 (2009) MathSciNetzbMATHCrossRefGoogle Scholar
  18. Cressie, N.: Statistics for Spatial Data. Wiley Series in Probability and Statistics. Wiley-Interscience, New York (1993). Rev. Sub. Ed. Google Scholar
  19. den Hertog, D., Kleijnen, J.P.C., Siem, A.Y.D.: The correct kriging variance estimated by bootstrapping. J. Oper. Res. Soc. 57(4), 400–409 (2006) zbMATHCrossRefGoogle Scholar
  20. Dette, H., Pepelyshev, A.: Generalized Latin hypercube design for computer experiments. Technometrics 52(4), 421–429 (2010) MathSciNetCrossRefGoogle Scholar
  21. Dette, H., Kunert, J., Pepelyshev, A.: Exact optimal designs for weighted least squares analysis with correlated errors. Stat. Sin. 18(1), 135–154 (2008) MathSciNetzbMATHGoogle Scholar
  22. Fang, K.T.: The uniform design: application of number theoretic methods in experimental design. Acta Math. Appl. Sin. 3, 363–372 (1980) zbMATHGoogle Scholar
  23. Fang, K.T., Li, R.: Uniform design for computer experiments and its optimal properties. Int. J. Mater. Prod. Technol. 25(1), 198–210 (2006) Google Scholar
  24. Fang, K.T., Wang, Y.: Number-Theoretic Methods in Statistics, 1st edn. Chapman & Hall/CRC Monographs on Statistics & Applied Probability. Chapman & Hall/CRC, London/Boca Raton (1993) Google Scholar
  25. Fang, K.T., Lin, D.K.J., Winker, P., Zhang, Y.: Uniform design: Theory and application. Technometrics 42(3), 237–248 (2000) MathSciNetzbMATHCrossRefGoogle Scholar
  26. Fang, K.T., Li, R., Sudjianto, A.: Design and Modeling for Computer Experiments. Chapman & Hall/CRC, London/Boca Raton (2005) CrossRefGoogle Scholar
  27. Fedorov, V.: Theory of Optimal Experiments. Academic Press, New York (1972) Google Scholar
  28. Fedorov, V.V., Hackl, P.: Model-Oriented Design of Experiments. Lecture Notes in Statistics, vol. 125. Springer, Berlin (1997) zbMATHCrossRefGoogle Scholar
  29. Franco, J.: Planification d’expériences numériques en phase exploratoire pour la simulation de phénomènes complexes. Ph.D. thesis, École Nationale Supérieure des Mines de Saint Etienne (2008) Google Scholar
  30. Franco, J., Bay, X., Corre, B., Dupuy, D.: Planification d’expériences numériques à partir du processus ponctuel de Strauss. Preprint, Département 3MI, École Nationale Supérieure des Mines de Saint-Etienne. (2008)
  31. Franco, J., Vasseur, O., Corre, B., Sergent, M.: Minimum Spanning Tree: a new approach to assess the quality of the design of computer experiments. Chemom. Intell. Lab. Syst. 97, 164–169 (2009) CrossRefGoogle Scholar
  32. Gensane, T.: Dense packings of equal spheres in a cube. Electron. J. Comb. 11 (2004) Google Scholar
  33. Glover, F., Kelly, J., Laguna, M.: Genetic algorithms and tabu search: hybrids for optimization. Comput. Oper. Res. 22(1), 111–134 (1995) zbMATHCrossRefGoogle Scholar
  34. Gramacy, R., Lee, H.: Cases for the nugget in modeling computer experiments. Tech. rep., (2010)
  35. Gramacy, R.B., Lee, H.K.: Adaptive design and analysis of supercomputer experiments. Technometrics 51(2), 130–144 (2009) MathSciNetCrossRefGoogle Scholar
  36. Griffith, D.: Spatial Autocorrelation and Spatial Filtering: Gaining Understanding through Theory and Scientific Visualization. Springer, Berlin (2003) Google Scholar
  37. Hall, P., Morton, S.: On the estimation of entropy. Ann. Inst. Stat. Math. 45(1), 69–88 (1993) MathSciNetzbMATHCrossRefGoogle Scholar
  38. Harville, D.A., Jeske, D.R.: Mean squared error of estimation or prediction under a general linear model. J. Am. Stat. Assoc. 87(419), 724–731 (1992) MathSciNetzbMATHCrossRefGoogle Scholar
  39. Havrda, M., Charvát, F.: Quantification method of classification processes: concept of structural α-entropy. Kybernetika 3, 30–35 (1967) MathSciNetzbMATHGoogle Scholar
  40. Herzberg, A.M., Huda, S.: A comparison of equally spaced designs with different correlation structures in one and more dimensions. Can. J. Stat. 9(2), 203–208 (1981) MathSciNetzbMATHCrossRefGoogle Scholar
  41. Hoeting, J.A., Davis, R.A., Merton, A.A., Thompson, S.E.: Model selection for geostatistical models. Ecol. Appl. 16(1), 87–98 (2006) CrossRefGoogle Scholar
  42. Husslage, B., Rennen, G., van Dam, E., den Hertog, D.: Space-filling Latin hypercube designs for computer experiments. Discussion paper 2006-18, Tilburg University, Center for Economic Research (2006) Google Scholar
  43. Iooss, B., Boussouf, L., Feuillard, V., Marrel, A.: Numerical studies of the metamodel fitting and validation processes. International Journal on Advances in Systems and Measurements 3(1–2), 11–21 (2010) Google Scholar
  44. Irvine, K., Gitelman, A., Hoeting, J.: Spatial designs and properties of spatial correlation: effects on covariance estimation. J. Agric. Biol. Environ. Stat. 12(4), 450–469 (2007) MathSciNetCrossRefGoogle Scholar
  45. Jin, R., Chen, W., Sudjianto, A.: An efficient algorithm for constructing optimal design of computer experiments. J. Stat. Plan. Inference 134(1), 268–287 (2005) MathSciNetzbMATHCrossRefGoogle Scholar
  46. Johnson, M., Moore, L., Ylvisaker, D.: Minimax and maximin distance designs. J. Stat. Plan. Inference 26, 131–148 (1990) MathSciNetCrossRefGoogle Scholar
  47. Johnson, R.T., Montgomery, D.C., Jones, B., Fowler, J.W.: Comparing designs for computer simulation experiments. In: WSC ’08: Proceedings of the 40th Conference on Winter Simulation, pp. 463–470 (2008) Google Scholar
  48. Joseph, V.: Limit kriging. Technometrics 48(4), 458–466 (2006) MathSciNetCrossRefGoogle Scholar
  49. Jourdan, A., Franco, J.: Optimal Latin hypercube designs for the Kullback-Leibler criterion. AStA Adv. Stat. Anal. 94, 341–351 (2010) MathSciNetCrossRefGoogle Scholar
  50. Kiefer, J., Wolfowitz, J.: The equivalence of two extremum problems. Can. J. Math. 12, 363–366 (1960) MathSciNetzbMATHCrossRefGoogle Scholar
  51. Kiseľák, J., Stehlík, M.: Equidistant and d-optimal designs for parameters of Ornstein–Uhlenbeck process. Stat. Probab. Lett. 78(12), 1388–1396 (2008) zbMATHCrossRefGoogle Scholar
  52. Kleijnen, J.P.C.: Design and Analysis of Simulation Experiments. Springer, New York (2009) Google Scholar
  53. Koehler, J., Owen, A.: Computer experiments. In: Ghosh, S., Rao, C.R. (eds.) Handbook of Statistics: Design and Analysis of Experiments, vol. 13, pp. 261–308. North-Holland, Amsterdam (1996) Google Scholar
  54. Kozachenko, L., Leonenko, N.: On statistical estimation of entropy of random vector. Probl. Inf. Transm. 23(2), 95–101 (1987) (translated from Problemy Peredachi Informatsii, in Russian, vol. 23, No. 2, pp. 9–16, 1987) MathSciNetzbMATHGoogle Scholar
  55. Leary, S., Bhaskar, A., Keane, A.: Optimal orthogonal-array-based Latin hypercubes. J. Appl. Stat. 30(5), 585–598 (2003) MathSciNetzbMATHCrossRefGoogle Scholar
  56. Leonenko, N., Pronzato, L., Savani, V.: A class of Rényi information estimators for multidimensional densities. Ann. Stat. 36(5), 2153–2182 (2008) (correction in Ann. Stat. 38(6), 3837–3838, 2010) MathSciNetzbMATHCrossRefGoogle Scholar
  57. Li, X.S., Fang, S.C.: On the entropic regularization method for solving min-max problems with applications. Math. Methods Oper. Res. 46, 119–130 (1997) MathSciNetzbMATHCrossRefGoogle Scholar
  58. McKay, M., Beckman, R., Conover, W.: A comparison of three methods for selecting values of input variables in the analysis of output from a computer code. Technometrics 21(2), 239–245 (1979) MathSciNetzbMATHCrossRefGoogle Scholar
  59. Melissen, H.: Packing and covering with circles. Ph.D. thesis, University of Utrecht (1997) Google Scholar
  60. Mitchell, T.: An algorithm for the construction of “D-optimal” experimental designs. Technometrics 16, 203–210 (1974) MathSciNetzbMATHCrossRefGoogle Scholar
  61. Morris, M., Mitchell, T.: Exploratory designs for computational experiments. J. Stat. Plan. Inference 43, 381–402 (1995) zbMATHCrossRefGoogle Scholar
  62. Müller, W.G.: Collecting Spatial Data: Optimum Design of Experiments for Random Fields, 3rd edn. Springer, Heidelberg (2007) zbMATHGoogle Scholar
  63. Müller, W.G., Stehlík, M.: Compound optimal spatial designs. Environmetrics 21(3–4), 354–364 (2010) MathSciNetGoogle Scholar
  64. Müller, W.G., Pronzato, L., Waldl, H.: Relations between designs for prediction and estimation in random fields: an illustrative case. In: E. Porcu, J.M. Montero, M. Schlather (eds.) Advances and Challenges in Space-Time Modelling of natural Events. Springer Lecture Notes in Statistics (2011, to appear) Google Scholar
  65. Nagy, B., Loeppky, J.L., Welch, W.J.: Fast Bayesian inference for Gaussian process models. Tech. rep., The University of British Columbia, Department of Statistics (2007) Google Scholar
  66. Narcowich, F.: Norms of inverses and condition numbers for matrices associated with scattered data. J. Approx. Theory 64, 69–94 (1991) MathSciNetzbMATHCrossRefGoogle Scholar
  67. Niederreiter, H.: Random Number Generation and Quasi-Monte Carlo Methods (CBMS-NSF Regional Conference Series in Applied Mathematics). SIAM, Philadelphia (1992) CrossRefGoogle Scholar
  68. Okabe, A., Books, B., Sugihama, K.: Spatial Tessellations. Concepts and Applications of Voronoi Diagrams. Wiley, New York (1992) zbMATHGoogle Scholar
  69. Oler, N.: A finite packing problem. Can. Math. Bull. 4, 153–155 (1961) MathSciNetzbMATHCrossRefGoogle Scholar
  70. Pebesma, E.J., Heuvelink, G.B.M.: Latin hypercube sampling of Gaussian random fields. Technometrics 41(4), 303–312 (1999) CrossRefGoogle Scholar
  71. Penrose, M., Yukich, J.: Laws of large numbers and nearest neighbor distances. In: Wells, M., Sengupta, A. (eds.) Advances in Directional and Linear Statistics. A Festschrift for Sreenivasa Rao Jammalamadaka, Chap. 13, pp. 189–199 (2011). arXiv:0911.0331v1
  72. Petelet, M., Iooss, B., Asserin, O., Loredo, A.: Latin hypercube sampling with inequality constraints. AStA Adv. Stat. Anal. 94, 325–339 (2010) MathSciNetCrossRefGoogle Scholar
  73. Picheny, V., Ginsbourger, D., Roustant, O., Haftka, R.T., Kim, N.H.: Adaptive designs of experiments for accurate approximation of a target region. J. Mech. Des. 132(7), 071,008 (2010) CrossRefGoogle Scholar
  74. Pistone, G., Vicario, G.: Comparing and generating Latin hypercube designs in Kriging models. AStA Adv. Stat. Anal. 94, 353–366 (2010) MathSciNetCrossRefGoogle Scholar
  75. Pronzato, L.: Optimal experimental design and some related control problems. Automatica 44(2), 303–325 (2008) MathSciNetCrossRefGoogle Scholar
  76. Putter, H., Young, A.: On the effect of covariance function estimation on the accuracy of kriging predictors. Bernoulli 7(3), 421–438 (2001) MathSciNetzbMATHCrossRefGoogle Scholar
  77. Qian, P.Z.G., Ai, M., Wu, C.F.J.: Construction of nested space-filling designs. Ann. Stat. 37(6A), 3616–3643 (2009) MathSciNetzbMATHCrossRefGoogle Scholar
  78. Redmond, C., Yukich, J.: Asymptotics for Euclidian functionals with power-weighted edges. Stoch. Process. Appl. 61, 289–304 (1996) MathSciNetzbMATHCrossRefGoogle Scholar
  79. Rennen, G., Husslage, B., van Dam, E., den Hertog, D.: Nested maximin Latin hypercube designs. Struct Multidisc Optiml 41, 371–395 (2010) CrossRefGoogle Scholar
  80. Rényi, A.: On measures of entropy and information. In: Proc. 4th Berkeley Symp. on Math. Statist. and Prob., pp. 547–561 (1961) Google Scholar
  81. Riccomagno, E., Schwabe, R., Wynn, H.P.: Lattice-based D-optimum design for Fourier regression. Ann. Stat. 25(6), 2313–2327 (1997) MathSciNetzbMATHCrossRefGoogle Scholar
  82. Royle, J., Nychka, D.: An algorithm for the construction of spatial coverage designs with implementation in SPLUS. Comput. Geosci. 24(5), 479–488 (1998) CrossRefGoogle Scholar
  83. Sacks, J., Welch, W., Mitchell, T., Wynn, H.: Design and analysis of computer experiments. Stat. Sci. 4(4), 409–435 (1989) MathSciNetzbMATHCrossRefGoogle Scholar
  84. Santner, T., Williams, B., Notz, W.: The Design and Analysis of Computer Experiments. Springer, Heidelberg (2003) zbMATHGoogle Scholar
  85. Schaback, R.: Lower bounds for norms of inverses of interpolation matrices for radial basis functions. J. Approx. Theory 79, 287–306 (1994) MathSciNetzbMATHCrossRefGoogle Scholar
  86. Schilling, M.F.: Spatial designs when the observations are correlated. Commun. Stat., Simul. Comput. 21(1), 243–267 (1992) MathSciNetzbMATHCrossRefGoogle Scholar
  87. Scott, D.: Multivariate Density Estimation. Wiley, New York (1992) zbMATHCrossRefGoogle Scholar
  88. Shewry, M., Wynn, H.: Maximum entropy sampling. Appl. Stat. 14, 165–170 (1987) CrossRefGoogle Scholar
  89. Sjöstedt-De-Luna, S., Young, A.: The bootstrap and kriging prediction intervals. Scand. J. Stat. 30(1), 175–192 (2003) zbMATHCrossRefGoogle Scholar
  90. Stein, M.: Interpolation of Spatial Data: Some Theory for Kriging. Springer, Heidelberg (1999) zbMATHCrossRefGoogle Scholar
  91. Stinstra, E., den Hertog, D., Stehouwer, P., Vestjens, A.: Constrained maximin designs for computer experiments. Technometrics 45(4), 340–346 (2003) MathSciNetCrossRefGoogle Scholar
  92. Sun, X.: Norm estimates for inverses of Euclidean distance matrices. J. Approx. Theory 70, 339–347 (1992) MathSciNetzbMATHCrossRefGoogle Scholar
  93. Tang, B.: Orthogonal array-based latin hypercubes. J. Am. Stat. Assoc. 88(424), 1392–1397 (1993) zbMATHCrossRefGoogle Scholar
  94. Tsallis, C.: Possible generalization of Boltzmann-Gibbs statistics. J. Stat. Phys. 52(1/2), 479–487 (1988) MathSciNetzbMATHCrossRefGoogle Scholar
  95. van Dam, E.: Two-dimensional minimax Latin hypercube designs. Discrete Appl. Math. 156(18), 3483–3493 (2007) Google Scholar
  96. van Dam, E., Hussage, B., den Hertog, D., Melissen, H.: Maximin Latin hypercube designs in two dimensions. Oper. Res. 55(1), 158–169 (2007) MathSciNetzbMATHCrossRefGoogle Scholar
  97. van Dam, E., Rennen, G., Husslage, B.: Bounds for maximin Latin hypercube designs. Oper. Res. 57(3), 595–608 (2009) MathSciNetzbMATHCrossRefGoogle Scholar
  98. van Groenigen, J.: The influence of variogram parameters on optimal sampling schemes for mapping by kriging. Geoderma 97(3–4), 223–236 (2000) CrossRefGoogle Scholar
  99. Walvoort, D.J.J., Brus, D.J., de Gruijter, J.J.: An R package for spatial coverage sampling and random sampling from compact geographical strata by k-means. Comput. Geosci. 36, 1261–1267 (2010) CrossRefGoogle Scholar
  100. Wendland, H.: Scattered Data Approximation. Cambridge University Press, Cambridge (2005) zbMATHGoogle Scholar
  101. Wolkowicz, H., Styan, G.: Bounds for eigenvalues using traces. Linear Algebra Appl. 29, 471–506 (1980) MathSciNetzbMATHCrossRefGoogle Scholar
  102. Wynn, H.: Maximum entropy sampling and general equivalence theory. In: Di Bucchianico, A., Läuter, H., Wynn, H. (eds.) mODa’7—Advances in Model–Oriented Design and Analysis, Proceedings of the 7th Int. Workshop, Heeze, Netherlands, pp. 211–218. Physica Verlag, Heidelberg (2004) Google Scholar
  103. Yfantis, E., Flatman, G., Behar, J.: Efficiency of kriging estimation for square, triangular, and hexagonal grids. Math. Geol. 19(3), 183–205 (1987) CrossRefGoogle Scholar
  104. Yukich, J.: Probability Theory of Classical Euclidean Optimization Problems. Springer, Berlin (1998) zbMATHGoogle Scholar
  105. Zagoraiou, M., Antognini, A.B.: Optimal designs for parameter estimation of the Ornstein-Uhlenbeck process. Appl. Stoch. Models Bus. Ind. 25(5), 583–600 (2009) MathSciNetzbMATHCrossRefGoogle Scholar
  106. Zhang, H., Zimmerman, D.: Towards reconciling two asymptotic frameworks in spatial statistics. Biometrika 92(4), 921–936 (2005) MathSciNetzbMATHCrossRefGoogle Scholar
  107. Zhu, Z., Stein, M.: Spatial sampling design for parameter estimation of the covariance function. J. Stat. Plan. Inference 134(2), 583–603 (2005) MathSciNetzbMATHCrossRefGoogle Scholar
  108. Zhu, Z., Zhang, H.: Spatial sampling design under the infill asymptotic framework. Environmetrics 17(4), 323–337 (2006) MathSciNetCrossRefGoogle Scholar
  109. Zimmerman, D.L.: Optimal network design for spatial prediction, covariance parameter estimation, and empirical prediction. Environmetrics 17(6), 635–652 (2006) MathSciNetCrossRefGoogle Scholar
  110. Zimmerman, D., Cressie, N.: Mean squared prediction error in the spatial linear model with estimated covariance parameters. Ann. Inst. Stat. Math. 44(1), 27–43 (1992) MathSciNetzbMATHCrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC 2011

Authors and Affiliations

  1. 1.Laboratoire I3S, bâtiment Euclide, les AlgorithmesUniversité de Nice-Sophia Antipolis/CNRSSophia Antipolis cedexFrance
  2. 2.Department of Applied StatisticsJohannes-Kepler-UniversityLinzAustria

Personalised recommendations