Structural and Multidisciplinary Optimization

, Volume 55, Issue 1, pp 221–235 | Cite as

Comparison study of sampling methods for computer experiments using various performance measures

  • Inyong Cho
  • Yongbin Lee
  • Dongheum Ryu
  • Dong-Hoon Choi


This study compares the performance of popular sampling methods for computer experiments using various performance measures to compare them. It is well known that the sample points, in the design space located by a sampling method, determine the quality of the meta-model generated based on expensive computer experiment (or simulation) results obtained at sample (or training) points. Thus, it is very important to locate the sample points using a sampling method suitable for the system of interest to be approximated. However, there is still no clear guideline for selecting an appropriate sampling method for computer experiments. As such, a sampling method, the optimal Latin hypercube design (OLHD), has been popularly used, and quasi-random sequences and the centroidal Voronoi tessellation (CVT) have begun to be noticed recently. Some literature on the CVT asserted that the performance of the CVT was better than that of the LHD, but this assertion seems unfair because those studies only employed space-filling performance measures in favor of the CVT. In this research, we performed the comparison study among the popular sampling methods for computer experiments (CVT, OLHD, and three quasi-random sequences) with employing both space-filling properties and a projective property as performance measures to fairly compare them. We also compared the root mean square error (RMSE) values of Kriging meta-models generated using the five sampling methods to evaluate their prediction performance. From the comparison results, we provided a guideline for selecting appropriate sampling methods for some systems of interest to be approximated.


Sampling method Space-filling property Projective property Root mean square error (RMSE) Optimal Latin hypercube design (OLHD) Centroidal Voronoi tesselation (CVT) 


  1. Chai T, Draxler RR (2014) Root mean square error (RMSE) or mean absolute error (MAE)? – Arguments against avoiding RSME in the literature. Geosci Model Dev 7:1247–1250CrossRefGoogle Scholar
  2. Clerck LD (1986) A method for exact calculation of the Star discrepancy of plane sets applied to the sequences of Hammersley. Mh Math 101:261–278CrossRefMATHGoogle Scholar
  3. Crombecq K, Laermans E, Dhaene T (2011) Efficient space-filling and non-collapsing sequential design strategies for simulation-based modeling. Eur J Oper Res 214:683–696CrossRefGoogle Scholar
  4. Du Q, Faber V, Gunzburger M (1999) Centroidal Voronoi tessellations: applications and algorithms. SIAM Rev 41(4):637–676MathSciNetCrossRefMATHGoogle Scholar
  5. Goel T, Haftka RT, Shyy W, Watson LT (2008) Pitfalls of using a single criterion for selecting experimental designs. Int J Numer Methods Eng 75:127–155CrossRefMATHGoogle Scholar
  6. Gorissen D, Crombecq K, Hendrickx W, Dhaene T (2006) Adaptive distributed metamodeling. High Perform Comput Computational Sci–VECPAR 4395:579–588Google Scholar
  7. Gunzburger MD, Burkardt JV (2004) Uniformity measures for point samples in hypercubes.
  8. Husslage B, Rennen G, van Dam E, den Hertog D (2011) Space-filling Latin hypercube designs for computer experiments. Optim Eng 12(4):611–630CrossRefMATHGoogle Scholar
  9. Jones DR (2001) A taxonomy of global optimization methods based on response surfaces. J Glob Optim 21:345–383MathSciNetCrossRefMATHGoogle Scholar
  10. Krige DG (1951) A statistical approach to some basic mine valuation problems on the Witwatersrand. J Chem Metal Mining Soc South Africa 52(6):119–139Google Scholar
  11. Krykova I (2003) Evaluating of path-dependent securities with low discrepancy methods. Worcester polytechnic institute.Google Scholar
  12. Locatelli M (2003) A note on the Griewank test function. J Glob Optim 25:169–174MathSciNetCrossRefMATHGoogle Scholar
  13. Loh WL (1996) On Latin hypercube sampling. Ann Stat 24(5):2058–2080MathSciNetCrossRefMATHGoogle Scholar
  14. Metheron G (1963) Principles of geostatistics. Econ Geol 58(8):1246–1266CrossRefGoogle Scholar
  15. Romero VJ, Burkardt JV, Gunzburger MD, Peterson JS (2006) Comparison of pure and “Latinized” centroidal Voronoi tessellation against various other statistical sampling methods. Reliab Eng Syst Saf 91:1266–1280CrossRefGoogle Scholar
  16. Sacks J, Welch WJ, Mitchell TJ, Wynn HP (1989) Design and analysis of computer experiments. Stat Sci 4(4):409–435MathSciNetCrossRefMATHGoogle Scholar
  17. Saka Y, Gunzburger MD, Burkardt JV (2007) Latinized, improved LHS, and CVT point sets in hypercubes. Int J Numer Anal Model 4(3–4):729–743MathSciNetMATHGoogle Scholar
  18. Simpson TW, Korte JJ, Mauery TM, Mistree F (1998) Comparison of response surface and kriging models for multidisciplinary design optimization. AIAA 98-4755:381–391Google Scholar
  19. Sobester A, Leary SJ, Keane AJ (2005) On the design of optimization strategies based on global response surface approximation models. J Glob Optim 33:31–59MathSciNetCrossRefMATHGoogle Scholar
  20. Sobol IM (1967) On the distribution of points in a cube and the approximate evaluation of integrals. USSR Comput Math Math Phys 7(4):86–112MathSciNetCrossRefMATHGoogle Scholar
  21. Sobol IM, Asotsky D, Kreinin A, Kucherenko S (2011) Construction and comparison of high-dimensional sobol’ generators. Wilmott 2011(56):64–79CrossRefGoogle Scholar
  22. Viana FAC, Venter G, Balabanov V (2010) An algorithm for fast optimal Latin hypercube design of experiments. Int J Numerical Method Eng 82(2):135–156MathSciNetMATHGoogle Scholar
  23. Wang GG (2003) Adaptive response surface method using inherited Latin hypercube designs. J Mech Des 125(2):210–220CrossRefGoogle Scholar
  24. Wang GG, Shan S (2007) Review of metamodeling techniques in support of engineering design optimization. J Mech Des 129:370–380CrossRefGoogle Scholar
  25. Wang GG, Simpson T (2004) Fuzzy clustering based hierarchical metamodeling for design space reduction and optimization. Eng Opt 36(3):313–335CrossRefGoogle Scholar
  26. Wang GG, Dong Z, Aitchison P (2001) Adaptive response surface method – a global optimization scheme for computation. Eng Opti 33(6):707–734CrossRefGoogle Scholar
  27. Wong TT, Luk WS, Heng PA (1997) Sampling with Hammersley and Halton points. J Graphics Tools 2(2):9–24CrossRefGoogle Scholar
  28. Xiong F, Xiong Y, Chen W, Yang S (2009) Optimizing Latin hypercube design for sequential sampling of computer experiments. Eng Opt 41(8):793–810CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2016

Authors and Affiliations

  • Inyong Cho
    • 1
  • Yongbin Lee
    • 2
  • Dongheum Ryu
    • 1
  • Dong-Hoon Choi
    • 3
  1. 1.Graduate School of Mechanical Convergence EngineeringHanyang UniversitySeoulKorea
  2. 2.PIDOTECH Inc., HIT #312Hanyang UniversitySeoulKorea
  3. 3.School of Mechanical EngineeringHanyang UniversitySeoulKorea

Personalised recommendations