This study compares the performance of popular sampling methods for computer experiments using various performance measures to compare them. It is well known that the sample points, in the design space located by a sampling method, determine the quality of the meta-model generated based on expensive computer experiment (or simulation) results obtained at sample (or training) points. Thus, it is very important to locate the sample points using a sampling method suitable for the system of interest to be approximated. However, there is still no clear guideline for selecting an appropriate sampling method for computer experiments. As such, a sampling method, the optimal Latin hypercube design (OLHD), has been popularly used, and quasi-random sequences and the centroidal Voronoi tessellation (CVT) have begun to be noticed recently. Some literature on the CVT asserted that the performance of the CVT was better than that of the LHD, but this assertion seems unfair because those studies only employed space-filling performance measures in favor of the CVT. In this research, we performed the comparison study among the popular sampling methods for computer experiments (CVT, OLHD, and three quasi-random sequences) with employing both space-filling properties and a projective property as performance measures to fairly compare them. We also compared the root mean square error (RMSE) values of Kriging meta-models generated using the five sampling methods to evaluate their prediction performance. From the comparison results, we provided a guideline for selecting appropriate sampling methods for some systems of interest to be approximated.
Sampling method Space-filling property Projective property Root mean square error (RMSE) Optimal Latin hypercube design (OLHD) Centroidal Voronoi tesselation (CVT)
Chai T, Draxler RR (2014) Root mean square error (RMSE) or mean absolute error (MAE)? – Arguments against avoiding RSME in the literature. Geosci Model Dev 7:1247–1250CrossRefGoogle Scholar
Clerck LD (1986) A method for exact calculation of the Star discrepancy of plane sets applied to the sequences of Hammersley. Mh Math 101:261–278CrossRefMATHGoogle Scholar
Crombecq K, Laermans E, Dhaene T (2011) Efficient space-filling and non-collapsing sequential design strategies for simulation-based modeling. Eur J Oper Res 214:683–696CrossRefGoogle Scholar
Romero VJ, Burkardt JV, Gunzburger MD, Peterson JS (2006) Comparison of pure and “Latinized” centroidal Voronoi tessellation against various other statistical sampling methods. Reliab Eng Syst Saf 91:1266–1280CrossRefGoogle Scholar