Comparing error estimation measures for polynomial and kriging approximation of noise-free functions

Abstract

Error estimation measures are useful for assessing uncertainty in surrogate predictions. We use a suite of test problems to appraise several error estimation measures for polynomial response surfaces and kriging. In addition, we study the performance of cross-validation error measures that can be used with any surrogate. We use 1,000 experimental designs to obtain the variability of error estimates with respect to the experimental designs for each problem. We find that the (actual) errors for polynomial response surfaces are less sensitive to the choice of experimental designs than the kriging errors. This is attributed to the variability in the maximum likelihood estimates of the kriging parameters. We find that no single error measure outperforms other measures on all the problems. Computationally expensive integrated local error measures (standard error for polynomials and mean square error for kriging) estimate the actual root mean square error very well. The distribution-free cross-validation error characterized the actual errors reasonably well. While the estimated root mean square error for polynomial response surface is a good estimate of the actual errors, the process variance for kriging is not. We explore a few methods of simultaneously using multiple error measures and demonstrate that the geometric means of several combinations of error measures improve the assessment of the actual errors over individual error measures.