Abstract
Error estimation measures are useful for assessing uncertainty in surrogate predictions. We use a suite of test problems to appraise several error estimation measures for polynomial response surfaces and kriging. In addition, we study the performance of cross-validation error measures that can be used with any surrogate. We use 1,000 experimental designs to obtain the variability of error estimates with respect to the experimental designs for each problem. We find that the (actual) errors for polynomial response surfaces are less sensitive to the choice of experimental designs than the kriging errors. This is attributed to the variability in the maximum likelihood estimates of the kriging parameters. We find that no single error measure outperforms other measures on all the problems. Computationally expensive integrated local error measures (standard error for polynomials and mean square error for kriging) estimate the actual root mean square error very well. The distribution-free cross-validation error characterized the actual errors reasonably well. While the estimated root mean square error for polynomial response surface is a good estimate of the actual errors, the process variance for kriging is not. We explore a few methods of simultaneously using multiple error measures and demonstrate that the geometric means of several combinations of error measures improve the assessment of the actual errors over individual error measures.
Similar content being viewed by others
References
Congdon CD, Martin JD (2007) On using standard residuals as a metric of kriging model quality. In: Proceedings of the 48th AIAA/ASME/ASCE/AHS/ASC structures, structural dynamics, and materials conference, Honolulu HI, 23 Apr–26 Apr, AIAA-2007-1928
Currin C, Mitchell TJ, Morris MD, Ylvisaker D (1998) A Bayesian approach to the design and analysis of computer experiments. Technical report ORNL-6498. Oak Ridge National Laboratory, Oak Ridge, TN
Den Hertog D, Kleijnen JPC, Siem AYD (2006) The correct kriging variance estimated by bootstrapping. J Oper Res Soc 57(4):400–409
Dixon LCW, Szegö GP (1978) Towards global optimization 2. North-Holland, Amsterdam
Goel T (2007) Multiple surrogates and error modeling in optimization of liquid rocket propulsion components. PhD thesis, Dept. of Mechanical and Aerospace Engineering, The University of Florida, Gainesville, FL
Goel T, Haftka RT, Shyy W, Queipo NV (2007) Ensemble of surrogates. Struct Multidisc Optim 33:199–216
Goel T, Haftka RT, Shyy W, Watson LT (2008) Pitfalls of using a single criterion for selecting experimental designs. Int J Numer Methods Eng 75(2):127–155
Jones DR, Schonlau M, Welch WJ (1998) Efficient global optimization of expensive black-box functions. J Glob Optim 13(4):455–492
Khuri AI, Cornell JA (1996) Response surfaces: designs and analysis. Marcel Dekker, New York
Kleijnen JPC, van Beers WCM (2004) Application-driven sequential designs for simulation experiments: kriging metamodeling. J Oper Res Soc 55(9):876–883
Lophaven SN, Nielsen HB, Sondergaard J (2002a) DACE: a MATLAB kriging toolbox, version 2.0, information and mathematical modeling. Technical University of Denmark, Denmark
Lophaven SN, Nielsen HB, Sondergaard J (2002b) DACE: aspects of the MATLAB toolbox DACE, information and mathematical modeling. Technical University of Denmark, Denmark
Martin JD (2005) A methodology for evaluating system-level uncertainty in the conceptual design of complex multidisciplinary systems. PhD thesis, Dept. of Mechanical Engineering, The Pennsylvania State University, University Park, PA
Martin JD, Simpson TW (2005) Use of kriging models to approximate deterministic computer models. AIAA Journal 43(4):853–863
Matheron G (1963) Principles of geostatistics. Econ Geol 58:1246–1266
MATLAB (2002) The language of technical computing, version 6.5 release 13. The MathWorks Inc., Natick, MA
McKay M, Conover W, Beckman R (1979) A comparison of three methods for selecting values of input variables in the analysis of output from a computer code. Technometrics 21:239–245
Meckesheimer M, Barton RR, Simpson TW, Booker A (2002) Computationally inexpensive metamodel assessment strategies. AIAA J 40(10):2053–2060
Mitchell TJ, Morris MD (1992) Bayesian design and analysis of computer experiments: two examples. Stat Sin 2(2):359–379
Myers RH, Montgomery DC (1995) Response surface methodology—process and product optimization using de signed experiments. Wiley, New York
Queipo NV, Haftka RT, Shyy W, Goel T, Vaidyanathan R, Tucker PK (2005) Surrogate-based analysis and optimization. Prog Aerosp Sci 41:1–28
Ueberhuber CW (1997) Numerical computation 2: methods, software and analysis. Springer, Berlin, p 71
Wu YT, Shin Y, Sues R, Cesare M (2001) Safety factor based approach for probability-based design optimization. In: Proceedings of the 42nd AIAA/ASME/ASCE/AHS/ASC structures, structural dynamics and materials conference, Seattle WA, 2001, AIAA-2001-1522
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Goel, T., Hafkta, R.T. & Shyy, W. Comparing error estimation measures for polynomial and kriging approximation of noise-free functions. Struct Multidisc Optim 38, 429–442 (2009). https://doi.org/10.1007/s00158-008-0290-z
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00158-008-0290-z