Performance assessment of a cross-validation sampling strategy with active surrogate model selection

  • Andrea GarboEmail author
  • Brian J. German
Research Paper


Surrogate Models (SM) are now regarded as powerful tools for engineering design applications; they approximate expensive responses obtained by either a computer simulation or a real experiment with more computationally efficient mathematical models. The goal of reducing the number of function evaluations required to create the training dataset led to the development of strategies capable of effectively sampling the design space. In particular, sequential adaptive techniques seem to be the most promising approaches for engineering design applications. Unfortunately, most of these techniques supervise the sampling phase with an SM created from the available training set, making their behavior dependent on the chosen SM formulation. This characteristic can cause sampling performance deterioration, in particular when an inaccurate SM formulation is used to supervise the sampling process. The solution proposed in this paper tries to mitigate this problem by coupling the model-dependent sequential adaptive technique with an active SM selection approach. This surrogate modeling architecture is tested on several example functions, and results indicate that it inherits the advantages of both constituent elements: an increased flexibility due to the active SM selection and a reduced total number of samples due to the sequential adaptive sampling strategy. Other parameters which may affect the sampling performance (such as the SM selection frequency and criterion) are also analyzed in an attempt to define general guidelines for the application of the method. The sampling technique considered in this study is based on cross-validation variance, and the surrogate modeling performance is assessed on nine two-dimensional and two five-dimensional test functions.


Adaptive sampling Model dependency Global surrogate model Leave-one-out validation 



This material is based upon work supported by the USA National Science Foundation under Grant No.1537782.

Supplementary material

158_2018_2190_MOESM1_ESM.pdf (137 kb)
(PDF 137 KB)


  1. Abate G, Mavris DN (2017) Cfd analysis of leading edge tubercle effects on wind turbine performance. In: 15th international energy conversion engineering conference, p 4626Google Scholar
  2. Adorio EP, Diliman U (2005) Mvf-multivariate test functions library in C for unconstrained global optimization.
  3. Chen Quin Lam M (2008) Sequential adaptive designs in computer experiments for response surface model. fit PhD thesis, The Ohio State University, USAGoogle Scholar
  4. Crombecq K, Gorissen D, Deschrijver D, Dhaene T (2011) A novel hybrid sequential design strategy for global surrogate modeling of computer experiments. SIAM J Sci Comput 33(4):1948–1974MathSciNetCrossRefzbMATHGoogle Scholar
  5. Duvenaud D (2014) Automatic model construction with gaussian processes. PhD thesis, University of Cambridge, UKGoogle Scholar
  6. Dyn N, Levin D, Rippa S (1986) Numerical procedures for surface fitting of scattered data by radial functions. SIAM J Sci Stat Comput 7(2):639–659MathSciNetCrossRefzbMATHGoogle Scholar
  7. Garbo A, German B (2016) Comparison of adaptive design space exploration methods applied to S-duct CFD simulation. In: 57th AIAA/ASCE/AHS/ASC structures, structural dynamics, and materials conference, p 0416Google Scholar
  8. Garbo A, German B (2017) Adaptive sampling with adaptive surrogate model selection for computer experiment applications. In: 18th AIAA/ISSMO multidisciplinary analysis and optimization conference, p 4430Google Scholar
  9. Goel T, Hafkta RT, Shyy W (2009) Comparing error estimation measures for polynomial and kriging approximation of noise-free functions. Struct Multidiscip Optim 38(5):429–442CrossRefGoogle Scholar
  10. Gorissen D (2007) Heterogeneous evolution of surrogate models. Master’s thesis, Katholieke Universiteit Leuven, BelgiumGoogle Scholar
  11. Gorissen D, De Tommasi L, Crombecq K, Dhaene T (2009) Sequential modeling of a low noise amplifier with neural networks and active learning. Neural Comput Appl 18(5):485–494CrossRefGoogle Scholar
  12. Gorodetsky A, Marzouk Y (2016) Mercer kernels and integrated variance experimental design: connections between Gaussian process regression and polynomial approximation. SIAM/ASA Journal on Uncertainty Quantification 4(1):796–828MathSciNetCrossRefzbMATHGoogle Scholar
  13. Gunn SR (1998) Support vector machines for classification and regression. ISIS Tech Rep 14:85–86Google Scholar
  14. Hardy RL (1971) Multiquadric equations of topography and other irregular surfaces. J Geophys Res 76 (8):1905–1915CrossRefGoogle Scholar
  15. Jin R, Chen W, Simpson TW (2001) Comparative studies of metamodelling techniques under multiple modelling criteria. Struct Multidiscip Optim 23(1):1–13CrossRefGoogle Scholar
  16. Jin R, Chen W, Sudjianto A (2002) On sequential sampling for global metamodeling in engineering design. In: ASME 2002 international design engineering technical conferences and computers and information in engineering conference, American society of mechanical engineers, pp 539–548Google Scholar
  17. Kleijnen JP, Van Beers WC (2004) Application-driven sequential designs for simulation experiments: Kriging metamodelling. J Oper Res Soc 55(8):876–883CrossRefzbMATHGoogle Scholar
  18. Lancaster P, Salkauskas K (1981) Surfaces generated by moving least squares methods. Math Comput 37 (155):141–158MathSciNetCrossRefzbMATHGoogle Scholar
  19. Lee TH (2003) The design of CMOS radio-frequency integrated circuits. Cambridge, UKCrossRefGoogle Scholar
  20. Liao X, Li Q, Yang X, Zhang W, Li W (2008) Multiobjective optimization for crash safety design of vehicles using stepwise regression model. Struct Multidiscip Optim 35(6):561–569CrossRefGoogle Scholar
  21. Liu H, Xu S, Ma Y, Chen X, Wang X (2016) An adaptive Bayesian sequential sampling approach for global metamodeling. J Mech Des 138(1):011,404CrossRefGoogle Scholar
  22. Liu H, Ong Y-S, Cai J (2018) A survey of adaptive sampling for global metamodeling in support of simulation-based complex engineering design. Struct Multidiscip Optim 57(1):393–416CrossRefGoogle Scholar
  23. Loeppky JL, Sacks J, Welch WJ (2009) Choosing the sample size of a computer experiment: a practical guide. Technometrics 51(4):366–376MathSciNetCrossRefGoogle Scholar
  24. Mackman T, Allen C, Ghoreyshi M, Badcock K (2013) Comparison of adaptive sampling methods for generation of surrogate aerodynamic models. AIAA J 51(4):797–808CrossRefGoogle Scholar
  25. Marcy P (2014) On the use and utility of gradient information in computer experiments. PhD thesis, University of Wyoming, Department of Statistics, USAGoogle Scholar
  26. Meckesheimer M, Booker AJ, Barton RR, Simpson TW (2002) Computationally inexpensive metamodel assessment strategies. AIAA J 40(10):2053–2060CrossRefGoogle Scholar
  27. Mehdad E, Kleijnen JP (2018) Stochastic intrinsic kriging for simulation metamodeling. Appl Stoch Model Bus Ind 34(3):322–337MathSciNetCrossRefGoogle Scholar
  28. Mehmani A, Chowdhury S, Messac A (2015) Predictive quantification of surrogate model fidelity based on modal variations with sample density. Struct Multidiscip Optim 52(2):353–373CrossRefGoogle Scholar
  29. Mishra PK, Nath SK, Sen MK, Fasshauer GE (2015) Hybrid gaussian-cubic radial basis functions for scattered data interpolation. arXiv:151207584
  30. Morris MD, Mitchell TJ, Ylvisaker D (1993) Bayesian design and analysis of computer experiments: use of derivatives in surface prediction. Technometrics 35(3):243–255MathSciNetCrossRefzbMATHGoogle Scholar
  31. Plumlee M, Apley DW (2016) Lifted brownian kriging models. Technometrics (just-accepted)Google Scholar
  32. Queipo NV, Haftka RT, Shyy W, Goel T, Vaidyanathan R, Tucker PK (2005) Surrogate-based analysis and optimization. Prog Aerosp Sci 41(1):1–28CrossRefGoogle Scholar
  33. Rasmussen CE, Williams CK (2006) Gaussian processes for machine learning. Tech. Rep. ISBN 0-262-18253-X Cambridge, MA, USAGoogle Scholar
  34. Sacks J, Schiller SB, Welch WJ (1989) Designs for computer experiments. Technometrics 31(1):41–47MathSciNetCrossRefGoogle Scholar
  35. Salemi P, Nelson BL, Staum J (2016) Moving least squares regression for high-dimensional stochastic simulation metamodeling. ACM Transactions on Modeling and Computer Simulation (TOMACS) 26(3):16MathSciNetCrossRefzbMATHGoogle Scholar
  36. Santner TJ (2013) The design and analysis of computer experiments. Springer, BerlinGoogle Scholar
  37. Simpson TW, Lin DK, Chen W (2001a) Sampling strategies for computer experiments: design and analysis. Int J Reliab Appl 2(3):209–240Google Scholar
  38. Simpson TW, Poplinski J, Koch PN, Allen JK (2001b) Metamodels for computer-based engineering design: survey and recommendations. Eng Comput 17(2):129–150CrossRefzbMATHGoogle Scholar
  39. Song CY, Choi HY, Lee J (2014) Approximate multi-objective optimization using conservative and feasible moving least squares method: application to automotive knuckle design. Struct Multidiscip Optim 49(5):851–861CrossRefGoogle Scholar
  40. Viana FA, Haftka RT, Steffen Jr V (2009) Multiple surrogates: how cross-validation errors can help us to obtain the best predictor. Struct Multidiscip Optim 39(4):439–457CrossRefGoogle Scholar
  41. Wang B, Hao P, Li G, Fang Y, Wang X, Zhang X (2013) Determination of realistic worst imperfection for cylindrical shells using surrogate model. Struct Multidiscip Optim 48(4):777–794CrossRefGoogle Scholar
  42. Wolpert DH (2002) The supervised learning no-free-lunch theorems. In: Soft computing and industry. Springer, pp 25–42Google Scholar
  43. Wu YT, Shin Y, Sues R, Cesare M (2001) Safety-factor based approach for probability-based design optimization. In: 19th AIAA applied aerodynamics conference, p 1522Google Scholar
  44. Yegnanarayana B (2009) Artificial neural networks. PHI Learning Pvt. LtdGoogle Scholar
  45. Zhang N, Apley DW (2014) Fractional Brownian fields for response surface metamodeling. J Qual Technol 46(4):285CrossRefGoogle Scholar
  46. Zhang N, Apley DW (2015) Brownian integrated covariance functions for gaussian process modeling: sigmoidal versus localized basis functions. J Am Stat Assoc (just-accepted) :00–00Google Scholar
  47. Zhou Q, Shao X, Jiang P, Gao Z, Zhou H, Shu L (2016) An active learning variable-fidelity metamodelling approach based on ensemble of metamodels and objective-oriented sequential sampling. J Eng Des 27(4-6):205–231CrossRefGoogle Scholar

Copyright information

© Springer-Verlag GmbH Germany, part of Springer Nature 2019

Authors and Affiliations

  1. 1.School of Aerospace EngineeringGeorgia Institute of TechnologyAtlantaUSA

Personalised recommendations