Abstract
Surrogate Models (SM) are now regarded as powerful tools for engineering design applications; they approximate expensive responses obtained by either a computer simulation or a real experiment with more computationally efficient mathematical models. The goal of reducing the number of function evaluations required to create the training dataset led to the development of strategies capable of effectively sampling the design space. In particular, sequential adaptive techniques seem to be the most promising approaches for engineering design applications. Unfortunately, most of these techniques supervise the sampling phase with an SM created from the available training set, making their behavior dependent on the chosen SM formulation. This characteristic can cause sampling performance deterioration, in particular when an inaccurate SM formulation is used to supervise the sampling process. The solution proposed in this paper tries to mitigate this problem by coupling the model-dependent sequential adaptive technique with an active SM selection approach. This surrogate modeling architecture is tested on several example functions, and results indicate that it inherits the advantages of both constituent elements: an increased flexibility due to the active SM selection and a reduced total number of samples due to the sequential adaptive sampling strategy. Other parameters which may affect the sampling performance (such as the SM selection frequency and criterion) are also analyzed in an attempt to define general guidelines for the application of the method. The sampling technique considered in this study is based on cross-validation variance, and the surrogate modeling performance is assessed on nine two-dimensional and two five-dimensional test functions.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Abate G, Mavris DN (2017) Cfd analysis of leading edge tubercle effects on wind turbine performance. In: 15th international energy conversion engineering conference, p 4626
Adorio EP, Diliman U (2005) Mvf-multivariate test functions library in C for unconstrained global optimization. http://www.geocities.ws/eadorio/mvf.pdf
Chen Quin Lam M (2008) Sequential adaptive designs in computer experiments for response surface model. fit PhD thesis, The Ohio State University, USA
Crombecq K, Gorissen D, Deschrijver D, Dhaene T (2011) A novel hybrid sequential design strategy for global surrogate modeling of computer experiments. SIAM J Sci Comput 33(4):1948–1974
Duvenaud D (2014) Automatic model construction with gaussian processes. PhD thesis, University of Cambridge, UK
Dyn N, Levin D, Rippa S (1986) Numerical procedures for surface fitting of scattered data by radial functions. SIAM J Sci Stat Comput 7(2):639–659
Garbo A, German B (2016) Comparison of adaptive design space exploration methods applied to S-duct CFD simulation. In: 57th AIAA/ASCE/AHS/ASC structures, structural dynamics, and materials conference, p 0416
Garbo A, German B (2017) Adaptive sampling with adaptive surrogate model selection for computer experiment applications. In: 18th AIAA/ISSMO multidisciplinary analysis and optimization conference, p 4430
Goel T, Hafkta RT, Shyy W (2009) Comparing error estimation measures for polynomial and kriging approximation of noise-free functions. Struct Multidiscip Optim 38(5):429–442
Gorissen D (2007) Heterogeneous evolution of surrogate models. Master’s thesis, Katholieke Universiteit Leuven, Belgium
Gorissen D, De Tommasi L, Crombecq K, Dhaene T (2009) Sequential modeling of a low noise amplifier with neural networks and active learning. Neural Comput Appl 18(5):485–494
Gorodetsky A, Marzouk Y (2016) Mercer kernels and integrated variance experimental design: connections between Gaussian process regression and polynomial approximation. SIAM/ASA Journal on Uncertainty Quantification 4(1):796–828
Gunn SR (1998) Support vector machines for classification and regression. ISIS Tech Rep 14:85–86
Hardy RL (1971) Multiquadric equations of topography and other irregular surfaces. J Geophys Res 76 (8):1905–1915
Jin R, Chen W, Simpson TW (2001) Comparative studies of metamodelling techniques under multiple modelling criteria. Struct Multidiscip Optim 23(1):1–13
Jin R, Chen W, Sudjianto A (2002) On sequential sampling for global metamodeling in engineering design. In: ASME 2002 international design engineering technical conferences and computers and information in engineering conference, American society of mechanical engineers, pp 539–548
Kleijnen JP, Van Beers WC (2004) Application-driven sequential designs for simulation experiments: Kriging metamodelling. J Oper Res Soc 55(8):876–883
Lancaster P, Salkauskas K (1981) Surfaces generated by moving least squares methods. Math Comput 37 (155):141–158
Lee TH (2003) The design of CMOS radio-frequency integrated circuits. Cambridge, UK
Liao X, Li Q, Yang X, Zhang W, Li W (2008) Multiobjective optimization for crash safety design of vehicles using stepwise regression model. Struct Multidiscip Optim 35(6):561–569
Liu H, Xu S, Ma Y, Chen X, Wang X (2016) An adaptive Bayesian sequential sampling approach for global metamodeling. J Mech Des 138(1):011,404
Liu H, Ong Y-S, Cai J (2018) A survey of adaptive sampling for global metamodeling in support of simulation-based complex engineering design. Struct Multidiscip Optim 57(1):393–416
Loeppky JL, Sacks J, Welch WJ (2009) Choosing the sample size of a computer experiment: a practical guide. Technometrics 51(4):366–376
Mackman T, Allen C, Ghoreyshi M, Badcock K (2013) Comparison of adaptive sampling methods for generation of surrogate aerodynamic models. AIAA J 51(4):797–808
Marcy P (2014) On the use and utility of gradient information in computer experiments. PhD thesis, University of Wyoming, Department of Statistics, USA
Meckesheimer M, Booker AJ, Barton RR, Simpson TW (2002) Computationally inexpensive metamodel assessment strategies. AIAA J 40(10):2053–2060
Mehdad E, Kleijnen JP (2018) Stochastic intrinsic kriging for simulation metamodeling. Appl Stoch Model Bus Ind 34(3):322–337
Mehmani A, Chowdhury S, Messac A (2015) Predictive quantification of surrogate model fidelity based on modal variations with sample density. Struct Multidiscip Optim 52(2):353–373
Mishra PK, Nath SK, Sen MK, Fasshauer GE (2015) Hybrid gaussian-cubic radial basis functions for scattered data interpolation. arXiv:151207584
Morris MD, Mitchell TJ, Ylvisaker D (1993) Bayesian design and analysis of computer experiments: use of derivatives in surface prediction. Technometrics 35(3):243–255
Plumlee M, Apley DW (2016) Lifted brownian kriging models. Technometrics (just-accepted)
Queipo NV, Haftka RT, Shyy W, Goel T, Vaidyanathan R, Tucker PK (2005) Surrogate-based analysis and optimization. Prog Aerosp Sci 41(1):1–28
Rasmussen CE, Williams CK (2006) Gaussian processes for machine learning. Tech. Rep. ISBN 0-262-18253-X Cambridge, MA, USA
Sacks J, Schiller SB, Welch WJ (1989) Designs for computer experiments. Technometrics 31(1):41–47
Salemi P, Nelson BL, Staum J (2016) Moving least squares regression for high-dimensional stochastic simulation metamodeling. ACM Transactions on Modeling and Computer Simulation (TOMACS) 26(3):16
Santner TJ (2013) The design and analysis of computer experiments. Springer, Berlin
Simpson TW, Lin DK, Chen W (2001a) Sampling strategies for computer experiments: design and analysis. Int J Reliab Appl 2(3):209–240
Simpson TW, Poplinski J, Koch PN, Allen JK (2001b) Metamodels for computer-based engineering design: survey and recommendations. Eng Comput 17(2):129–150
Song CY, Choi HY, Lee J (2014) Approximate multi-objective optimization using conservative and feasible moving least squares method: application to automotive knuckle design. Struct Multidiscip Optim 49(5):851–861
Viana FA, Haftka RT, Steffen Jr V (2009) Multiple surrogates: how cross-validation errors can help us to obtain the best predictor. Struct Multidiscip Optim 39(4):439–457
Wang B, Hao P, Li G, Fang Y, Wang X, Zhang X (2013) Determination of realistic worst imperfection for cylindrical shells using surrogate model. Struct Multidiscip Optim 48(4):777–794
Wolpert DH (2002) The supervised learning no-free-lunch theorems. In: Soft computing and industry. Springer, pp 25–42
Wu YT, Shin Y, Sues R, Cesare M (2001) Safety-factor based approach for probability-based design optimization. In: 19th AIAA applied aerodynamics conference, p 1522
Yegnanarayana B (2009) Artificial neural networks. PHI Learning Pvt. Ltd
Zhang N, Apley DW (2014) Fractional Brownian fields for response surface metamodeling. J Qual Technol 46(4):285
Zhang N, Apley DW (2015) Brownian integrated covariance functions for gaussian process modeling: sigmoidal versus localized basis functions. J Am Stat Assoc (just-accepted) :00–00
Zhou Q, Shao X, Jiang P, Gao Z, Zhou H, Shu L (2016) An active learning variable-fidelity metamodelling approach based on ensemble of metamodels and objective-oriented sequential sampling. J Eng Des 27(4-6):205–231
Acknowledgements
This material is based upon work supported by the USA National Science Foundation under Grant No.1537782.
Author information
Authors and Affiliations
Corresponding author
Additional information
Responsible Editor: Shapour Azarm
Publisher’s note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
About this article
Cite this article
Garbo, A., German, B.J. Performance assessment of a cross-validation sampling strategy with active surrogate model selection. Struct Multidisc Optim 59, 2257–2272 (2019). https://doi.org/10.1007/s00158-018-02190-7
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00158-018-02190-7