Abstract
Tasks such as analysis, design optimization, and uncertainty quantification can be computationally expensive. Surrogate modeling is often the tool of choice for reducing the burden associated with such data-intensive tasks. However, even after years of intensive research, surrogate modeling still involves a struggle to achieve maximum accuracy within limited resources. This work summarizes various advanced, yet often straightforward, statistical tools that help. We focus on four techniques with increasing popularity in the surrogate modeling community: (i) variable screening and dimensionality reduction in both the input and the output spaces, (ii) data sampling techniques or design of experiments, (iii) simultaneous use of multiple surrogates, and (iv) sequential sampling. We close the paper with some suggestions for future research.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Notes
In practice, k-fold implementations are influenced by the size of datasets. With small to medium datasets, using clustering algorithms, such as k-means (Kanungo et al. 2002), improves robustness. When datasets are very large, clustering is very time consuming and potentially less important. Therefore, random selection is commonly used.
References
Acar E, Rais-Rohani M (2009) Ensemble of metamodels with optimized weight factors. Struct Multidiscip Optim 37(3):279–294. https://doi.org/10.1007/s00158-008-0230-y
Adams BM, Bohnhoff WJ, Dalbey KR, Ebeida MS, Eddy JP, Eldred MS, Hooper RW, Hough PD, Hu KT, Jakeman JD, Khalil M, Maupin KA, Monschke JA, Ridgway EM, Rushdi AA, Seidl DT, Stephens JA, Swiler LP, Winokur JG (2020) Dakota, a multilevel parallel object-oriented framework for design optimization, parameter estimation, uncertainty quantification, and sensitivity analysis: Version 6.12 user’s manual. Technical Report SAND2020-12495, Sandia National Laboratories
Agostinelli C (2002) Robust stepwise regression. J Appl Stat 29(6):825–840. https://doi.org/10.1080/02664760220136168
Andres TH, Hajas WC (1993) Using iterated fractional factorial design to screen parameters in sensitivity analysis of a probabilistic risk assessment model Mathematical Methods and Supercomputing in Nuclear Applications, Germany. http://inis.iaea.org/search/search.aspx?orig_q=RN:25062495
Arnst M, Soize C, Bulthuis K (2020) Computation of sobol indices in global sensitivity analysis from small data sets by probabilistic learning on manifolds. International Journal for Uncertainty Quantification https://doi.org/10.1615/int.j.uncertaintyquantification.2020032674
Audet C, Dennis J, Moore D, Booker A, Frank P (2000) A surrogate-model-based method for constrained optimization In: 8th AIAA/NASA/USAF/ISSMO Symposium on Multidisciplinary Analysis and Optimization. Long Beach, USA. https://doi.org/10.2514/6.2000-4891
Ballester-Ripoll R, Paredes EG, Pajarola R (2019) Sobol tensor trains for global sensitivity analysis. Reliability Engineering & System Safety 183:311–322. https://doi.org/10.1016/j.ress.2018.11.007
Barthelemy JFM, Haftka RT (1993) Approximation concepts for optimum structural design – a review. Struct Multidiscip Optim 5(3):129–144. https://doi.org/10.1007/BF01743349
Bartoli N, Lefebvre T, Dubreuil S, Olivanti R, Priem R, Bons N, Martins J, Morlier J (2019) Adaptive modeling strategy for constrained global optimization with application to aerodynamic wing design. Aerosp Sci Technol 90:85–102. https://doi.org/10.1016/j.ast.2019.03.041, https://www.sciencedirect.com/science/article/pii/S1270963818306011
Beale EML, Kendall MG, Mann DW (1967) The discarding of variables in multivariate analysis. Biometrika 54(3-4):357–366. https://doi.org/10.1093/biomet/54.3-4.357
Bect J, Ginsbourger D, Li L, Picheny V, Vazquez E (2012) Sequential design of computer experiments for the estimation of a probability of failure. Stat Comput 22(3):773–793. https://doi.org/10.1007/s11222-011-9241-4
Belsley D, Kuh E, Welsch R (2004) Regression diagnostics: Identifying influential data and sources of collinearity. Wiley-IEEE, New York
Berthelin G, Dubreuil S, Salaün M, Gogu C, Bartoli N (2020) Model order reduction coupled with efficient global optimization for multidisciplinary optimization. In: 14th World Congress on Computational Mechanics (WCCM)
Bettonvil B (1990) Detection of important factors by sequential bifurcation. Tilburg University Press, Tilburg
Bichon BJ, Eldred MS, Swiler LP, Mahadevan S, McFarland JM (2008) Efficient global reliability analysis for nonlinear implicit performance functions. AIAA J 46(10):2459–2468. https://doi.org/10.2514/1.34321
Bichon BJ, McFarland JM, Mahadevan S (2011) Efficient surrogate models for reliability analysis of systems with multiple failure modes. Reliability Engineering & System Safety 96(10):1386–1395. https://doi.org/10.1016/j.ress.2011.05.008. https://www.sciencedirect.com/science/article/pii/S0951832011001062
Bischl B, Richter J, Bossek J, Horn D, Thomas J, Lang M (2017) mlrMBO: a modular framework for model-based optimization of expensive black-box functions. arXiv preprint arXiv:170303373
Borgonovo E, Plischke E (2016) Sensitivity analysis: a review of recent advances. Eur J Oper Res 248(3):869–887. https://doi.org/10.1016/j.ejor.2015.06.032. https://www.sciencedirect.com/science/article/pii/S0377221715005469
Bouhlel MA, Bartoli N, Otsmane A, Morlier J (2016) Improving kriging surrogates of high-dimensional design models by partial least squares dimension reduction. Struct Multidiscip Optim 53(5):935–952. https://doi.org/10.1007/s00158-015-1395-9
Bouhlel MA, Hwang JT, Bartoli N, Lafage R, Morlier J, Martins JRRA (2019) A python surrogate modeling framework with derivatives. Adv Eng Softw 135:102662. https://doi.org/10.1016/j.advengsoft.2019.03.005. https://www.sciencedirect.com/science/article/pii/S0965997818309360
Box GEP, Hunter JS, Hunter WG (1978) Statistics for experimenters: an introduction to design, data analysis, and model building. John Wiley & Sons Inc, New York, USA
Box GEP, Tidwell PW (1962) Transformation of the independent variables. Technometrics 4(4):531–550. https://doi.org/10.1080/00401706.1962.10490038, https://www.tandfonline.com/doi/abs/10.1080/00401706.1962.10490038
Breitkopf P, Coelho RF (2013) Multidisciplinary design optimization in computational mechanics. John Wiley & Sons
Buckingham E (1914) On physically similar systems; illustrations of the use of dimensional equations. Phys Rev 4(4):345–376. https://doi.org/10.1103/PhysRev.4.345
Budinger M, Passieux JC, Gogu C, Fraj A (2014) Scaling-law- based metamodels for the sizing of mechatronic systems. Mechatronics 24(7):775–787. https://doi.org/10.1016/j.mechatronics.2013.11.012. https://doi.org/10.1016/j.mechatronics.2013.11.012. https://doi.org/https://www.sciencedirect.com/science/article/pii/S0957415813002328
Calandra R, Peters J, Rasmussen CE, Deisenroth MP (2016) Manifold gaussian processes for regression. In: 2016 International joint conference on neural networks (IJCNN), IEEE Vancouver, Canada, pp 3338–3345. https://doi.org/10.1109/IJCNN.2016.7727626
Chaudhuri A, Marques AN, Willcox K (2021) mfEGRA: Multifidelity efficient global reliability analysis through active learning for failure boundary location. Structural and Multidisciplinary Optimization https://doi.org/10.1007/s00158-021-02892-5
Chaudhuri A, Lam R, Willcox K (2018) Multifidelity uncertainty propagation via adaptive surrogates in coupled multidisciplinary systems. AIAA J 56(1):235–249. https://doi.org/10.2514/1.J055678
Chen VCP, Tsui KL, Barton RR, Meckesheimer M (2006) A review on design, modeling and applications of computer experiments. IIE Transactions 38(4):273–291, https://doi.org/10.1080/07408170500232495
Cheng B, Titterington DM (1994) Neural networks: a review from a statistical perspective. Stat Sci 9(1):2–54. https://doi.org/10.1214/ss/1177010638
Chernoff H (1959) Sequential design of experiments. The Annals of Mathematical Statistics 30(3):755–770. http://www.jstor.org/stable/2237415
Cho YC, Jayaraman B, Viana FAC, Haftka RT, Shyy W (2010) Surrogate modelling for characterising the performance of a dielectric barrier discharge plasma actuator. International Journal of Computational Fluid Dynamics 24(7):281–301 https://doi.org/10.1080/10618562.2010.521129
Christopher FH, Patil SR (2002) Identification and review of sensitivity analysis methods. Risk Analysis 22(3):553–578 https://doi.org/10.1111/0272-4332.00039, https://onlinelibrary.wiley.com/doi/abs/10.1111/0272-4332.00039
Chun H, Keleş S (2010) Sparse partial least squares regression for simultaneous dimension reduction and variable selection. Journal of the Royal Statistical Society: Series B (Statistical Methodology) 72(1):3–25 https://doi.org/10.1111/j.1467-9868.2009.00723.x. https://rss.onlinelibrary.wiley.com/doi/abs/10.1111/j.1467-9868.2009.00723.x
Cibin R, Sudheer KP, Chaubey I (2010) Sensitivity and identifiability of stream flow generation parameters of the swat model. Hydrological Processes 24(9):1133–1148, https://doi.org/10.1002/hyp.756810.1002/hyp.7568. https://onlinelibrary.wiley.com/doi/abs/10.1002/hyp.7568
Clark SC, Liu E, Frazier PI, Wang J, Oktay D, Vesdapunt N (2014) Metrics optimization engine. http://yelp.github.io/MOE
Coatanéa E, Nonsiri S, Ritola T, Tumer IY, Jensen DC (2011) A framework for building dimensionless behavioral models to aid in function-based failure propagation analysis. Journal of Mechanical Design 133(12) https://doi.org/10.1115/1.4005230
Coelho RF, Breitkopf P, Knopf-Lenoir C (2008a) Model reduction for multidisciplinary optimization - application to a 2d wing. Structural and Multidisciplinary Optimization 37(1):29–48 https://doi.org/10.1007/s00158-007-0212-5
Coelho RF, Breitkopf P, Knopf-Lenoir C, Villon P (2008b) Bi-level model reduction for coupled problems. Structural and Multidisciplinary Optimization 39(4):401–418, https://doi.org/10.1007/s00158-008-0335-3
Conti S, O’Hagan A (2010) Bayesian emulation of complex multi-output and dynamic computer models. Journal of Statistical Planning and Inference 140(3):640–651, https://doi.org/10.1016/j.jspi.2009.08.006https://www.sciencedirect.com/science/article/pii/S0378375809002559
Cotter SC (1979) A screening design for factorial experiments with interactions. Biometrika 66(2):317–320 https://doi.org/10.1093/biomet/66.2.317
Couckuyt I, Deschrijver D, Dhaene T (2013) Fast calculation of multiobjective probability of improvement and expected improvement criteria for pareto optimization. Journal of Global Optimization 60(3):575–594 https://doi.org/10.1007/s10898-013-0118-2
Crombecq K, Gorissen D, Deschrijver D, Dhaene T (2011) A novel hybrid sequential design strategy for global surrogate modeling of computer experiments. SIAM Journal on Scientific Computing 33(4):1948–1974 https://doi.org/10.1137/090761811
Currin C, Mitchell T, Morris M, Ylvisaker D (1991) Bayesian prediction of deterministic functions, with applications to the design and analysis of computer experiments. Journal of the American Statistical Association 86(416):953–963 https://doi.org/10.1080/01621459.1991.10475138
Damianou A, Lawrence ND (2013) Deep Gaussian processes. In: Carvalho CM, Ravikumar P (eds) Proceedings of the Sixteenth International Conference on Artificial Intelligence and Statistics, PMLR, Scottsdale, Arizona, USA, Proceedings of Machine Learning Research, vol 31, pp 207–215. http://proceedings.mlr.press/v31/damianou13a.html
Daniel C (1973) One-at-a-time plans. Journal of the American Statistical Association 68(342):353–360 https://doi.org/10.1080/01621459.1973.10482433
de G Matthews AG, van der Wilk M, Nickson T, Fujii K, Boukouvalas A, León-Villagrá P, Ghahramani Z, Hensman J (2017) Gpflow: A gaussian process library using tensorflow. Journal of Machine Learning Research 18(40):1–6. http://jmlr.org/papers/v18/16-537.html
den Hertog D, Kleijnen JPC, Siem AYD (2006) The correct kriging variance estimated by bootstrapping. Journal of the Operational Research Society 57(4):400–409, https://doi.org/10.1057/palgrave.jors.2601997
Doebling S, Schultze J, Hemez F (2002) Overview of structural dynamics model validation activities at los alamos national laboratory. In: 43rd AIAA/ASME/ASCE/AHS/ASC Structures, Structural Dynamics, and Materials Conference, American Institute of Aeronautics and Astronautics, Denver, CO https://doi.org/10.2514/6.2002-1643
Dovi VG, Reverberi AP, Maga L, De Marchi G (1991) Improving the statistical accuracy of dimensional analysis correlations for precise coefficient estimation and optimal design of experiments. International Communications in Heat and Mass Transfer 18(4):581–590 https://doi.org/10.1016/0735-1933(91)90071-B, https://www.sciencedirect.com/science/article/pii/073519339190071B
Durantin C, Marzat J, Balesdent M (2016) Analysis of multi-objective kriging-based methods for constrained global optimization. Computational Optimization and Applications 63(3):903–926 https://doi.org/10.1007/s10589-015-9789-6
Duvenaud D, Lloyd J, Grosse R, Tenenbaum J, Zoubin G (2013) Structure discovery in nonparametric regression through compositional kernel search. In: Dasgupta S, McAllester D (eds) Proceedings of the 30th International Conference on Machine Learning, PMLR, Atlanta, Georgia, USA, Proceedings of Machine Learning Research, vol 28, pp 1166–1174, http://proceedings.mlr.press/v28/duvenaud13.html
Echard B, Gayton N, Lemaire M (2011) Ak-mcs: an active learning reliability method combining kriging and monte carlo simulation. Struct Saf 33(2):145–154
Edvall1 MM, Göran A, Holmström K (2020) TOMLAB - Version 8.7. Vallentuna, Sweden. https://tomopt.com/docs/tomlab/tomlab.php
Efroymson MA (1960) Multiple regression analysis. In: Ralston A, Wilf H (eds) mathematical methods for digital computers. John Wiley, New York, pp 191–203
Flury B (1988) Common principal components and related multivariate models, Wiley, New York
Forrester AI, Keane AJ (2009) Recent advances in surrogate-based optimization. Progress in Aerospace Sciences 45(1):50–79 https://doi.org/10.1016/j.paerosci.2008.11.001, https://www.sciencedirect.com/science/article/pii/S0376042108000766
Forrester AIJ, Keane AJ, Bressloff NW (2006) Design and analysis of “noisy” computer experiments. AIAA Journal 44(10):2331–2339 https://doi.org/10.2514/1.20068
Forrester AIJ, Sobester A, Keane AJ (2008) Engineering design via surrogate modelling: a practical guide. John Wiley & Sons Ltd. Chichester, UK
Frazier PI (2018) Bayesian optimization. In: Recent Advances in Optimization and Modeling of Contemporary Problems, INFORMS, pp 255–278 https://doi.org/10.1287/educ.2018.0188
Fricker TE, Oakley JE, Urban NM (2012) Multivariate Gaussian process emulators with nonseparable covariance structures. Technometrics 55(1):47–56. https://doi.org/10.1080/00401706.2012.715835
Frits A, Reynolds K, Weston N, Mavris D (2004) Benefits of non-dimensionalization in creation of designs of experiments for sizing torpedo systems. In: 10th AIAA/ISSMO Multidisciplinary Analysis and Optimization Conference, American Institute of Aeronautics and Astronautics, Albany, USA, pp AIAA 2004–4491 https://doi.org/10.2514/6.2004-4491
Gardner JR, Pleiss G, Bindel D, Weinberger KQ, Wilson AG (2018) Gpytorch: Blackbox matrix-matrix Gaussian process inference with GPU acceleration. arXiv preprint arXiv:180911165
Garnett R, Osborne MA, Hennig P (2014) Active learning of linear embeddings for gaussian processes. In: Proceedings of the Thirtieth Conference on Uncertainty in Artificial Intelligence, AUAI Press, Arlington, Virginia, USA, UAI’14, p 230–239
Garno J, Ouellet F, Bae S, Jackson TL, Kim NH, Haftka RT, Hughes KT, Balachandar S (2020) Calibration of reactive burn and Jones-Wilkins-Lee parameters for simulations of a detonation-driven flow experiment with uncertainty quantification. Physical Review Fluids 5(12):123201 https://doi.org/10.1103/PhysRevFluids.5.123201
Garside MJ (1965) The best sub-set in multiple regression analysis. Journal of the Royal Statistical Society: Series C (Applied Statistics) 14(2-3):196–200, https://doi.org/10.2307/2985341. https://rss.onlinelibrary.wiley.com/doi/abs/10.2307/2985341
Gattiker JR (2008) Gaussian process models for simulation analysis (GPM/SA) command, function, and data structure reference. Technical report LA-UR-08-08057, los alamos national laboratory los alamos. New Mexico, USA
Gattiker J, Klein N, Lawrence E, Hutchings G (2014) lanl/SEPIA. Los Alamos, New Mexico, USA https://doi.org/10.5281/zenodo.4048801. https://github.com/lanl/SEPIA
Getis A, Griffith DA (2002) Comparative spatial filtering in regression analysis. Geographical Analysis 34(2):130–140 https://doi.org/10.1111/j.1538-4632.2002.tb01080.x, https://onlinelibrary.wiley.com/doi/abs/10.1111/j.1538-4632.2002.tb01080.x
Ghoreishi SF, Allaire D (2018) Multi-information source constrained Bayesian optimization. Structural and Multidisciplinary Optimization 59(3):977–991, https://doi.org/10.1007/s00158-018-2115-z
Ginsbourger D, Le Riche R, Carraro L (2010) Kriging is well-suited to parallelize optimization, Springer Berlin Heidelberg, Berlin, Heidelberg, pp 131–162. https://doi.org/10.1007/978-3-642-10701-6_6
Ginsbourger D, Le Riche R, Carraro L (2007) Multi-points criterion for deterministic parallel global optimization based on kriging In: International Conference on Nonconvex Programming. Rouen, France
Giunta A (2002) Use of data sampling, surrogate models, and numerical optimization in engineering design. In: 40th AIAA Aerospace Sciences Meeting & Exhibit, American Institute of Aeronautics and Astronautics, Reno, NV, pp AIAA 2002–0538 https://doi.org/10.2514/6.2002-538
Giunta A, Watson LT (1998) A comparison of approximation modeling techniques: polynomial versus interpolating models. In: 7th AIAA/USAF/NASA/ISSMO Symposium on Multidisciplinary Analysis and Optimization, St, St. Louis, USA, p –98–4758 https://doi.org/10.2514/6.1998-4758
Giunta AA, Wojtkiewicz S, Eldred M (2003) Overview of modern design of experiments methods for computational simulations. In: 41st Aerospace Sciences Meeting and Exhibit, Reno, NV, pp AIAA–2003–0649 https://doi.org/10.2514/6.2003-649
Glaz B, Goel T, Liu L, Friedmann PP, Haftka RT (2009) Multiple-surrogate approach to helicopter rotor blade vibration reduction. AIAA Journal 47(1):271–282 https://doi.org/10.2514/1.40291
Goel T, Haftka RT, Shyy W, Queipo NV (2007) Ensemble of surrogates. Struct Multidiscip Optim 32(3):199–216. https://doi.org/10.1007/s00158-006-0051-9
Goel T, Haftka RT, Shyy W, Watson LT (2008) Pitfalls of using a single criterion for selecting experimental designs. International Journal for Numerical Methods in Engineering 75(2):127–155 https://doi.org/10.1002/nme.2242. https://onlinelibrary.wiley.com/doi/abs/10.1002/nme.2242
Gogu C, Bapanapalli SK, Haftka RT, Sankar BV (2009a) Comparison of materials for an integrated thermal protection system for spacecraft reentry. Journal of Spacecraft and Rockets 46(3):501–513 https://doi.org/10.2514/1.35669
Gogu C, Haftka RT, Bapanapalli SK, Sankar BV (2009c) Dimensionality reduction approach for response surface approximations: Application to thermal design. AIAA Journal 47(7):1700–1708 https://doi.org/10.2514/1.41414
Gogu C, Yin W, Haftka R, Ifju P, Molimard J, Le Riche R, Vautrin A (2012) Bayesian identification of elastic constants in multi-directional laminate from moiré interferometry displacement fields. Experimental Mechanics 53(4):635–648 https://doi.org/10.1007/s11340-012-9671-8
Gogu C, Haftka R, Le Riche R, Molimard J (2009b) Dimensional analysis aided surrogate modelling for natural frequencies of free plates In: 8th World Congress on Structural and Multidisciplinary Optimization. Lisbon, Portugal
Goldberger AS (1961) Stepwise least squares: residual analysis and specification error. Journal of the American Statistical Association 56(296):998–1000, https://doi.org/10.1080/01621459.1961.10482142. https://doi.org/https://www.tandfonline.com/doi/abs/10.1080/01621459.1961.10482142
Goldberger AS, Jochems DB (1961) Note on stepwise least squares. Journal of the American Statistical Association 56(293):105–110 https://doi.org/10.1080/01621459.1961.10482095, https://www.tandfonline.com/doi/abs/10.1080/01621459.1961.10482095
Goodfellow I, Bengio Y, Courville A (2016) Deep learning. MIT Press. http://www.deeplearningbook.org
Gorissen D, Couckuyt I, Demeester P, Dhaene T, Crombecq K (2010) A surrogate modeling and adaptive sampling toolbox for computer based design. Journal of Machine Learning Research 11(68):2051–2055. http://jmlr.org/papers/v11/gorissen10a.html
Gorissen D, Dhaene T, Turck FD (2009) Evolutionary model type selection for global surrogate modeling. Journal of Machine Learning Research 10(71):2039–2078, http://jmlr.org/papers/v10/gorissen09a.html
Gramacy RB (2016) lagp: Large-scale spatial modeling via local approximate gaussian processes in r. Journal of Statistical Software 72(1):1–46 https://doi.org/10.18637/jss.v072.i01, https://www.jstatsoft.org/v072/i01
Gray JS, Hwang JT, Martins JRRA, Moore KT, Naylor BA (2019) OpenMDAO: an open-source framework for multidisciplinary design, analysis, and optimization. Struct Multidiscip Optim 59(4):1075–1104. https://doi.org/10.1007/s00158-019-02211-z
Guo D, Jin Y, Ding J, Chai T (2019) Heterogeneous ensemble-based infill criterion for evolutionary multiobjective optimization of expensive problems. IEEE Transactions on Cybernetics 49(3):1012–1025. https://doi.org/10.1109/TCYB.2018.2794503
Haftka RT, Villanueva D, Chaudhuri A (2016) Parallel surrogate-assisted global optimization with expensive functions – a survey. Structural and Multidisciplinary Optimization 54(1):3–13 https://doi.org/10.1007/s00158-016-1432-3
Hazyuk I, Budinger M, Sanchez F, Gogu C (2017) Optimal design of computer experiments for surrogate models with dimensionless variables. Structural and Multidisciplinary Optimization 56(3):663–679 https://doi.org/10.1007/s00158-017-1683-7
Henkenjohann N, Kunert J (2007) An efficient sequential optimization approach based on the multivariate expected improvement criterion. Quality Engineering 19(4):267–280, https://doi.org/10.1080/08982110701621312
Hinton GE, Salakhutdinov RR (2006) Reducing the dimensionality of data with neural networks. Science 313(5786):504–507 https://doi.org/10.1126/science.1127647, https://science.sciencemag.org/content/313/5786/504
Hu Z, Du X (2015) Mixed efficient global optimization for time-dependent reliability analysis. Journal of Mechanical Design 137(5) https://doi.org/10.1115/1.4029520
Huang H (2018) Mechanisms of dimensionality reduction and decorrelation in deep neural networks. Physical Review E 98(6):062313 https://doi.org/10.1103/PhysRevE.98.062313
Iooss B, Lemaître P (2015) A review on global sensitivity analysis methods. In: Uncertainty Management in Simulation-Optimization of Complex Systems, Springer US, pp 101–122, https://doi.org/10.1007/978-1-4899-7547-8_5
Jilla CD, Miller DW (2004) Multi-objective, multidisciplinary design optimization methodology for distributed satellite systems. Journal of Spacecraft and Rockets 41(1):39–50, https://doi.org/10.2514/1.9206
Jiménez J, Ginebra J (2017) pygpgo: Bayesian optimization for python. Journal of Open Source Software 2(19):431 https://doi.org/10.21105/joss.00431
Jin R, Chen W, Simpson T (2001) Comparative studies of metamodelling techniques under multiple modelling criteria. Structural and Multidisciplinary Optimization 23(1):1–13, https://doi.org/10.1007/s00158-001-0160-4
Jin R, Chen W, Sudjianto A (2002) On sequential sampling for global metamodeling in engineering design. In: ASME 2002 International Design Engineering Technical Conferences & Computers and Information in Engineering Conference, ASME, Montreal, Canada, pp 539–548 https://doi.org/10.1115/detc2002/dac-34092
Jin R, Chen W, Sudjianto A (2004) Analytical metamodel-based global sensitivity analysis and uncertainty propagation for robust design. SAE Transactions 113:121–128, http://www.jstor.org/stable/44699914
Jin R, Du X, Chen W (2003) The use of metamodeling techniques for optimization under uncertainty. Structural and Multidisciplinary Optimization 25(2):99–16, https://doi.org/10.1007/s00158-002-0277-0
Jolliffe I (2002) Principal component analysis, 2nd edn. Springer, New York
Jones DR (2001) A taxonomy of global optimization methods based on response surfaces. Journal of Global Optimization 21(4):345–383 https://doi.org/10.1023/a:1012771025575
Jones DR, Martins JRRA (2020) The DIRECT algorithm: 25 years later. Journal of Global Optimization https://doi.org/10.1007/s10898-020-00952-6
Jones DR, Schonlau M, Welch WJ (1998) Efficient global optimization of expensive black-box functions. Journal of Global Optimization 13(4):455–492 https://doi.org/10.1023/a:1008306431147
Kambhatla N, Leen TK (1997) Dimension reduction by local principal component analysis. Neural computation 9(7):1493–1516 https://doi.org/10.1162/neco.1997.9.7.1493
Kanungo T, Mount D, Netanyahu N, Piatko C, Silverman R, Wu A (2002) An efficient k-means clustering algorithm: analysis and implementation, vol 24. https://doi.org/10.1109/TPAMI.2002.1017616
Kaufman M, Balabanov V, Grossman B, Mason WH, Watson LT, Haftka RT (1996) Multidisciplinary optimization via response surface techniques. In: 36th Israel Conference on Aerospace Sciences, Omanuth, Israel, p 57–67
Khatamsaz D, Peddareddygari L, Friedman S, Allaire D (2021) Bayesian optimization of multiobjective functions using multiple information sources. AIAA Journal 59(6):1964–1974, https://doi.org/10.2514/1.j059803
Khuri AI, Cornell JA (1996) Response surfaces: designs and analyses, 2nd edn. Dekker
Kim H, Lee TH, Kwon T (2021) Normalized neighborhood component feature selection and feasible-improved weight allocation for input variable selection. Knowl-Based Syst 218:106855
Kingma DP, Welling M (2013) Auto-encoding variational bayes. arXiv preprint arXiv:13126114
Kleijnen JP (1997) Sensitivity analysis and related analyses: a review of some statistical techniques. Journal of Statistical Computation and Simulation 57(1-4):111–142, https://doi.org/10.1080/00949659708811805
Kleijnen JPC, Sanchez SM, Lucas TW, Cioppa TM (2005) State-of-the-art review: a user’s guide to the brave new world of designing simulation experiments. INFORMS Journal on Computing 17(3):263–289 https://doi.org/10.1287/ijoc.1050.0136
Kleijnen JPC, van Beers WCM (2004) Application-driven sequential designs for simulation experiments: kriging metamodelling. Journal of the Operational Research Society 55(8):876–883, https://doi.org/10.1057/palgrave.jors.2601747
Knudde N, Dutordoir V, Herten JVD, Couckuyt I, Dhaene T (2020) Hierarchical gaussian process models for improved metamodeling. ACM Transactions on Modeling and Computer Simulation 30(4):1–17 https://doi.org/10.1145/3384470
Knudde N, van der Herten J, Dhaene T, Couckuyt I (2017) GPflowOpt: a Bayesian optimization library using TensorFlow. arXiv preprint arXiv:171103845
Koch PN, Simpson TW, Allen JK, Mistree F (1999) Statistical approximations for multidisciplinary design optimization: The problem of size. Journal of Aircraft 36(1):275–286, https://doi.org/10.2514/2.2435
Kohavi R (1995) A study of cross-validation and bootstrap for accuracy estimation and model selection. in: Fourteenth international joint conference on artificial intelligence, IJCAII, Montreal, QC, CA, pp 1137–1143
Konakli K, Sudret B (2016) Global sensitivity analysis using low-rank tensor approximations. Reliability Engineering & System Safety 156:64–83, https://doi.org/10.1016/j.ress.2016.07.012. https://www.sciencedirect.com/science/article/pii/S0951832016302423
Kristensen J, Subber W, Zhang Y, Ghosh S, Kumar NC, Khan G, Wang L (2020) Industrial applications of intelligent adaptive sampling methods for multi-objective optimization. In: Design and Manufacturing, IntechOpen https://doi.org/10.5772/intechopen.88213
Kushner HJ (1964) A new method of locating the maximum point of an arbitrary multipeak curve in the presence of noise. Journal of Basic Engineering 86(1):97–106, https://doi.org/10.1115/1.3653121
Lacey D, Steele C (2006) The use of dimensional analysis to augment design of experiments for optimization and robustification. Journal of Engineering Design 17(1):55–73, https://doi.org/10.1080/09544820500275594
LeCun Y, Bengio Y, Hinton G (2015) Deep learning. Nature 521(7553):436–444 https://doi.org/10.1038/nature14539
Leary S, Bhaskar A, Keane A (2003) Optimal orthogonal-array-based latin hypercubes. Journal of Applied Statistics 30(5):585–598 https://doi.org/10.1080/0266476032000053691
Letham B, Karrer B, Ottoni G, Bakshy E (2019) Constrained bayesian optimization with noisy experiments. Bayesian Analysis 14(2) https://doi.org/10.1214/18-ba1110
Lewis M (2007) Stepwise versus hierarchical regression: pros and cons In. Proceedings of the Annual Meeting of the Southwest Educational Research Association. San Antonio
Li C, Gupta S, Rana S, Nguyen V, Venkatesh S, Shilton A (2017) High dimensional bayesian optimization using dropout. In: Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, IJCAI-17, pp 2096–2102 https://doi.org/10.24963/ijcai.2017/291
Li CC, Lee YC (1990) A statistical procedure for model building in dimensional analysis. International journal of heat and mass transfer; A statistical procedure for model building in dimensional analysis 33(7):1566–1567
Li Z, Ruan S, Gu J, Wang X, Shen C (2016) Investigation on parallel algorithms in efficient global optimization based on multiple points infill criterion and domain decomposition. Structural and Multidisciplinary Optimization 54(4):747–773 https://doi.org/10.1007/s00158-016-1441-2
Liu H, Cai J, Ong YS (2018) Remarks on multi-output Gaussian process regression. Knowledge-Based Systems 144:102–121 https://doi.org/10.1016/j.knosys.2017.12.034https://www.sciencedirect.com/science/article/pii/S0950705117306123
Liu Q, Ding F (2018) Auxiliary model-based recursive generalized least squares algorithm for multivariate output-error autoregressive systems using the data filtering. Circuits, Systems, and Signal Processing 38(2):590–610 https://doi.org/10.1007/s00034-018-0871-z
Liu Y, Li K, Wang S, Cui P, Song X, Sun W (2021) A sequential sampling generation method for multi-fidelity model based on Voronoi region and sample density. Journal of Mechanical Design pp 1–17 https://doi.org/10.1115/1.4051014
Lophaven SN, Nielsen HB, Søndergaard J (2002) Aspects of the Matlab toolbox DACE. Roskilde, Denmark. http://www.omicron.dk/dace/dace-aspects.pdf
Lubineau G (2009) A goal-oriented field measurement filtering technique for the identification of material model parameters. Computational Mechanics 44(5):591–603, https://doi.org/10.1007/s00466-009-0392-5
Mack Y, Goel T, Shyy W, Haftka R, Queipo N (2005) Multiple surrogates for the shape optimization of bluff body-facilitated mixing. In: 43rd AIAA aerospace sciences meeting and exhibit, AIAA, Reno, NV, USA, pp AIAA–2005–0333 https://doi.org/10.2514/6.2005-333
Mandel J (1982) Use of the singular value decomposition in regression analysis. The American Statistician 36(1):15–24 https://doi.org/10.1080/00031305.1982.10482771, https://amstat.tandfonline.com/doi/abs/10.1080/00031305.1982.10482771
Mendez PF, Ordóñez F (2005) Scaling laws from statistical data and dimensional analysis. Journal of Applied Mechanics 72(5):648–657 https://doi.org/10.1115/1.1943434
Meunier M (2009) Simulation and optimization of flow control strategies for novel high-lift configurations. AIAA Journal 47(5):1145–1157 https://doi.org/10.2514/1.38245
Montgomery DC (2012) Design and analysis of experiments. John wiley & sons inc. New York, USA
Morris MD (1991) Factorial sampling plans for preliminary computational experiments. Technometrics 33(2):161–174 https://doi.org/10.1080/00401706.1991.10484804, https://amstat.tandfonline.com/doi/abs/10.1080/00401706.1991.10484804
Myers DE (1982) Matrix formulation of co-kriging. Journal of the International Association for Mathematical Geology 14(3):249–257 https://doi.org/10.1007/bf01032887
Myers RH, Montgomery DC (1995) Response surface methodology: process and product optimization using designed experiments. John wiley & sons inc, New York, USA
Nayebi A, Munteanu A, Poloczek M (2019) A framework for Bayesian optimization in embedded subspaces. In: Chaudhuri K, Salakhutdinov R (eds) Proceedings of the 36th International Conference on Machine Learning, PMLR, Proceedings of Machine Learning Research, vol 97, pp 4752–4761. http://proceedings.mlr.press/v97/nayebi19a.html
Oh C, Gavves E, Welling M (2018) BOCK : Bayesian optimization with cylindrical kernels. In: Dy J, Krause A (eds) Proceedings of the 35th International Conference on Machine Learning, PMLR, Stockholmsmässan, Stockholm Sweden, Proceedings of Machine Learning Research, vol 80, pp 3868–3877. http://proceedings.mlr.press/v80/oh18a.html
Pamadi B, Covell P, Tartabini P, Murphy K (2004) Aerodynamic characteristics and glide-back performance of langley glide-back booster. In: 22nd Applied Aerodynamics Conference and Exhibit, AIAA, Providence, USA https://doi.org/10.2514/6.2004-5382. https://arc.aiaa.org/doi/abs/10.2514/6.2004-5382
Parr JM, Keane AJ, Forrester AI, Holden CM (2012) Infill sampling criteria for surrogate-based optimization with constraint handling. Engineering Optimization 44(10):1147–1166, https://doi.org/10.1080/0305215x.2011.637556
Picard RR, Cook RD (1984) Cross-validation of regression models. J Am Stat Assoc 79 (387):575–583
Picheny V, Ginsbourger D, Roustant O, Haftka RT, Kim NH (2010) Adaptive designs of experiments for accurate approximation of a target region. Journal of Mechanical Design 132(7), https://doi.org/10.1115/1.4001873
Picheny V, Wagner T, Ginsbourger D (2013) A benchmark of kriging-based infill criteria for noisy optimization. Structural and Multidisciplinary Optimization 48(3):607–626, https://doi.org/10.1007/s00158-013-0919-4
Ponweiser W, Wagner T, Vincze M (2008) Clustered multiple generalized expected improvement: a novel infill sampling criterion for surrogate models. In 2008 IEEE Congress on Evolutionary Computation (IEEE World Congress on Computational Intelligence), Hong Kong, China, pp 3515–3522. https://doi.org/10.1109/CEC.2008.4631273
Queipo NV, Haftka RT, Shyy W, Goel T, Vaidyanathan R, Kevin Tucker P (2005) Surrogate-based analysis and optimization. Progress in Aerospace Sciences 41(1):1–28, https://doi.org/10.1016/j.paerosci.2005.02.001. https://www.sciencedirect.com/science/article/pii/S0376042105000102
Queipo NV, Verde A, Pintos S, Haftka RT (2009) Assessing the value of another cycle in gaussian process surrogate-based optimization. Structural and Multidisciplinary Optimization 39(5):459–475 https://doi.org/10.1007/s00158-008-0346-0
Shumway RH, Stoffer DS (2017) Time series regression and exploratory data analysis. Springer, New York, New York, NY https://doi.org/10.1007/0-387-36276-2_2
Raghavan B, Xia L, Breitkopf P, Rassineux A, Villon P (2013) Towards simultaneous reduction of both input and output spaces for interactive simulation-based structural design. Computer Methods in Applied Mechanics and Engineering 265:174–185 https://doi.org/10.1016/j.cma.2013.06.010. https://www.sciencedirect.com/science/article/pii/S0045782513001710
Rai R (2006) Qualitative and quantitative sequential sampling. PhD thesis University of Texas, Austin, USA
Rama RR, Skatulla S (2020) Towards real-time modelling of passive and active behaviour of the human heart using podi-based model reduction. Computers & Structures 232:105897, https://doi.org/10.1016/j.compstruc.2018.01.002. https://www.sciencedirect.com/science/article/pii/S0045794917306727
Rama RR, Skatulla S, Sansour C (2016) Real-time modelling of diastolic filling of the heart using the proper orthogonal decomposition with interpolation. International Journal of Solids and Structures 96:409–422 https://doi.org/10.1016/j.ijsolstr.2016.04.003. https://doi.org/https://www.sciencedirect.com/science/article/pii/S002076831630021X
Rana S, Li C, Gupta S, Nguyen V, Venkatesh S (2017) High dimensional Bayesian optimization with elastic Gaussian process. In: Precup D, Teh YW (eds) Proceedings of the 34th International Conference on Machine Learning, PMLR, International Convention Centre, Sydney, Australia, Proceedings of Machine Learning Research, vol 70, pp 2883–2891, http://proceedings.mlr.press/v70/rana17a.html
Ranjan P, Bingham D, Michailidis G (2008) Sequential experiment design for contour estimation from complex computer codes. Technometrics 50(4):527–541, https://doi.org/10.1198/004017008000000541
Rasmussen CE, Williams CKI (2006) Gaussian Processes for Machine Learning. The MIT Press
Rennen G, Husslage B, Van Dam ER, Den Hertog D (2010) Nested maximin latin hypercube designs. Structural and Multidisciplinary Optimization 41(3):371–395, https://doi.org/10.1007/s00158-009-0432-y
Rezende DJ, Mohamed S, Wierstra D (2014) Stochastic backpropagation and approximate inference in deep generative models. In: Xing EP, Jebara T (eds) Proceedings of the 31st International Conference on Machine Learning, PMLR, Bejing, China, Proceedings of Machine Learning Research, vol 32, pp 1278–1286, http://proceedings.mlr.press/v32/rezende14.html
Rocha H, Li W, Hahn A (2006) Principal component regression for fitting wing weight data of subsonic transports. Journal of Aircraft 43(6):1925–1936 https://doi.org/10.2514/1.21934
Rojas-Gonzalez S, Van Nieuwenhuyse I (2020) A survey on kriging-based infill algorithms for multiobjective simulation optimization. Computers & Operations Research 116:104869, https://doi.org/10.1016/j.cor.2019.104869. https://www.sciencedirect.com/science/article/pii/S0305054819303119
Roustant O, Ginsbourger D, Deville Y (2012) DiceKriging, DiceOptim: Two R packages for the analysis of computer experiments by kriging-based metamodeling and optimization. HAL preprint: 00495766 https://hal.archives-ouvertes.fr/hal-00495766
Sacks J, Welch WJ, Mitchell TJ, Wynn HP (1989) Design and analysis of computer experiments. Statistical Science 4(4):409–423. http://www.jstor.org/stable/2245858
Saltelli A (2009) Sensitivity analysis. John Wiley & Sons, New York
Saltelli A, Andres T, Homma T (1995) Sensitivity analysis of model output. performance of the iterated fractional factorial design method. Computational Statistics & Data Analysis 20(4):387–407 https://doi.org/10.1016/0167-9473(95)92843-M. https://www.sciencedirect.com/science/article/pii/016794739592843M
Samad A, Kim KY, Goel T, Haftka RT, Shyy W (2008) Multiple surrogate modeling for axial compressor blade shape optimization. Journal of Propulsion and Power 24(2):301–310, https://doi.org/10.2514/1.28999
Sanchez F, Budinger M, Hazyuk I (2017) Dimensional analysis and surrogate models for the thermal modeling of multiphysics systems. Applied Thermal Engineering 110:758–771, https://doi.org/10.1016/j.applthermaleng.2016.08.117. https://www.sciencedirect.com/science/article/pii/S1359431116314776
Schölkopf B, Smola A, Müller KR (1998) Nonlinear component analysis as a kernel eigenvalue problem. Neural computation 10(5):1299–1319 https://doi.org/10.1162/089976698300017467
Schölkopf B, Smola AJ (2002) Learning with kernels the MIT press. Cambridge, USA
Simpson T, Booker A, Ghosh D, Giunta A, Koch P, Yang RJ (2004) Approximation methods in multidisciplinary analysis and optimization: a panel discussion. Structural and Multidisciplinary Optimization 27(5) https://doi.org/10.1007/s00158-004-0389-9
Simpson T, Poplinski J, Koch PN, Allen J (2001a) Metamodels for computer-based engineering design: survey and recommendations. Engineering with Computers 17(2):129–150, https://doi.org/10.1007/pl00007198
Simpson T, Poplinski J, Koch PN, Allen J (2001b) Metamodels for computer-based engineering design: survey and recommendations. Engineering with Computers 17(2):129–150, https://doi.org/10.1007/pl00007198
Simpson TW, Lin DKJ, Chen W (2001c) Sampling strategies for computer experiments: design and analysis. International Journal of Reliability and Applications 2(3):209–240
Singh G, Viana FAC, Subramaniyan AK, Wang L, DeCesare D, Khan G, Wiggs G (2012) Multimodal particle swarm optimization: enhancements and applications. In: 53rd AIAA / ASME / ASCE / AHS / ASC Structures, Structural Dynamics and Materials Conference, AIAA, Honolulu, USA https://doi.org/10.2514/6.2012-5570
Smola AJ, Schölkopf B (2004) A tutorial on support vector regression. Stat Comput 14 (3):199–222. https://doi.org/10.1023/B:STCO.0000035301.49549.88
Snelson E, Ghahramani Z (2006) Variable noise and dimensionality reduction for sparse gaussian processes. In: Proceedings of the Twenty-Second Conference on Uncertainty in Artificial Intelligence, AUAI Press, Arlington, Virginia, USA, pp 461–468
Snoek J, Swersky K, Zemel R, Adams R (2014) Input warping for bayesian optimization of non-stationary functions. In: Xing EP, Jebara T (eds) Proceedings of the 31st International Conference on Machine Learning, PMLR, Bejing, China, Proceedings of Machine Learning Research, vol 32, pp 1674–1682. http://proceedings.mlr.press/v32/snoek14.html
Sobol I (1993) Sensitivity estimates for non-linear mathematical models. Mathematical Modelling and Computational Experiment 4:407–414
Song W, Keane A (2006) Parameter screening using impact factors and surrogate-based ANOVA techniques. In: 11th AIAA/ISSMO Multidisciplinary Analysis and Optimization Conference, American Institute of Aeronautics and Astronautics, Portsmouth, Virginia, pp AIAA 2006–7088 https://doi.org/10.2514/6.2006-7088
Sonin AA (2001) The physical basis of dimensional analysis, 2nd edn Massachusetts Institute of Technology, Cambridge, MA
Stander N, Roux W, Giger M, Redhe M, Fedorova N, Haarhoff J (2004) A comparison of metamodeling techniques for crashworthiness optimization. In: 10th AIAA/ISSMO Multidisciplinary Analysis and Optimization Conference, American Institute of Aeronautics and Astronautics, Albany, NY https://doi.org/10.2514/6.2004-4489
Stein ML (1999) Interpolation of spatial data: some theory for kriging. Springer
Tabachnick BG, Fidell LS (2007) Experimental designs using ANOVA. Thomson/Brooks/Cole Belmont, CA
Tang B (1993) Orthogonal array-based latin hypercubes. Journal of the American Statistical Association 88(424):1392–1397 https://doi.org/10.1080/01621459.1993.10476423, https://www.tandfonline.com/doi/abs/10.1080/01621459.1993.10476423
Tenne Y, Izui K, Nishiwaki S (2010) Dimensionality-reduction frameworks for computationally expensive problems. In: IEEE Congress on evolutionary computation, IEEE. Barcelona, Spain, pp 1–8. https://doi.org/10.1109/CEC.2010.5586251
Turner CJ, Crawford RH, Campbell MI (2007) Multidimensional sequential sampling for nurbs-based metamodel development. Engineering with Computers 23(3):155–174, https://doi.org/10.1007/s00366-006-0051-9
Utans J, Moody J (1991) Selecting neural network architectures via the prediction risk: application to corporate bond rating prediction. In: IEEE 1St international conference on AI applications on wall street, IEEE, New York, NY, USA, pp 35-41
Vaidyanathan R, Goel T, Haftka R, Quiepo N, Shyy W, Tucker K (2004) Global sensitivity and trade-off analyses for multi-objective liquid rocket injector design. In: 40th AIAA/ASME/SAE/ASEE Joint Propulsion Conference and Exhibit, Fort Lauderdale, FL, pp AIAA 2004–4007 https://doi.org/10.2514/6.2004-4007
van Beek A, Ghumman UF, Munshi J, Tao S, Chien T, Balasubramanian G, Plumlee M, Apley D, Chen W (2021) Scalable adaptive batch sampling in simulation-based design with heteroscedastic noise. J Mech Des 143(3):031709. (15 pages) https://doi.org/10.1115/1.4049134
van Beers WC, Kleijnen JP (2008) Customized sequential designs for random simulation experiments: kriging metamodeling and bootstrapping. European Journal of Operational Research 186(3):1099–1113 https://doi.org/10.1016/j.ejor.2007.02.035. https://www.sciencedirect.com/science/article/pii/S0377221707002895
Varma S, Simon R (2006) Bias in error estimation when using cross-validation for model selection. BMC Bioinforma 7(91):1290–1300. https://doi.org/10.1186/1471-2105-7-91
Vaschy A (1882) Sur les lois de similitude en physique. Annales Té,légraphiques 19:25
Vazquez E, Villemonteix J, Sidorkiewicz M, Walter É (2008) Global optimization based on noisy evaluations: an empirical study of two statistical approaches. Journal of Physics: Conference Series 135:012100. p 1088. https://doi.org/10.1088/1742-6596/135/1/012100
Venter G, Haftka RT, Starnes JH (1998) Construction of response surface approximations for design optimization. AIAA Journal 36(12):2242–2249 https://doi.org/10.2514/2.333
Viana FAC (2011) SURROGATES Toolbox User’s Guide. Gainesville, FL, USA, version 3.0 edn. https://sites.google.com/site/srgtstoolbox
Viana FAC (2016) A tutorial on Latin hypercube design of experiments. Quality and Reliability Engineering International 32(5):1975–1985 https://doi.org/10.1002/qre.1924, https://onlinelibrary.wiley.com/doi/abs/10.1002/qre.1924
Viana FAC, Haftka RT (2009) Cross validation can estimate how well prediction variance correlates with error. AIAA Journal 47(9):2266–2270 https://doi.org/10.2514/1.42162
Viana FAC, Haftka RT, Steffen V (2009) Multiple surrogates: how cross-validation errors can help us to obtain the best predictor. Structural and Multidisciplinary Optimization 39(4):439–457 https://doi.org/10.1007/s00158-008-0338-0
Viana FAC, Haftka RT, Watson LT (2012) Sequential sampling for contour estimation with concurrent function evaluations. Structural and Multidisciplinary Optimization 45(4):615–618, https://doi.org/10.1007/s00158-011-0733-9
Viana FAC, Haftka RT, Watson LT (2013) Efficient global optimization algorithm assisted by multiple surrogate techniques. Journal of Global Optimization 56(2):669–689, https://doi.org/10.1007/s10898-012-9892-5
Viana FAC, Simpson TW, Balabanov V, Toropov V (2014) Metamodeling in multidisciplinary design optimization: how far have we really come? AIAA Journal 52(4):670–690, https://doi.org/10.2514/1.J052375. https://arc.aiaa.org/doi/abs/10.2514/1.J052375
Viana FAC, Venter G, Balabanov V (2010) An algorithm for fast optimal latin hypercube design of experiments. International Journal for Numerical Methods in Engineering 82(2):135–156, https://doi.org/10.1002/nme.2750. https://onlinelibrary.wiley.com/doi/abs/10.1002/nme.2750
Vignaux GA, Scott JL (1999) Simplifying regression models using dimensional analysis. Austalia & New Zealand Journal of Statistics 41(2):31–41
Villemonteix J, Vazquez E, Walter E (2008) An informational approach to the global optimization of expensive-to-evaluate functions. Journal of Global Optimization 44(4):509–534, https://doi.org/10.1007/s10898-008-9354-2
von Luxburg U (2007) A tutorial on spectral clustering. Statistics and Computing 17(4):395–416 https://doi.org/10.1007/s11222-007-9033-z
Wang DQ (2011) Least squares-based recursive and iterative estimation for output error moving average systems using data filtering. IET Control Theory & Applications 5:1648–1657(9), https://doi.org/10.1049/iet-cta.2010.0416. https://digital-library.theiet.org/content/journals/10.1049/iet-cta.2010.0416
Wang GG, Shan S (2004) Design space reduction for multi-objective optimization and robust design optimization problems. SAE Transactions 113:101–110, http://www.jstor.org/stable/44699911
Wang GG, Shan S (2006) Review of metamodeling techniques in support of engineering design optimization. Journal of Mechanical Design 129(4):370–380 https://doi.org/10.1115/1.2429697
Wang W, Huang Y, Wang Y, Wang L (2014) Generalized autoencoder: A neural network framework for dimensionality reduction. In: Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pp 490–497
Wang Z, Li C, Jegelka S, Kohli P (2017) Batched high-dimensional Bayesian optimization via structural kernel learning. In: Precup D, Teh YW (eds) Proceedings of the 34th International Conference on Machine Learning, PMLR, Proceedings of Machine Learning Research, vol 70, pp 3656–3664. http://proceedings.mlr.press/v70/wang17h.html
Welch WJ, Buck RJ, Sacks J, Wynn HP, Mitchell TJ, Morris MD (1992) Screening, predicting, and computer experiments. Technometrics 34(1):15–25 https://doi.org/10.1080/00401706.1992.10485229, https://amstat.tandfonline.com/doi/abs/10.1080/00401706.1992.10485229
Wold H (1983) Systems analysis by partial least squares. Iiasa collaborative paper, IIASA, IIASA, Laxenburg, Austria. http://pure.iiasa.ac.at/id/eprint/2336/
Wolpert DH (1996) The lack of a priori distinctions between learning algorithms. Neural Computation 8(7):1341–1390 https://doi.org/10.1162/neco.1996.8.7.1341
Wu J, Toscano-Palmerin S, Frazier PI, Wilson AG (2020) Practical multi-fidelity Bayesian optimization for hyperparameter tuning. In: Adams RP, Gogate V (eds) Proceedings of The 35th Uncertainty in Artificial Intelligence Conference, PMLR, Tel Aviv, Israel, Proceedings of Machine Learning Research, vol 115, pp 788–798, http://proceedings.mlr.press/v115/wu20a.html
Yang Y (2003) Regression with multiple candidate models: selecting or mixing? Stat Sin 13 (3):783–809. https://doi.org/10.1.1.17.753
Yang W, Wang K, Zuo W (2012) Neighborhood component feature selection for high-dimensional data. JCP 7(1):161–168
Yee TW (2000) Vector splines and other vector smoothers. In: eds (ed) Proc. Computational Statistics COMPSTAT 2000, Physica-Verlag HD, Bethlehem, J.G., Van der Heijden, P.G.M, pp 529–534 https://doi.org/10.1007/978-3-642-57678-2_75
Yondo R, Andrés E, Valero E (2018) A review on design of experiments and surrogate models in aircraft real-time and many-query aerodynamic analyses. Progress in Aerospace Sciences 96:23–61 https://doi.org/10.1016/j.paerosci.2017.11.003. https://www.sciencedirect.com/science/article/pii/S0376042117300611
Zaefferer M, Stork J, Friese M, Fischbach A, Naujoks B, Bartz-Beielstein T (2014) Efficient global optimization for combinatorial problems. In: Proceedings of the 2014 Annual Conference on Genetic and Evolutionary Computation, ACM https://doi.org/10.1145/2576768.2598282
Zahm O, Constantine PG, Prieur C, Marzouk YM (2020) Gradient-based dimension reduction of multivariate vector-valued functions. SIAM Journal on Scientific Computing 42(1):A534–A558, https://doi.org/10.1137/18M1221837
Zerpa LE, Queipo NV, Pintos S, Salager JL (2005) An optimization methodology of alkaline-surfactant-polymer flooding processes using field scale numerical simulation and multiple surrogates. J Pet Sci Eng 7(3–4):197–208. https://doi.org/10.1016/j.petrol.2005.03.002
Zhang P (1993) Model selection via multifold cross validation. The Annals of Statistics 21 (1):299–313. https://doi.org/10.1214/aos/1176349027
Zhou Q, Wang Y, Choi SK, Jiang P, Shao X, Hu J (2017) A sequential multi-fidelity metamodeling approach for data regression. Knowledge-Based Systems 134:199–212, https://doi.org/10.1016/j.knosys.2017.07.033https://doi.org/https://www.sciencedirect.com/science/article/pii/S0950705117303556https://doi.org/https://www.sciencedirect.com/science/article/pii/S0950705117303556
Acknowledgements
This paper is dedicated to Dr. Raphael T. Haftka. All three authors of this paper had the honor to have Dr. Haftka as their doctoral supervisor. Dr. Haftka’s legacy in multi-disciplinary optimization, surrogate modeling, uncertainty quantification, structural analysis, and other engineering disciplines is everlasting. His influence in the education of scientists and engineers will remain alive for many years to come.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
The authors declare that they have no conflict of interest.
Additional information
Responsible Editor: Nam Ho Kim
Replication of results
This is a literature review paper. Therefore, we do not provide results that could be replicated.
Publisher’s note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Special Issue dedicated to Dr. Raphael T. Haftka
Rights and permissions
About this article
Cite this article
Viana, F.A.C., Gogu, C. & Goel, T. Surrogate modeling: tricks that endured the test of time and some recent developments. Struct Multidisc Optim 64, 2881–2908 (2021). https://doi.org/10.1007/s00158-021-03001-2
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00158-021-03001-2