Skip to main content
SpringerLink
Log in

Remarks on multi-fidelity surrogates

  • RESEARCH PAPER
  • Published:
Structural and Multidisciplinary Optimization Aims and scope Submit manuscript

Abstract

Different multi-fidelity surrogate (MFS) frameworks have been used for optimization or uncertainty quantification. This paper investigates differences between various MFS frameworks with the aid of examples including algebraic functions and a borehole example. These MFS include three Bayesian frameworks using 1) a model discrepancy function, 2) low fidelity model calibration and 3) a comprehensive approach combining both. Three counterparts in simple frameworks are also included, which have the same functional form but can be built with ready-made surrogates. The sensitivity of frameworks to the choice of design of experiments (DOE) is investigated by repeating calculations with 100 different DOEs. Computational cost savings and accuracy improvement over a single fidelity surrogate model are investigated as a function of the ratio of the sampling costs between low and high fidelity simulations. For the examples considered, MFS frameworks were found to be more useful for saving computational time rather than improving accuracy. For the Hartmann 6 function example, the maximum cost saving for the same accuracy was 86 %, while the maximum accuracy improvement for the same cost was 51 %. It was also found that DOE can substantially change the relative standing of different frameworks. The cross-validation error appears to be a reasonable candidate for estimating poor MFS frameworks for a specific problem but it does not perform well compared to choosing single fidelity surrogates.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

Abbreviations

δ :

A discrepancy data set for given ρ (δ = y H  − ρ y c L ).

δ(x):

An unknown true value of a discrepancy function value at x.

\( \widehat{\delta}\left(\mathbf{x}\right) \) :

A predictor for a discrepancy function value at x.

Δ(x):

A prior model (GP model) for predicting a discrepancy function value at x for Bayesian MFS frameworks. Note that a discrepancy function can be a function for given ρ.

Δ(x)|δ :

An updated discrepancy function model with a discrepancy data set.

λ :

A roughness parameter vector.

θ :

A calibrated parameter vector (a constant vector).

ρ :

A scalar for a low fidelity function.

σ :

A process standard deviation.

ξ(x):

A vector of shape functions.

b :

A coefficient vector (a constant vector).

q :

A calibration variable vector (a variable vector).

x :

An input variable vector (a variable vector).

y :

A data set.

y(x):

An unknown true function value at x.

ŷ(x):

A surrogate predictor for a function value at x.

Y(x):

A prior model for predicting a function at x for Bayesian frameworks and Kriging surrogate. A prior model is a fitted GP model (Z(x)) parameterized with a linear polynomial trend function, which approximates the true function, and the corresponding uncertainty in the trend function. The parameters of a prior model are found by samples for maximum consistency.

Y(x)|y :

An updated model with a data set. The trend function and the corresponding uncertainty of a prior model are updated with samples.

y H :

A high fidelity data set.

y i h :

The i-th data point of a high fidelity data set.

y H (x):

An unknown true high fidelity function value at x.

ŷ H (x):

An MFS predictor for a high fidelity function value at x.

Y H (x):

A prior model (GP model) for predicting a high fidelity function value at x for Bayesian MFS frameworks. This model can be a linear combination of a low fidelity model and a discrepancy function model.

Y H (x)|y H , y L :

An updated high fidelity model with low and high fidelity data sets.

y L :

A low fidelity data set.

y c L :

A low fidelity data set at locations common to those of high fidelity data points.

y i l :

The i-th data point of a low fidelity data set.

y L (x):

An unknown true low fidelity function value at x.

ŷ L (x):

An MFS predictor for a low fidelity function value at x.

ŷ L (x, θ):

An MFS predictor for a low fidelity function value at x for a given calibrated parameter vector θ.

Y L (x):

A prior model (GP model) for predicting a low fidelity function value at x for Bayesian MFS frameworks.

Y L (x, θ):

A prior model (GP model) for a prediction of a low fidelity function value at x for a given calibrated parameter vector θ.

Y L (x)|y L :

An updated low fidelity model with a low fidelity data set.

References

  • Acar E, Rais-Rohani M (2009) Ensemble of metamodels with optimized weight factors. Struct Multidiscip Optim 37(3):279–294

    Article  Google Scholar 

  • Balabanov V, Haftka RT, Grossman B, Mason WH, Watson LT (1998) Multifidelity response surface model for HSCT wing bending material weight. In Proceedings of 7th AIAA/USAF/NASA/ISSMO Symposium on Multidisciplinary Analysis and Optimization

  • Bayarri MJ, Berger JO, Paulo R, Sacks J, Cafeo JA, Cavendish J, Tu J (2007) A framework for validation of computer models. Technometrics 49:138–154

    Article  MathSciNet  Google Scholar 

  • Coppe A, Pais MJ, Haftka RT, Kim NH (2012) Using a simple crack growth model in predicting remaining useful life. J Aircr 49(6):1965–1973

    Article  Google Scholar 

  • Ellis MW, Mathews EH (2001) A new simplified thermal design tool for architects. Build Environ 36(9):1009–1021

    Article  Google Scholar 

  • Fischer CC, Grandhi RV (2014) Utilizing an adjustment factor to scale between multiple fidelities within a design process: a stepping stone to dialable fidelity design. In 16th AIAA Non-Deterministic Approaches Conference

  • Fischer CC, Grandhi RV (2015) A surrogate-based adjustment factor approach to multi-fidelity design optimization. In 17th AIAA Non-Deterministic Approaches Conference

  • Forrester AI, Sóbester A, Keane AJ (2007) Multi-fidelity optimization via surrogate modelling. Proc R Soc A 463(2088):3251–3269

  • Han Z, Zimmerman R, Görtz S (2012) Alternative Cokriging method for variable-fidelity surrogate modeling. AIAA J 50(5):1205–1210

    Article  Google Scholar 

  • Higdon D, Kennedy M, Cavendish JC, Cafeo JA, Ryne RD (2004) Combining field data and computer simulations for calibration and prediction. SIAM J Sci Comput 26(2):448–466

    Article  MathSciNet  MATH  Google Scholar 

  • Jin R, Chen W, Sudjianto A (2005) An efficient algorithm for constructing optimal design of computer experiments. J Stat Plann Inference 134(1):268–287

    Article  MathSciNet  MATH  Google Scholar 

  • Kennedy MC, O’Hagan A (2001). Supplementary details on Bayesian calibration of computer models. Internal Report. URL http://www.shef.ac.uk/~st1ao/ps/calsup.ps

  • Kennedy MC, O’Hagan A (2000) Predicting the output from a complex computer code when fast approximations are available. Biometrika 87(1):1–13

    Article  MathSciNet  MATH  Google Scholar 

  • Kennedy MC, O’Hagan A (2001b) Bayesian calibration of computer models. J R Stat Soc Ser B (Stat Methodol) 63(3):425–464

    Article  MathSciNet  MATH  Google Scholar 

  • Knill DL, Giunta AA, Baker CA, Grossman B, Mason WH, Haftka RT, Watson LT (1999) Response surface models combining linear and Euler aerodynamics for supersonic transport design. J Aircr 36(1):75–86

    Article  Google Scholar 

  • Kosonen R, Shemeikka J (1997) The use of a simple simulation tool for energy analysis. VTT Building Technology

  • Kuya Y, Takeda K, Zhang X, Forrester AIJ (2011) Multifidelity surrogate modeling of experimental and computational aerodynamic data sets. AIAA J 49(2):289–298

    Article  Google Scholar 

  • Le Gratiet L (2013) Multi-fidelity Gaussian process regression for computer experiments (Doctoral dissertation, Université Paris-Diderot-Paris VII)

  • Lee S, Youn BD, Sodano HA (2008) Computer model calibration and design comparison on piezoelectric energy harvester. In Proc. 12th AIAA/ISSMO Multidisciplinary Analysis and Optimization Conference (Victoria)

  • Lophaven SN, Nielsen HB, Søndergaard J (2002) DACE-A Matlab Kriging toolbox, version 2.0

  • Martin JD, Simpson TW (2005) Use of kriging models to approximate deterministic computer models. AIAA J 43(4):853–863

    Article  Google Scholar 

  • Mason BH, Haftka RT, Johnson ER, Farley GL (1998) Variable complexity design of composite fuselage frames by response surface techniques. Thin-Walled Struct 32(4):235–261

    Article  Google Scholar 

  • McFarland J, Mahadevan S, Romero V, Swiler L (2008) Calibration and uncertainty analysis for computer simulations with multivariate output. AIAA J 46(5):1253–1265

    Article  Google Scholar 

  • Morris MD, Mitchell TJ, Ylvisaker D (1993) Bayesian design and analysis of computer experiments: use of derivatives in surface prediction. Technometrics 35(3):243–255

    Article  MathSciNet  MATH  Google Scholar 

  • O’Hagan A (1992) Some Bayesian numerical analysis. Bayesian Stat 4(345–363):4–2

    MathSciNet  Google Scholar 

  • Owen AK, Daugherty A, Garrard D, Reynolds HC, Wright RD (1998) A parametric starting study of an axial-centrifugal gas turbine engine using a one-dimensional dynamic engine model and comparisons to experimental results: part 2—simulation calibration and trade-off study. In ASME 1998 International Gas Turbine and Aeroengine Congress and Exhibition (pp. V002T02A012-V002T02A012). Am Soc Mech Eng

  • Prudencio EE, Schulz KW (2012) The parallel C++ statistical library ‘QUESO’: Quantification of Uncertainty for Estimation, Simulation and Optimization. In Euro-Par 2011: Parallel Processing Workshops (pp. 398–407). Springer Berlin Heidelberg

  • Qian PZ, Wu CJ (2008) Bayesian hierarchical modeling for integrating low-accuracy and high-accuracy experiments. Technometrics 50(2):192–204

    Article  MathSciNet  Google Scholar 

  • Rasmussen CE (2004) Gaussian processes in machine learning. In Advanced lectures on machine learning (pp. 63–71). Springer Berlin Heidelberg

  • Ryu JS, Kim MS, Cha KJ, Lee TH, Choi DH (2002) Kriging interpolation methods in geostatistics and DACE model. KSME Int J 16(5):619–632

    Article  Google Scholar 

  • Sacks J, Welch WJ, Mitchell TJ, Wynn HP (1989) Design and analysis of computer experiments. Stat Sci 409–423

  • Sanchez E, Pintos S, Queipo NV (2008) Toward an optimal ensemble of kernel-based approximations with engineering applications. Struct Multidiscip Optim 36(3):247–261

    Article  Google Scholar 

  • Viana FA, Haftka RT (2009) Cross validation can estimate how well prediction variance correlates with error. AIAA J 47(9):2266–2270

    Article  Google Scholar 

  • Viana FA, Haftka RT, Steffen V Jr (2009) Multiple surrogates: how cross-validation errors can help us to obtain the best predictor. Struct Multidiscip Optim 39(4):439–457

    Article  Google Scholar 

  • Xiong S, Qian PZ, Wu CJ (2013) Sequential design and analysis of high-accuracy and low-accuracy computer codes. Technometrics 55(1):37–46

    Article  MathSciNet  Google Scholar 

  • Yoo MY, Choi JH (2013) Probabilistic calibration of computer model and application to reliability analysis of elasto-plastic insertion problem. Trans Korean Soc Mech Eng A 37(9):1133–1140

    Article  Google Scholar 

  • Zheng L, Hedrick TL, Mittal R (2013) A multi-fidelity modelling approach for evaluation and optimization of wing stroke aerodynamics in flapping flight. J Fluid Mech 721:118–154

Download references

Acknowledgments

This work is supported by the U.S. Department of Energy, National Nuclear Security Administration, Advanced Simulation and Computing Program, as a Cooperative Agreement under the Predictive Science Academic Alliance Program, under Contract No. DE-NA0002378.

Author information

Authors and Affiliations

  1. Department of Mechanical and Aerospace Engineering, University of Florida, PO Box 116250, Gainesville, FL, 32611-6250, USA

    Chanyoung Park, Raphael T. Haftka & Nam H. Kim

Authors
  1. Chanyoung Park
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Raphael T. Haftka
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Nam H. Kim
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Chanyoung Park.

Appendices

Appendix A: Statistical study of the 1-D function with 100 DOEs

The 1D function examples presented in the main text were selected to illustrate differences between frameworks because they exhibit distinctive differences. The Bayesian discrepancy function gave a significantly better prediction than the simple prediction and the Bayesian comprehensive framework gave very different calibration results than the other frameworks using calibration. The results were observed from the selected DOEs from randomly generated 100 DOEs. In this section, we show statistical representation for all 100 DOEs. The same trend functions and parameter bounds were used for fitting MFSs. For generating samples, we intendedly increased randomness by allowing a small number of iterations for LHS for initial low and high fidelity samples to see various cases.

Figure 12a presents the 100 RMSEs of the discrepancy based frameworks, SDR and BDR in the form of a boxplot. The center red line indicates the median (50 %) and the bottom and top of boxes are lower (25 %) and upper (75 %) quartiles of 100 RMSEs. The default distances of upper and lower whiskers between the upper and lower quartiles are 1.5w where w is the inter quartile distance which is the distance between upper and lower quartiles. If maximum or minimum samples are within the default bounds, whiskers are adjusted. Samples out of the default bounds are considered as outliers and they indicated with red crosses. The Bayesian discrepancy framework significantly outperforms the simple discrepancy framework statistically. The median RMSEs of BDR and SDR are 3.5 and 0.6, respectively. The correlation coefficient of the RMSEs of the two frameworks is 0.25 which is weak that one bad DOE for one framework may be a good DOE for the other. However, the mean and standard deviation of BDR are significantly smaller than SDR that SDR is better than BDR for a few DOEs with negligible difference. The means of regression scalar ρ of SDR and BDR are 0.54 and 1.92, respectively. That tells the ways of estimating ρ are responsible for the difference. There was weak correlation between RMSEs of the two frameworks but the worst DOEs for BDR are also bad DOEs for SDR RMSEs. The worst and the second worst RMSEs of BDR are 6.7 and 4.5 and the corresponding RSMEs of SDR are respectively 5.8 and 4.3.

Figure 12b shows box plots of the frameworks using calibration, SCR, BCR, SCDR and BCDR. In terms of median RMSE, all frameworks show similar performance. Table 9 shows the correlation coefficients between RMSEs of the four frameworks. Unlike the previous discrepancy frameworks, they have very strong correlations. That means that a good DOE for one is highly likely to be a good DOE for the others that a good DOE is a necessary condition for constructing a good MFS. We searched for a DOE that was good for a framework and bad for another, but we could not find a single such DOE from the 100 DOEs.

Appendix B: Median RMSEs for Different Sample Size Ratios

In the previous example section, only the median RMSEs for cost ratio of 30 were presented for both the Hartmann 6 function example and the borehole function example since there is no noticeable difference in the behavior between cost ratios of 30 and 10. In this appendix, the median RMSEs for cost ratio of 10 are presented in Fig. 13 for the Hartmann 6 function example and in Fig. 14 for the borehole function example.

Fig. 12
figure 12

Performance variations for 100 DOEs. a Frameworks using a discrepancy function. b Frameworks using calibration

Full size image
Fig. 13
figure 13

Median RMSEs for different sample size ratios and cost ratio of 10. a Best frameworks for 56H total budget. b Best frameworks for 28H total budget. c Discrepancy based frameworks for 56H total budget. d Discrepancy based frameworks for 28H total budget. e Frameworks using calibration for 56H total budget. f Frameworks using calibration for 28H total budget

Full size image
Fig. 14
figure 14

Median RMSEs for different sample size ratios and cost ratio of 10. a Best frameworks for 10H total budget. b Best frameworks for 5H total budget. c Discrepancy based frameworks for 10H total budget. d Discrepancy based frameworks for 5H total budget. e Frameworks using calibration for 10H total budget. f Frameworks using calibration for 5H total budget

Full size image
Table 9 Correlation coefficients between 100 RMSEs of the four frameworks
Full size table

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Park, C., Haftka, R.T. & Kim, N.H. Remarks on multi-fidelity surrogates. Struct Multidisc Optim 55, 1029–1050 (2017). https://doi.org/10.1007/s00158-016-1550-y

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00158-016-1550-y

Keywords

Search

Navigation