Advertisement

Structural and Multidisciplinary Optimization

, Volume 60, Issue 4, pp 1619–1644 | Cite as

Review of statistical model calibration and validation—from the perspective of uncertainty structures

  • Guesuk Lee
  • Wongon Kim
  • Hyunseok OhEmail author
  • Byeng D. YounEmail author
  • Nam H. Kim
Review Article
  • 371 Downloads

Abstract

Computer-aided engineering (CAE) is now an essential instrument that aids in engineering decision-making. Statistical model calibration and validation has recently drawn great attention in the engineering community for its applications in practical CAE models. The objective of this paper is to review the state-of-the-art and trends in statistical model calibration and validation, based on the available extensive literature, from the perspective of uncertainty structures. After a brief discussion about uncertainties, this paper examines three problem categories—the forward problem, the inverse problem, and the validation problem—in the context of techniques and applications for statistical model calibration and validation.

Keywords

Forward problem Inverse problem Validation problem Uncertainty quantification Statistical model calibration Validity check 

Notes

Funding information

This work was supported by a grant from the Institute of Advanced Machinery and Design at Seoul National University (SNU-IAMD) and the Korea Institute of Machinery and Materials (Project No.: NK213E).

Compliance with ethical standards

Conflict of interest

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

References

  1. Abbas T, Morgenthal G (2016) Framework for sensitivity and uncertainty quantification in the flutter assessment of bridges. Probab Eng Mech 43:91–105.  https://doi.org/10.1016/j.probengmech.2015.12.007 CrossRefGoogle Scholar
  2. Acar E, Rais-Rohani M (2009) Ensemble of metamodels with optimized weight factors. Struct Multidiscip Optim 37:279–294CrossRefGoogle Scholar
  3. Adamowski K (1985) Nonparametric kernel estimation of flood frequencies. Water Resour Res 21:1585–1590CrossRefGoogle Scholar
  4. American Society of Mechanical Engineers (2009) Standard for verification and validation in computational fluid dynamics and heat transfer. American Society of Mechanical Engineers, New York City, NY, USAGoogle Scholar
  5. Anderson MG, Bates PD (2001) Model validation: perspectives in hydrological science. Chichester, West Sussex, England. John Wiley & Sons, LtdGoogle Scholar
  6. Anderson TW, Darling DA (1952) Asymptotic theory of certain “goodness of fit” criteria based on stochastic processes Ann Math Stat 23:193–212Google Scholar
  7. Anderson TW, Darling DA (1954) A test of goodness of fit. J Am Stat Assoc 49:765–769CrossRefzbMATHGoogle Scholar
  8. Arendt PD, Apley DW, Chen W (2012a) Quantification of model uncertainty: calibration, model discrepancy, and identifiability. J Mech Des 134:100908CrossRefGoogle Scholar
  9. Arendt PD, Apley DW, Chen W, Lamb D, Gorsich D (2012b) Improving identifiability in model calibration using multiple responses. J Mech Des 134:100909CrossRefGoogle Scholar
  10. Arlot S, Celisse A (2010) A survey of cross-validation procedures for model selection. Stat Surv 4:40–79CrossRefMathSciNetzbMATHGoogle Scholar
  11. Atamturktur S, Hegenderfer J, Williams B, Egeberg M, Lebensohn R, Unal C (2014) A resource allocation framework for experiment-based validation of numerical models. Mech Adv Mater Struct 22:641–654Google Scholar
  12. Au S, Beck JL (1999) A new adaptive importance sampling scheme for reliability calculations. Struct Saf 21:135–158CrossRefGoogle Scholar
  13. Babuska I, Oden JT (2004) Verification and validation in computational engineering and science: basic concepts. Comput Methods Appl Mech Eng 193:4057–4066CrossRefMathSciNetzbMATHGoogle Scholar
  14. Babuška I, Nobile F, Tempone R (2008) A systematic approach to model validation based on Bayesian updates and prediction related rejection criteria. Comput Methods Appl Mech Eng 197:2517–2539.  https://doi.org/10.1016/j.cma.2007.08.031 CrossRefzbMATHGoogle Scholar
  15. Bae H-R, Grandhi RV, Canfield RA (2003) Uncertainty quantification of structural response using evidence theory. AIAA J 41:2062–2068CrossRefGoogle Scholar
  16. Bae H-R, Grandhi RV, Canfield RA (2004) An approximation approach for uncertainty quantification using evidence theory. Reliab Eng Syst Saf 86:215–225CrossRefGoogle Scholar
  17. Bae HR, Grandhi RV, Canfield RA (2006) Sensitivity analysis of structural response uncertainty propagation using evidence theory. Struct Multidiscip Optim 31:270–279.  https://doi.org/10.1007/s00158-006-0606-9 CrossRefGoogle Scholar
  18. Bao N, Wang C (2015) A Monte Carlo simulation based inverse propagation method for stochastic model updating. Mech Syst Signal Process 61:928–944Google Scholar
  19. Baroni G, Tarantola S (2014) A general probabilistic framework for uncertainty and global sensitivity analysis of deterministic models: a hydrological case study. Environ Model Softw 51:26–34.  https://doi.org/10.1016/j.envsoft.2013.09.022 CrossRefGoogle Scholar
  20. Bayarri MJ et al (2007) A framework for validation of computer models Technometrics 49:138–154Google Scholar
  21. Benek JA, Kraft EM, Lauer RF (1998) Validation issues for engine-airframe integration. AIAA J 36:759–764CrossRefGoogle Scholar
  22. Bianchi M (1997) Testing for convergence: evidence from non-parametric multimodality tests. J Appl Econ 12:393–409CrossRefGoogle Scholar
  23. Borg A, Paulsen Husted B, Njå O (2014) The concept of validation of numerical models for consequence analysis. Reliab Eng Syst Saf 125:36–45.  https://doi.org/10.1016/j.ress.2013.09.009 CrossRefGoogle Scholar
  24. Box GE, Meyer RD (1986) An analysis for unreplicated fractional factorials. Technometrics 28:11–18CrossRefMathSciNetzbMATHGoogle Scholar
  25. Bucher CG (1988) Adaptive sampling—an iterative fast Monte Carlo procedure. Struct Saf 5:119–126CrossRefGoogle Scholar
  26. Buranathiti T, Cao J, Chen W, Baghdasaryan L, Xia ZC (2006) Approaches for model validation: methodology and illustration on a sheet metal flanging process. J Manuf Sci Eng 128:588–597CrossRefGoogle Scholar
  27. Cacuci DG, Ionescu-Bujor M (2004) A comparative review of sensitivity and uncertainty analysis of large-scale systems—II: statistical methods. Nucl Sci Eng 147:204–217CrossRefGoogle Scholar
  28. Campbell K (2006) Statistical calibration of computer simulations. Reliab Eng Syst Saf 91:1358–1363CrossRefGoogle Scholar
  29. Cao R, Cuevas A, Manteiga WG (1994) A comparative study of several smoothing methods in density estimation. Comput Stat Data Anal 17:153–176CrossRefzbMATHGoogle Scholar
  30. Carley H, Taylor M (2002) A new proof of Sklar’s theorem. In: Distributions with given marginals and statistical modelling. Springer, pp 29–34Google Scholar
  31. Cha S-H (2007) Comprehensive survey on distance/similarity measures between probability density functions. Int J Math Models Methods Appl Sci 4:300–307Google Scholar
  32. Charnes A, Frome E, Yu P-L (1976) The equivalence of generalized least squares and maximum likelihood estimates in the exponential family. J Am Stat Assoc 71:169–171CrossRefMathSciNetzbMATHGoogle Scholar
  33. Chen W, Baghdasaryan L, Buranathiti T, Cao J (2004) Model validation via uncertainty propagation and data transformations. AIAA J 42:1406–1415CrossRefGoogle Scholar
  34. Chen W, Jin R, Sudjianto A (2005) Analytical variance-based global sensitivity analysis in simulation-based design under uncertainty. J Mech Des 127:875–886CrossRefGoogle Scholar
  35. Chen W, Xiong Y, Tsui K-L, Wang S (2008) A design-driven validation approach using Bayesian prediction models. J Mech Des 130:021101CrossRefGoogle Scholar
  36. Cho H, Bae S, Choi K, Lamb D, Yang R-J (2014) An efficient variable screening method for effective surrogate models for reliability-based design optimization. Struct Multidiscip Optim 50:717–738CrossRefGoogle Scholar
  37. Choi K, Youn BD, Yang R-J (2001) Moving least square method for reliability-based design optimization. In: 4th world congress of structural and multidisciplinary optimization. Liaoning Electronic Press, Shenyang, PRC, pp 4–8Google Scholar
  38. Choi J, An D, Won J (2010a) Bayesian approach for structural reliability analysis and optimization using the Kriging Dimension Reduction Method. J Mech Des 132:051003CrossRefGoogle Scholar
  39. Choi S-S, Cha S-H, Tappert CC (2010b) A survey of binary similarity and distance measures. J Syst Cybern Inform 8:43–48Google Scholar
  40. Christopher Frey H, Patil SR (2002) Identification and review of sensitivity analysis methods. Risk Anal 22:553–578CrossRefGoogle Scholar
  41. Clarke SM, Griebsch JH, Simpson TW (2005) Analysis of support vector regression for approximation of complex engineering analyses. J Mech Des 127:1077–1087CrossRefGoogle Scholar
  42. Cooke RM, Goossens LH (2004) Expert judgement elicitation for risk assessments of critical infrastructures. J Risk Res 7:643–656CrossRefGoogle Scholar
  43. Crestaux T, Le Maıˆtre O, Martinez J-M (2009) Polynomial chaos expansion for sensitivity analysis. Reliab Eng Syst Saf 94:1161–1172CrossRefGoogle Scholar
  44. da Silva Hack P, Schwengber ten Caten C (2012) Measurement uncertainty: literature review and research trends. IEEE Trans Instrum Meas 61:2116–2124.  https://doi.org/10.1109/tim.2012.2193694 CrossRefGoogle Scholar
  45. Dowding KJ, Pilch M, Hills RG (2008) Formulation of the thermal problem. Comput Methods Appl Mech Eng 197:2385–2389CrossRefzbMATHGoogle Scholar
  46. Dubourg V, Sudret B, Deheeger F (2013) Metamodel-based importance sampling for structural reliability analysis. Probab Eng Mech 33:47–57.  https://doi.org/10.1016/j.probengmech.2013.02.002 CrossRefGoogle Scholar
  47. Echard B, Gayton N, Lemaire M, Relun N (2013) A combined importance sampling and kriging reliability method for small failure probabilities with time-demanding numerical models. Reliab Eng Syst Saf 111:232–240.  https://doi.org/10.1016/j.ress.2012.10.008 CrossRefGoogle Scholar
  48. Egeberg M (2014) Optimal design of validation experiments for calibration and validation of complex numerical models, Master Thesis, Clemson University, Clemson, SC, USAGoogle Scholar
  49. Eldred M, Burkardt J (2009) Comparison of non-intrusive polynomial chaos and stochastic collocation methods for uncertainty quantification. In: Proceedings of the 47th AIAA aerospace sciences meeting and exhibit, vol 1. pp 976, Orlando, FL, January 5-9Google Scholar
  50. Eldred MS, Swiler LP, Tang G (2011) Mixed aleatory-epistemic uncertainty quantification with stochastic expansions and optimization-based interval estimation. Reliab Eng Syst Saf 96:1092–1113CrossRefGoogle Scholar
  51. Epanechnikov VA (1969) Non-parametric estimation of a multivariate probability density. Theor Probab Appl 14:153–158CrossRefMathSciNetGoogle Scholar
  52. Fang S-E, Ren W-X, Perera R (2012) A stochastic model updating method for parameter variability quantification based on response surface models and Monte Carlo simulation. Mech Syst Signal Process 33:83–96CrossRefGoogle Scholar
  53. Farajpour I, Atamturktur S (2012) Error and uncertainty analysis of inexact and imprecise computer models. J Comput Civ Eng 27:407–418CrossRefGoogle Scholar
  54. Fender J, Duddeck F, Zimmermann M (2014) On the calibration of simplified vehicle crash models. Struct Multidiscip Optim 49:455–469CrossRefGoogle Scholar
  55. Ferson S, Ginzburg LR (1996) Different methods are needed to propagate ignorance and variability. Reliabil Eng Syst Saf 54:133–144CrossRefGoogle Scholar
  56. Ferson S, Oberkampf WL (2009) Validation of imprecise probability models. Int J Reliab Saf 3:3–22CrossRefGoogle Scholar
  57. Ferson S, Joslyn CA, Helton JC, Oberkampf WL, Sentz K (2004) Summary from the epistemic uncertainty workshop: consensus amid diversity. Reliab Eng Syst Saf 85:355–369CrossRefGoogle Scholar
  58. Ferson S, Oberkampf WL, Ginzburg L (2008) Model validation and predictive capability for the thermal challenge problem. Comput Methods Appl Mech Eng 197:2408–2430.  https://doi.org/10.1016/j.cma.2007.07.030 CrossRefzbMATHGoogle Scholar
  59. Fonseca JR, Friswell MI, Mottershead JE, Lees AW (2005) Uncertainty identification by the maximum likelihood method. J Sound Vib 288:587–599CrossRefGoogle Scholar
  60. Fu Y, Zhan Z, Yang R-J (2010) A study of model validation method for dynamic systems. SAE 2010-01-0419Google Scholar
  61. Gan Y et al (2014) A comprehensive evaluation of various sensitivity analysis methods: a case study with a hydrological model. Environ Model Softw 51:269–285.  https://doi.org/10.1016/j.envsoft.2013.09.031 CrossRefGoogle Scholar
  62. Geers TL (1984) An objective error measure for the comparison of calculated and measured transient response histories. Shock and Vibration Information Center The Shock and Vibration Bull 54, Pt 2, p 99–108(SEE N 85-18388 09-39)Google Scholar
  63. Genest C, Rivest L-P (1993) Statistical inference procedures for bivariate Archimedean copulas. J Am Stat Assoc 88:1034–1043CrossRefMathSciNetzbMATHGoogle Scholar
  64. Ghanem RG, Doostan A, Red-Horse J (2008) A probabilistic construction of model validation. Comput Methods Appl Mech Eng 197:2585–2595CrossRefzbMATHGoogle Scholar
  65. Giunta A, McFarland J, Swiler L, Eldred M (2006) The promise and peril of uncertainty quantification using response surface approximations. Struct Infrastruct Eng 2:175–189CrossRefGoogle Scholar
  66. Goel T, Haftka RT, Shyy W, Queipo NV (2007) Ensemble of surrogates. Struct Multidiscip Optim 33:199–216CrossRefGoogle Scholar
  67. Goel T, Hafkta RT, Shyy W (2009) Comparing error estimation measures for polynomial and kriging approximation of noise-free functions. Struct Multidiscip Optim 38:429–442CrossRefGoogle Scholar
  68. Gomes HM, Awruch AM (2004) Comparison of response surface and neural network with other methods for structural reliability analysis. Struct Saf 26:49–67CrossRefGoogle Scholar
  69. Gorissen D, Couckuyt I, Demeester P, Dhaene T, Crombecq K (2010) A surrogate modeling and adaptive sampling toolbox for computer based design. J Mach Learn Res 11:2051–2055Google Scholar
  70. Govers Y, Link M (2010) Stochastic model updating—covariance matrix adjustment from uncertain experimental modal data. Mech Syst Signal Process 24:696–706.  https://doi.org/10.1016/j.ymssp.2009.10.006 CrossRefGoogle Scholar
  71. Halder A, Bhattacharya R (2011) Model validation: a probabilistic formulation. In: Decision and Control and European Control Conference (CDC-ECC), 2011 50th IEEE Conference on. IEEE, pp 1692–1697Google Scholar
  72. Hanss M, Turrin S (2010) A fuzzy-based approach to comprehensive modeling and analysis of systems with epistemic uncertainties. Struct Saf 32:433–441CrossRefGoogle Scholar
  73. Harmel RD, Smith PK, Migliaccio KW (2010) Modifying goodness-of-fit indicators to incorporate both measurement and model uncertainty in model calibration and validation. Trans ASABE 53:55–63CrossRefGoogle Scholar
  74. Hasselman T, Yap K, Lin C-H, Cafeo J (2005) A case study in model improvement for vehicle crashworthiness simulation. In: Proceedings of the 23rd International Modal Analysis Conference, Orlando, FL, USA, January 31 - February 3Google Scholar
  75. Hearst MA, Dumais ST, Osman E, Platt J, Scholkopf B (1998) Support vector machines. IEEE Intell Syst Appl 13:18–28CrossRefGoogle Scholar
  76. Helton JC (1997) Uncertainty and sensitivity analysis in the presence of stochastic and subjective uncertainty. J Stat Comput Simul 57:3–76CrossRefzbMATHGoogle Scholar
  77. Helton JC, Burmaster DE (1996) Guest editorial: treatment of aleatory and epistemic uncertainty in performance assessments for complex systems. Reliab Eng Syst Saf 54:91–94CrossRefGoogle Scholar
  78. Helton JC, Davis FJ (2003) Latin hypercube sampling and the propagation of uncertainty in analyses of complex systems. Reliab Eng Syst Saf 81:23–69CrossRefGoogle Scholar
  79. Helton JC, Oberkampf WL (2004) Alternative representations of epistemic uncertainty. Reliab Eng Syst Saf 85:1–10.  https://doi.org/10.1016/j.ress.2004.03.001 CrossRefGoogle Scholar
  80. Helton JC, Johnson JD, Oberkampf WL (2004) An exploration of alternative approaches to the representation of uncertainty in model predictions. Reliab Eng Syst Saf 85:39–71.  https://doi.org/10.1016/j.ress.2004.03.025 CrossRefGoogle Scholar
  81. Helton JC, Davis F, Johnson JD (2005) A comparison of uncertainty and sensitivity analysis results obtained with random and Latin hypercube sampling. Reliab Eng Syst Saf 89:305–330CrossRefGoogle Scholar
  82. Helton JC, Johnson JD, Oberkampf W, Sallaberry CJ (2006a) Sensitivity analysis in conjunction with evidence theory representations of epistemic uncertainty. Reliab Eng Syst Saf 91:1414–1434CrossRefGoogle Scholar
  83. Helton JC, Johnson JD, Sallaberry CJ, Storlie CB (2006b) Survey of sampling-based methods for uncertainty and sensitivity analysis. Reliab Eng Syst Saf 91:1175–1209.  https://doi.org/10.1016/j.ress.2005.11.017 CrossRefGoogle Scholar
  84. Helton JC, Johnson JD, Oberkampf WL, Sallaberry CJ (2010) Representation of analysis results involving aleatory and epistemic uncertainty. Int J Gen Syst 39:605–646CrossRefMathSciNetzbMATHGoogle Scholar
  85. Hemez F, Atamturktur HS, Unal C (2010) Defining predictive maturity for validated numerical simulations. Comput Struct 88:497–505CrossRefGoogle Scholar
  86. Higdon D, Nakhleh C, Gattiker J, Williams B (2008) A Bayesian calibration approach to the thermal problem. Comput Methods Appl Mech Eng 197:2431–2441CrossRefzbMATHGoogle Scholar
  87. Hill WJ, Hunter WG (1966) A review of response surface methodology: a literature survey. Technometrics 8:571–590CrossRefMathSciNetGoogle Scholar
  88. Hills RG (2006) Model validation: model parameter and measurement uncertainty. J Heat Transf 128:339.  https://doi.org/10.1115/1.2164849 CrossRefGoogle Scholar
  89. Hills RG, Pilch M, Dowding KJ, Red-Horse J, Paez TL, Babuška I, Tempone R (2008) Validation challenge workshop. Comput Methods Appl Mech Eng 197:2375–2380.  https://doi.org/10.1016/j.cma.2007.10.016 MathSciNetCrossRefzbMATHGoogle Scholar
  90. Hoffman FO, Hammonds JS (1994) Propagation of uncertainty in risk assessments: the need to distinguish between uncertainty due to lack of knowledge and uncertainty due to variability. Risk Anal 14:707–712CrossRefGoogle Scholar
  91. Homma T, Saltelli A (1996) Importance measures in global sensitivity analysis of nonlinear models. Reliab Eng Syst Saf 52:1–17CrossRefGoogle Scholar
  92. Hu C, Youn BD (2011) Adaptive-sparse polynomial chaos expansion for reliability analysis and design of complex engineering systems. Struct Multidiscip Optim 43:419–442CrossRefMathSciNetzbMATHGoogle Scholar
  93. Hu Z, Hu C, Mourelatos ZP, Mahadevan S (2019) Model discrepancy quantification in simulation-based design of dynamical systems. J Mech Des 141:011401CrossRefGoogle Scholar
  94. Hua X, Ni Y, Chen Z, Ko J (2008) An improved perturbation method for stochastic finite element model updating. Int J Numer Methods Eng 73:1845–1864CrossRefzbMATHGoogle Scholar
  95. Huntington D, Lyrintzis C (1998) Improvements to and limitations of Latin hypercube sampling. Probab Eng Mech 13:245–253CrossRefGoogle Scholar
  96. Hurtado J, Barbat AH (1998) Monte Carlo techniques in computational stochastic mechanics. Arch Comput Methods Eng 5:3–29CrossRefMathSciNetGoogle Scholar
  97. Jiang X, Mahadevan S (2007) Bayesian risk-based decision method for model validation under uncertainty. Reliab Eng Syst Saf 92:707–718CrossRefGoogle Scholar
  98. Jiang X, Mahadevan S (2008) Bayesian validation assessment of multivariate computational models. J Appl Stat 35:49–65CrossRefMathSciNetzbMATHGoogle Scholar
  99. Jiang X, Mahadevan S (2009a) Bayesian inference method for model validation and confidence extrapolation. J Appl Stat 36:659–677CrossRefMathSciNetzbMATHGoogle Scholar
  100. Jiang X, Mahadevan S (2009b) Bayesian structural equation modeling method for hierarchical model validation. Reliab Eng Syst Saf 94:796–809.  https://doi.org/10.1016/j.ress.2008.08.008 CrossRefGoogle Scholar
  101. Jiang X, Mahadevan S (2011) Wavelet spectrum analysis approach to model validation of dynamic systems. Mech Syst Signal Process 25:575–590CrossRefGoogle Scholar
  102. Jiang X, Yuan Y, Mahadevan S, Liu X (2013a) An investigation of Bayesian inference approach to model validation with non-normal data. J Stat Comput Simul 83:1829–1851CrossRefMathSciNetGoogle Scholar
  103. Jiang Z, Chen W, Fu Y, Yang R-J (2013b) Reliability-based design optimization with model bias and data uncertainty. SAE Int J Mater Manuf 6:502–516CrossRefGoogle Scholar
  104. Joe H (1990) Families of min-stable multivariate exponential and multivariate extreme value distributions. Stat Probab lett 9:75–81CrossRefMathSciNetzbMATHGoogle Scholar
  105. Joe H, Hu T (1996) Multivariate distributions from mixtures of max-infinitely divisible distributions. J Multivar Anal 57:240–265CrossRefMathSciNetzbMATHGoogle Scholar
  106. Johnson NL (1949) Systems of frequency curves generated by methods of translation. Biometrika 36:149–176CrossRefMathSciNetzbMATHGoogle Scholar
  107. Jung BC, Lee D-H, Youn BD (2009) Optimal design of constrained-layer damping structures considering material and operational condition variability. AIAA J 47:2985–2995CrossRefGoogle Scholar
  108. Jung BC, Lee D, Youn BD, Lee S (2011) A statistical characterization method for damping material properties and its application to structural-acoustic system design. J Mech Sci Technol 25:1893–1904CrossRefGoogle Scholar
  109. Jung BC, Park J, Oh H, Kim J, Youn BD (2014) A framework of model validation and virtual product qualification with limited experimental data based on statistical inference. Struct Multidiscip Optim 51:573–583Google Scholar
  110. Jung BC, Yoon H, Oh H, Lee G, Yoo M, Youn BD, Huh YC (2016) Hierarchical model calibration for designing piezoelectric energy harvester in the presence of variability in material properties and geometry. Struct Multidiscip Optim 53:161–173CrossRefGoogle Scholar
  111. Kang S-C, Koh H-M, Choo JF (2010) An efficient response surface method using moving least squares approximation for structural reliability analysis. Probab Eng Mech 25:365–371CrossRefGoogle Scholar
  112. Kat C-J, Els PS (2012) Validation metric based on relative error. Math Comput Model Dyn Syst 18:487–520CrossRefMathSciNetzbMATHGoogle Scholar
  113. Kaymaz I (2005) Application of kriging method to structural reliability problems. Struct Saf 27:133–151CrossRefGoogle Scholar
  114. Kennedy MC, O’Hagan A (2001) Bayesian calibration of computer models. J R Stat Soc B 63(3):425–464Google Scholar
  115. Kersaudy P, Sudret B, Varsier N, Picon O, Wiart J (2015) A new surrogate modeling technique combining kriging and polynomial chaos expansions – application to uncertainty analysis in computational dosimetry. J Comput Phys 286:103–117.  https://doi.org/10.1016/j.jcp.2015.01.034 MathSciNetCrossRefzbMATHGoogle Scholar
  116. Khodaparast HH, Mottershead JE, Friswell MI (2008) Perturbation methods for the estimation of parameter variability in stochastic model updating. Mech Syst Signal Process 22:1751–1773.  https://doi.org/10.1016/j.ymssp.2008.03.001 CrossRefGoogle Scholar
  117. Khodaparast HH, Mottershead JE, Badcock KJ (2011) Interval model updating with irreducible uncertainty using the kriging predictor. Mech Syst Signal Process 25:1204–1226.  https://doi.org/10.1016/j.ymssp.2010.10.009 CrossRefGoogle Scholar
  118. Kim T, Lee G, Youn BD (2018) Uncertainty characterization under measurement errors using maximum likelihood estimation: cantilever beam end-to-end UQ test problem Struct Multidiscip Optim 59:323–333Google Scholar
  119. Kleijnen JPC, Mehdad E (2014) Multivariate versus univariate Kriging metamodels for multi-response simulation models. Eur J Oper Res 236:573–582.  https://doi.org/10.1016/j.ejor.2014.02.001 MathSciNetCrossRefzbMATHGoogle Scholar
  120. Kokkolaras M, Hulbert G, Papalambros P, Mourelatos Z, Yang R, Brudnak M, Gorsich D (2013) Towards a comprehensive framework for simulation-based design validation of vehicle systems. Int J Veh Des 61:233–248CrossRefGoogle Scholar
  121. Kutluay E, Winner H (2014) Validation of vehicle dynamics simulation models – a review. Veh Syst Dyn 52:186–200.  https://doi.org/10.1080/00423114.2013.868500 CrossRefGoogle Scholar
  122. Kwaśniewski L (2009) On practical problems with verification and validation of computational models. Arch Civ Eng 55:323–346Google Scholar
  123. Lancaster P, Salkauskas K (1981) Surfaces generated by moving least squares methods. Math Comput 37:141–158CrossRefMathSciNetzbMATHGoogle Scholar
  124. Lee SH, Chen W (2009) A comparative study of uncertainty propagation methods for black-box-type problems. Struct Multidiscip Optim 37:239–253CrossRefGoogle Scholar
  125. Lee JH, Gard K (2014) Vehicle–soil interaction: testing, modeling, calibration and validation. J Terrramech 52:9–21.  https://doi.org/10.1016/j.jterra.2013.12.001 CrossRefGoogle Scholar
  126. Lee I, Choi KK, Du L, Gorsich D (2008a) Dimension reduction method for reliability-based robust design optimization. Comput Struct 86:1550–1562.  https://doi.org/10.1016/j.compstruc.2007.05.020 CrossRefzbMATHGoogle Scholar
  127. Lee I, Choi KK, Du L, Gorsich D (2008b) Inverse analysis method using MPP-based dimension reduction for reliability-based design optimization of nonlinear and multi-dimensional systems. Comput Methods Appl Mech Eng 198:14–27.  https://doi.org/10.1016/j.cma.2008.03.004 CrossRefzbMATHGoogle Scholar
  128. Lee I, Choi K, Gorsich D (2010) System reliability-based design optimization using the MPP-based dimension reduction method. Struct Multidiscip Optim 41:823–839CrossRefGoogle Scholar
  129. Lee I, Choi K, Zhao L (2011) Sampling-based RBDO using the stochastic sensitivity analysis and dynamic kriging method. Struct Multidiscip Optim 44:299–317CrossRefMathSciNetzbMATHGoogle Scholar
  130. Lee T-R, Greene MS, Jiang Z, Kopacz AM, Decuzzi P, Chen W, Liu WK (2014) Quantifying uncertainties in the microvascular transport of nanoparticles. Biomech Model Mechanobiol 13:515–526CrossRefGoogle Scholar
  131. Lee G, Yi G, Youn BD (2018) Special issue: a comprehensive study on enhanced optimization-based model calibration using gradient information. Struct Multidiscip Optim 57:2005–2025CrossRefMathSciNetGoogle Scholar
  132. Levin D (1998) The approximation power of moving least-squares. Math Comput Am Math Soc 67:1517–1531CrossRefMathSciNetzbMATHGoogle Scholar
  133. Li C, Mahadevan S (2016) Role of calibration, validation, and relevance in multi-level uncertainty integration. Reliab Eng Syst Saf 148:32–43.  https://doi.org/10.1016/j.ress.2015.11.013 CrossRefGoogle Scholar
  134. Li W, Chen W, Jiang Z, Lu Z, Liu Y (2014) New validation metrics for models with multiple correlated responses. Reliab Eng Syst Saf 127:1–11.  https://doi.org/10.1016/j.ress.2014.02.002 CrossRefGoogle Scholar
  135. Liang B, Mahadevan S (2011) Error and uncertainty quantification and sensitivity analysis in mechanics computational models Int J Uncertain Quantif 11:147–161Google Scholar
  136. Liang C, Mahadevan S (2014) Bayesian framework for multidisciplinary uncertainty quantification and optimization. In: SciTech, 16th AIAA Non-Deterministic Approaches Conference, pp 2014-1347Google Scholar
  137. Liang C, Mahadevan S, Sankararaman S (2015) Stochastic multidisciplinary analysis under epistemic uncertainty. J Mech Des 137:021404CrossRefGoogle Scholar
  138. Lima Azevedo C, Ciuffo B, Cardoso JL, Ben-Akiva ME (2015) Dealing with uncertainty in detailed calibration of traffic simulation models for safety assessment. Transp Res Part C: Emerg Technol 58:395–412.  https://doi.org/10.1016/j.trc.2015.01.029 CrossRefGoogle Scholar
  139. Ling Y, Mahadevan S (2013) Quantitative model validation techniques: new insights. Reliab Eng Syst Saf 111:217–231CrossRefGoogle Scholar
  140. Liu F, Bayarri M, Berger J, Paulo R, Sacks J (2008) A Bayesian analysis of the thermal challenge problem. Comput Methods Appl Mech Eng 197:2457–2466CrossRefzbMATHGoogle Scholar
  141. Liu Y, Chen W, Arendt P, Huang H-Z (2011) Toward a better understanding of model validation metrics. J Mech Des 133:071005CrossRefGoogle Scholar
  142. Mahadevan S, Rebba R (2005) Validation of reliability computational models using Bayes networks. Reliab Eng Syst Saf 87:223–232CrossRefGoogle Scholar
  143. Manfren M, Aste N, Moshksar R (2013) Calibration and uncertainty analysis for computer models – a meta-model based approach for integrated building energy simulation. Appl Energy 103:627–641CrossRefGoogle Scholar
  144. Mara TA, Tarantola S (2012) Variance-based sensitivity indices for models with dependent inputs. Reliab Eng Syst Saf 107:115–121.  https://doi.org/10.1016/j.ress.2011.08.008 CrossRefGoogle Scholar
  145. Mares C, Mottershead JE, Friswell MI (2006) Stochastic model updating: part 1—theory and simulated example. Mech Syst Signal Process 20:1674–1695.  https://doi.org/10.1016/j.ymssp.2005.06.006 CrossRefGoogle Scholar
  146. Massey FJ Jr (1951) The Kolmogorov-Smirnov test for goodness of fit. J Am Stat Assoc 46:68–78CrossRefzbMATHGoogle Scholar
  147. McCusker JR, Danai K, Kazmer DO (2010) Validation of dynamic models in the time-scale domain. J Dyn Syst Meas Control 132:061402.  https://doi.org/10.1115/1.4002479 CrossRefGoogle Scholar
  148. McFarland J, Mahadevan S (2008a) Error and variability characterization in structural dynamics modeling. Comput Methods Appl Mech Eng 197:2621–2631.  https://doi.org/10.1016/j.cma.2007.07.029 CrossRefzbMATHGoogle Scholar
  149. McFarland J, Mahadevan S (2008b) Multivariate significance testing and model calibration under uncertainty. Comput Methods Appl Mech Eng 197:2467–2479CrossRefzbMATHGoogle Scholar
  150. McFarland J, Mahadevan S, Romero V, Swiler L (2008) Calibration and uncertainty analysis for computer simulations with multivariate output. AIAA J 46:1253–1265CrossRefGoogle Scholar
  151. McKay M, Meyer M (2000) Critique of and limitations on the use of expert judgements in accident consequence uncertainty analysis. Radiat Prot Dosim 90:325–330CrossRefGoogle Scholar
  152. McNeil AJ (2008) Sampling nested Archimedean copulas. J Stat Comput Simul 78:567–581CrossRefMathSciNetzbMATHGoogle Scholar
  153. McNeil AJ, Nešlehová J (2009) Multivariate Archimedean copulas, d-monotone functions and ℓ1-norm symmetric distributions Ann Stat 37:3059–3097Google Scholar
  154. Melchers R (1989) Importance sampling in structural systems. Struct Saf 6:3–10CrossRefGoogle Scholar
  155. Mongiardini M, Ray MH, Anghileri M. Development of a software for the comparison of curves during the verification and validation of numerical models. In: Proceedings of the 7th European LS-DYNA Conference, Salzburg, 2009Google Scholar
  156. Mongiardini M, Ray MH, Anghileri M (2010) Acceptance criteria for validation metrics in roadside safety based on repeated full-scale crash tests. Int J Reliab Saf 4:69–88CrossRefGoogle Scholar
  157. Mongiardini M, Ray M, Plaxico C (2013) Development of a programme for the quantitative comparison of a pair of curves. Int J Comput Appl Technol 46:128–141CrossRefGoogle Scholar
  158. Montgomery DC (2008) Design and analysis of experiments. John Wiley & SonsGoogle Scholar
  159. Moon M-Y, Choi K, Cho H, Gaul N, Lamb D, Gorsich D (2017) Reliability-based design optimization using confidence-based model validation for insufficient experimental data. J Mech Des 139:031404CrossRefGoogle Scholar
  160. Moon M-Y, Cho H, Choi K, Gaul N, Lamb D, Gorsich D (2018) Confidence-based reliability assessment considering limited numbers of both input and output test data. Struct Multidiscip Optim 57:2027–2043CrossRefMathSciNetGoogle Scholar
  161. Moore R, Lodwick W (2003) Interval analysis and fuzzy set theory. Fuzzy Sets Syst 135:5–9CrossRefMathSciNetzbMATHGoogle Scholar
  162. Mosegaard K, Sambridge M (2002) Monte Carlo analysis of inverse problems. Inverse Probl 18:29–54Google Scholar
  163. Mousaviraad SM, He W, Diez M, Stern F (2013) Framework for convergence and validation of stochastic uncertainty quantification and relationship to deterministic verification and validation. Int J Uncertain Quantif 3:371–395Google Scholar
  164. Mullins J, Ling Y, Mahadevan S, Sun L, Strachan A (2016) Separation of aleatory and epistemic uncertainty in probabilistic model validation. Reliab Eng Syst Saf 147:49–59.  https://doi.org/10.1016/j.ress.2015.10.003 CrossRefGoogle Scholar
  165. Murmann R, Harzheim L, Dominico S, Immel R (2016) CoSi: correlation of signals—a new measure to assess the correlation of history response curves. Mech Syst Signal Process 80:482–502Google Scholar
  166. Murray_smith D (1998) Methods for the external validation of continuous system simulation models: a review. Math Comput Model Dyn Syst 4:5–31CrossRefGoogle Scholar
  167. Myers RH, Montgomery DC, Anderson-Cook CM (1995) Response surface methodology: process and product optimization using designed experiments. Hoboken, NJ, USA. John Wiley & Sons, IncGoogle Scholar
  168. Myers RH, Montgomery DC, Anderson-Cook CM (2016) Response surface methodology: process and product optimization using designed experiments. John Wiley & SonsGoogle Scholar
  169. Myung IJ (2003) Tutorial on maximum likelihood estimation. J Math Psychol 47:90–100CrossRefMathSciNetzbMATHGoogle Scholar
  170. Najm HN (2009) Uncertainty quantification and polynomial chaos techniques in computational fluid dynamics. Annu Rev Fluid Mech 41:35–52.  https://doi.org/10.1146/annurev.fluid.010908.165248 MathSciNetCrossRefzbMATHGoogle Scholar
  171. Nelsen RB (2002) Concordance and copulas: a survey. In: Distributions with given marginals and statistical modelling. Springer, pp 169–177Google Scholar
  172. Newey WK, West KD (1987) Hypothesis testing with efficient method of moments estimation. Int Econ Rev 28:777–787Google Scholar
  173. Oberkampf WL, Barone MF (2006) Measures of agreement between computation and experiment: validation metrics. J Comput Phys 217:5–36.  https://doi.org/10.1016/j.jcp.2006.03.037 CrossRefzbMATHGoogle Scholar
  174. Oberkampf WL, Trucano TG (2002) Verification and validation in computational fluid dynamics. Prog Aerosp Sci 38:209–272CrossRefGoogle Scholar
  175. Oberkampf WL, Trucano TG (2008) Verification and validation benchmarks. Nucl Eng Des 238:716–743.  https://doi.org/10.1016/j.nucengdes.2007.02.032 CrossRefGoogle Scholar
  176. Oberkampf WL, DeLand SM, Rutherford BM, Diegert KV, Alvin KF (2002) Error and uncertainty in modeling and simulation. Reliab Eng Syst Saf 75:333–357CrossRefGoogle Scholar
  177. Oberkampf WL, Helton JC, Joslyn CA, Wojtkiewicz SF, Ferson S (2004a) Challenge problems: uncertainty in system response given uncertain parameters. Reliab Eng Syst Saf 85:11–19CrossRefGoogle Scholar
  178. Oberkampf WL, Trucano TG, Hirsch C (2004b) Verification, validation, and predictive capability in computational engineering and physics. Appl Mech Rev 57:345–384CrossRefGoogle Scholar
  179. Oden JT, Prudencio EE, Bauman PT (2013) Virtual model validation of complex multiscale systems: applications to nonlinear elastostatics. Comput Methods Appl Mech Eng 266:162–184.  https://doi.org/10.1016/j.cma.2013.07.011 MathSciNetCrossRefzbMATHGoogle Scholar
  180. Oh H, Kim J, Son H, Youn BD, Jung BC (2016) A systematic approach for model refinement considering blind and recognized uncertainties in engineered product development. Struct Multidiscip Optim 54:1527–1541CrossRefGoogle Scholar
  181. Oladyshkin S, Nowak W (2012) Data-driven uncertainty quantification using the arbitrary polynomial chaos expansion. Reliab Eng Syst Saf 106:179–190.  https://doi.org/10.1016/j.ress.2012.05.002 CrossRefGoogle Scholar
  182. Oliver TA, Terejanu G, Simmons CS, Moser RD (2015) Validating predictions of unobserved quantities. Comput Methods Appl Mech Eng 283:1310–1335.  https://doi.org/10.1016/j.cma.2014.08.023 CrossRefGoogle Scholar
  183. Pan H, Xi Z, Yang R-J (2016) Model uncertainty approximation using a copula-based approach for reliability based design optimization. Struct Multidiscip Optim 54:1543–1556CrossRefMathSciNetGoogle Scholar
  184. Panchenko V (2005) Goodness-of-fit test for copulas. Phys A: Stat Mech Appl 355:176–182CrossRefMathSciNetGoogle Scholar
  185. Park I, Grandhi RV (2014) A Bayesian statistical method for quantifying model form uncertainty and two model combination methods. Reliab Eng Syst Saf 129:46–56CrossRefGoogle Scholar
  186. Park B, Turlach B (1992) Practical performance of several data driven bandwidth selectors. Université catholique de Louvain, Center for Operations Research and Econometrics (CORE)Google Scholar
  187. Park I, Amarchinta HK, Grandhi RV (2010) A Bayesian approach for quantification of model uncertainty. Reliab Eng Syst Saf 95:777–785.  https://doi.org/10.1016/j.ress.2010.02.015 CrossRefGoogle Scholar
  188. Park C, Kim NH, Haftka RT (2015) The effect of ignoring dependence between failure modes on evaluating system reliability. Struct Multidiscip Optim 52:251–268CrossRefGoogle Scholar
  189. Park C, Choi J-H, Haftka RT (2016a) Teaching a verification and validation course using simulations and experiments with paper helicopters. J Verif Valid Uncertain Quantif 1:031002Google Scholar
  190. Park C, Haftka RT, Kim NH (2016b) Remarks on multi-fidelity surrogates Struct Multidiscip Optim 55:1029–1050Google Scholar
  191. Park C, Haftka RT, Kim NH (2017) Simple alternative to Bayesian multi-fidelity surrogate framework. In: 58th AIAA/ASCE/AHS/ASC Structures, Structural Dynamics, and Materials Conference, p 0135Google Scholar
  192. Parry GW (1996) The characterization of uncertainty in probabilistic risk assessments of complex systems. Reliab Eng Syst Saf 54:119–126CrossRefGoogle Scholar
  193. Pettit CL (2004) Uncertainty quantification in aeroelasticity: recent results and research challenges. J Aircr 41:1217–1229.  https://doi.org/10.2514/1.3961 CrossRefGoogle Scholar
  194. Pianosi F, Beven K, Freer J, Hall JW, Rougier J, Stephenson DB, Wagener T (2016) Sensitivity analysis of environmental models: a systematic review with practical workflow. Environ Model Softw 79:214–232.  https://doi.org/10.1016/j.envsoft.2016.02.008 CrossRefGoogle Scholar
  195. Plackett RL (1983) Karl Pearson and the chi-squared test. Int Stat Rev 51:59–72Google Scholar
  196. Plackett RL, Burman JP (1946) The design of optimum multifactorial experiments. Biometrika 33:305–325CrossRefMathSciNetzbMATHGoogle Scholar
  197. Plischke E, Borgonovo E, Smith CL (2013) Global sensitivity measures from given data. Eur J Oper Res 226:536–550CrossRefMathSciNetzbMATHGoogle Scholar
  198. Pradlwarter HJ, Schuëller GI (2008) The use of kernel densities and confidence intervals to cope with insufficient data in validation experiments. Comput Methods Appl Mech Eng 197:2550–2560.  https://doi.org/10.1016/j.cma.2007.09.028 CrossRefzbMATHGoogle Scholar
  199. Qian PZ, Wu CJ (2008) Bayesian hierarchical modeling for integrating low-accuracy and high-accuracy experiments. Technometrics 50:192–204CrossRefMathSciNetGoogle Scholar
  200. Rahman S, Xu H (2004) A univariate dimension-reduction method for multi-dimensional integration in stochastic mechanics. Probab Eng Mech 19:393–408CrossRefGoogle Scholar
  201. Rebba R, Mahadevan S (2006) Validation of models with multivariate output. Reliab Eng Syst Saf 91:861–871CrossRefGoogle Scholar
  202. Rebba R, Mahadevan S (2008) Computational methods for model reliability assessment. Reliab Eng Syst Saf 93:1197–1207CrossRefGoogle Scholar
  203. Rebba R, Huang S, Liu Y, Mahadevan S (2005) Statistical validation of simulation models. Int J Mater Prod Technol 25:164–181CrossRefGoogle Scholar
  204. Rebba R, Mahadevan S, Huang S (2006) Validation and error estimation of computational models. Reliab Eng Syst Saf 91:1390–1397CrossRefGoogle Scholar
  205. Romero V (2019) Real-space model validation and predictor-corrector extrapolation applied to the Sandia cantilever beam end-to-end UQ problem, Paper AIAA-2019-1488, 21st AIAA Non-Deterministic Approaches Conference, AIAA SciTech, Jan. 7–11, San Diego, CAGoogle Scholar
  206. Roussouly N, Petitjean F, Salaun M (2013) A new adaptive response surface method for reliability analysis. Probab Eng Mech 32:103–115CrossRefGoogle Scholar
  207. Roy CJ, Oberkampf WL (2010) A complete framework for verification, validation, and uncertainty quantification in scientific computing. In: 48th AIAA Aerospace Sciences Meeting Including the New Horizons Forum and Aerospace Exposition, 4–7Google Scholar
  208. Roy CJ, Oberkampf WL (2011) A comprehensive framework for verification, validation, and uncertainty quantification in scientific computing. Comput Methods Appl Mech Eng 200:2131–2144CrossRefMathSciNetzbMATHGoogle Scholar
  209. Rui Q, Ouyang H, Wang H (2013) An efficient statistically equivalent reduced method on stochastic model updating. Appl Math Model 37:6079–6096CrossRefMathSciNetGoogle Scholar
  210. Rüschendorf L (2009) On the distributional transform, Sklar’s theorem, and the empirical copula process. J Stat Plann Infer 139:3921–3927CrossRefMathSciNetzbMATHGoogle Scholar
  211. Russell DM (1997) Error measures for comparing transient data: part I: development of a comprehensive error measure. In: Proceedings of the 68th Shock and Vibration Symposium, Hunt Valley. pp 175–184Google Scholar
  212. Salehghaffari S, Rais-Rohani M (2013) Material model uncertainty quantification using evidence theory. Proc Inst Mech Eng C J Mech Eng Sci 227:2165–2181CrossRefGoogle Scholar
  213. Sankararaman S, Mahadevan S (2011a) Likelihood-based representation of epistemic uncertainty due to sparse point data and/or interval data. Reliab Eng Syst Saf 96:814–824CrossRefGoogle Scholar
  214. Sankararaman S, Mahadevan S (2011b) Model validation under epistemic uncertainty. Reliab Eng Syst Saf 96:1232–1241CrossRefGoogle Scholar
  215. Sankararaman S, Mahadevan S (2013) Separating the contributions of variability and parameter uncertainty in probability distributions. Reliab Eng Syst Saf 112:187–199.  https://doi.org/10.1016/j.ress.2012.11.024 CrossRefGoogle Scholar
  216. Sankararaman S, Mahadevan S (2015) Integration of model verification, validation, and calibration for uncertainty quantification in engineering systems. Reliab Eng Syst Saf 138:194–209Google Scholar
  217. Sankararaman S, Ling Y, Mahadevan S (2011) Uncertainty quantification and model validation of fatigue crack growth prediction. Eng Fract Mech 78:1487–1504CrossRefGoogle Scholar
  218. Sankararaman S, McLemore K, Mahadevan S, Bradford SC, Peterson LD (2013) Test resource allocation in hierarchical systems using Bayesian networks. AIAA J 51:537–550CrossRefGoogle Scholar
  219. Sargent RG (2013) Verification and validation of simulation models. J Simul 7:12–24CrossRefGoogle Scholar
  220. Sarin H, Kokkolaras M, Hulbert G, Papalambros P, Barbat S, Yang R-J (2010) Comparing time histories for validation of simulation models: error measures and metrics. J Dyn Syst Meas Control 132:061401CrossRefGoogle Scholar
  221. Savu C, Trede M (2010) Hierarchies of Archimedean copulas. Quant Finan 10:295–304CrossRefMathSciNetzbMATHGoogle Scholar
  222. Scholz F (1985) Maximum likelihood estimation. Encycl Stat Sci 5:340–351Google Scholar
  223. Schwer LE (2007) Validation metrics for response histories: perspectives and case studies. Eng Comput 23:295–309CrossRefGoogle Scholar
  224. Shah H, Hosder S, Winter T (2015) Quantification of margins and mixed uncertainties using evidence theory and stochastic expansions. Reliab Eng Syst Saf 138:59–72.  https://doi.org/10.1016/j.ress.2015.01.012 CrossRefGoogle Scholar
  225. Shan S, Wang GG (2010) Survey of modeling and optimization strategies to solve high-dimensional design problems with computationally-expensive black-box functions. Struct Multidiscip Optim 41:219–241CrossRefMathSciNetzbMATHGoogle Scholar
  226. Shi L, Yang R, Zhu P (2012) A method for selecting surrogate models in crashworthiness optimization. Struct Multidiscip Optim 46:159–170CrossRefGoogle Scholar
  227. Shields MD, Zhang J (2016) The generalization of Latin hypercube sampling. Reliab Eng Syst Saf 148:96–108.  https://doi.org/10.1016/j.ress.2015.12.002 CrossRefGoogle Scholar
  228. Shields MD, Teferra K, Hapij A, Daddazio RP (2015) Refined stratified sampling for efficient Monte Carlo based uncertainty quantification. Reliab Eng Syst Saf 142:310–325.  https://doi.org/10.1016/j.ress.2015.05.023 CrossRefGoogle Scholar
  229. Silva AS, Ghisi E (2014) Uncertainty analysis of the computer model in building performance simulation. Energy and Buildings 76:258–269.  https://doi.org/10.1016/j.enbuild.2014.02.070 CrossRefGoogle Scholar
  230. Simpson TW, Korte JJ, Mauery TM, Mistree F (1998) Comparison of response surface and kriging models for multidisciplinary design optimization. In: Proceedings of the 7th AIAA/USAF/ NASA/ISSMO Symp. vol 1, pp 381–391, St. Louis, MO, USA, September 2-4Google Scholar
  231. Simpson TW, Mauery TM, Korte JJ, Mistree F (2001a) Kriging models for global approximation in simulation-based multidisciplinary design optimization. AIAA J 39:2233–2241CrossRefGoogle Scholar
  232. Simpson TW, Poplinski J, Koch PN, Allen JK (2001b) Metamodels for computer-based engineering design: survey and recommendations. Eng Comput 17:129–150CrossRefzbMATHGoogle Scholar
  233. Smirnov N (1948) Table for estimating the goodness of fit of empirical distributions. Ann Math Stat 19:279–281CrossRefMathSciNetzbMATHGoogle Scholar
  234. Smola AJ, Schölkopf B (2004) A tutorial on support vector regression. Stat Comput 14:199–222CrossRefMathSciNetGoogle Scholar
  235. Sobol IM (2001) Global sensitivity indices for nonlinear mathematical models and their Monte Carlo estimates. Math Comput Simul 55:271–280CrossRefMathSciNetzbMATHGoogle Scholar
  236. Sobol' IM (1990) On sensitivity estimation for nonlinear mathematical models. Math Model Comput Simul 2:112–118zbMATHGoogle Scholar
  237. Solomon H, Stephens MA (1978) Approximations to density functions using Pearson curves. J Am Stat Assoc 73:153–160CrossRefGoogle Scholar
  238. Sprague M, Geers T (2004) A spectral-element method for modelling cavitation in transient fluid-structure interaction. Int J Numer Methods Eng 60:2467–2499CrossRefzbMATHGoogle Scholar
  239. Stein M (1987) Large sample properties of simulations using Latin hypercube sampling. Technometrics 29:143–151CrossRefMathSciNetzbMATHGoogle Scholar
  240. Steinberg DM, Hunter WG (1984) Experimental design: review and comment. Technometrics 26:71–97CrossRefMathSciNetzbMATHGoogle Scholar
  241. Stephens MA (1974) EDF statistics for goodness of fit and some comparisons. J Am Stat Assoc 69:730–737CrossRefGoogle Scholar
  242. Sudret B (2008) Global sensitivity analysis using polynomial chaos expansions. Reliab Eng Syst Saf 93:964–979CrossRefGoogle Scholar
  243. Swiler LP, Paez TL, Mayes RL (2009) Epistemic uncertainty quantification tutorial. In: Proceedings of the 27th International Modal Analysis Conference, Orlando, FL, USA, February 9-12Google Scholar
  244. Tabatabaei M, Hakanen J, Hartikainen M, Miettinen K, Sindhya K (2015) A survey on handling computationally expensive multiobjective optimization problems using surrogates: non-nature inspired methods. Struct Multidiscip Optim 52:1–25CrossRefMathSciNetGoogle Scholar
  245. Teferra K, Shields MD, Hapij A, Daddazio RP (2014) Mapping model validation metrics to subject matter expert scores for model adequacy assessment. Reliab Eng Syst Saf 132:9–19.  https://doi.org/10.1016/j.ress.2014.07.010 CrossRefGoogle Scholar
  246. Thacker BH, Paez TL (2014) A simple probabilistic validation metric for the comparison of uncertain model and test results. In: Proceedings of the 16th AIAA Non-Deterministic Approaches Conference, National Harbor, MD, USA, January 13-17Google Scholar
  247. Thorne M, Williams MMR (1992) A review of expert judgment techniques with reference to nuclear safety. Prog Nucl Energy 27:83–254Google Scholar
  248. Trucano TG, Swiler LP, Igusa T, Oberkampf WL, Pilch M (2006) Calibration, validation, and sensitivity analysis: what’s what. Reliab Eng Syst Saf 91:1331–1357.  https://doi.org/10.1016/j.ress.2005.11.031 CrossRefGoogle Scholar
  249. Twisk D, Spit H, Beebe M, Depinet P (2007) Effect of dummy repeatability on numerical model accuracy. SAE 2007-01-1173Google Scholar
  250. Tyssedal J (2008) Plackett–Burman designs. Encycl. Stat in Qual and Reliab, New York, John Wiley & Sons, Inc, 1:1361–1125. https://onlinelibrary.wiley.com/doi/book/10.1002/9780470061572
  251. Urbina A, Mahadevan S, Paez TL (2011) Quantification of margins and uncertainties of complex systems in the presence of aleatoric and epistemic uncertainty. Reliab Eng Syst Saf 96:1114–1125CrossRefGoogle Scholar
  252. Uusitalo L, Lehikoinen A, Helle I, Myrberg K (2015) An overview of methods to evaluate uncertainty of deterministic models in decision support. Environ Model Softw 63:24–31.  https://doi.org/10.1016/j.envsoft.2014.09.017 CrossRefGoogle Scholar
  253. Viana FA, Haftka RT, Steffen V Jr (2009) Multiple surrogates: how cross-validation errors can help us to obtain the best predictor. Struct Multidiscip Optim 39:439–457CrossRefGoogle Scholar
  254. Viana FAC, Pecheny V, Haftka RT (2010) Using cross validation to design conservative surrogates. AIAA J 48:2286–2298CrossRefGoogle Scholar
  255. Viana FA, Simpson TW, Balabanov V, Toropov V (2014) Metamodeling in multidisciplinary design optimization: how far have we really come? AIAA J 52:670–690CrossRefGoogle Scholar
  256. Voyles IT, Roy CJ (2015) Evaluation of model validation techniques in the presence of aleatory and epistemic input uncertainties. In: 17th AIAA Non-Deterministic Approaches Conference. ARC, pp 1–16Google Scholar
  257. Warner JE, Aquino W, Grigoriu MD (2015) Stochastic reduced order models for inverse problems under uncertainty. Comput Methods Appl Mech Eng 285:488–514.  https://doi.org/10.1016/j.cma.2014.11.021 MathSciNetCrossRefzbMATHGoogle Scholar
  258. Weathers JB, Luck R, Weathers JW (2009) An exercise in model validation: comparing univariate statistics and Monte Carlo-based multivariate statistics. Reliab Eng Syst Saf 94:1695–1702.  https://doi.org/10.1016/j.ress.2009.04.007 CrossRefGoogle Scholar
  259. Wei D, Cui Z, Chen J (2008) Uncertainty quantification using polynomial chaos expansion with points of monomial cubature rules. Comput Struct 86:2102–2108CrossRefGoogle Scholar
  260. Wei P, Lu Z, Song J (2015) Variable importance analysis: a comprehensive review. Reliab Eng Syst Saf 142:399–432.  https://doi.org/10.1016/j.ress.2015.05.018 CrossRefGoogle Scholar
  261. Whang B, Gilbert WE, Zilliacus S (1994) Two visually meaningful correlation measures for comparing calculated and measured response histories. Shock Vib 1:303–316CrossRefGoogle Scholar
  262. Willmott CJ, Robeson SM, Matsuura K (2012) A refined index of model performance. Int J Climatol 32:2088–2094CrossRefGoogle Scholar
  263. Winkler RL (1996) Uncertainty in probabilistic risk assessment. Reliab Eng Syst Saf 54:127–132CrossRefGoogle Scholar
  264. Wojtkiewicz S, Eldred M, Field R, Urbina A, Red-Horse J (2001) In: Proceedings of the 42rd AIAA/ASME/ASCE/AHS/ASC Structures, Structural Dynamics, and Materials Conference, pp 1455, Seattle, WA, USA, April 16-19Google Scholar
  265. Wu YT, Mohanty S (2006) Variable screening and ranking using sampling-based sensitivity measures. Reliab Eng Syst Saf 91:634–647.  https://doi.org/10.1016/j.ress.2005.05.004 CrossRefGoogle Scholar
  266. Wu J, Apostolakis G, Okrent D (1990) Uncertainties in system analysis: probabilistic versus nonprobabilistic theories. Reliab Eng Syst Saf 30:163–181CrossRefGoogle Scholar
  267. Xi Z, Fu Y, Yang RJ (2013) Model bias characterization in the design space under uncertainty. Int J Perform Eng 9:433Google Scholar
  268. Xi Z, Pan H, Fu Y, Yang R-J (2015) Validation metric for dynamic system responses under uncertainty. SAE Int J Mater Manuf 8:309–314CrossRefGoogle Scholar
  269. Xie K, Wells L, Camelio JA, Youn BD (2007) Variation propagation analysis on compliant assemblies considering contact interaction. J Manuf Sci Eng 129:934–942CrossRefGoogle Scholar
  270. Xiong Y, Chen W, Tsui K-L, Apley DW (2009) A better understanding of model updating strategies in validating engineering models. Comput Methods Appl Mech Eng 198:1327–1337.  https://doi.org/10.1016/j.cma.2008.11.023 CrossRefzbMATHGoogle Scholar
  271. Xu H, Rahman S (2004) A generalized dimension-reduction method for multidimensional integration in stochastic mechanics. Int J Numer Methods Eng 61:1992–2019CrossRefzbMATHGoogle Scholar
  272. Yates F (1934) Contingency tables involving small numbers and the χ2 test. Suppl J R Stat Soc 1:217–235CrossRefzbMATHGoogle Scholar
  273. Youn BD, Choi KK (2004) An investigation of nonlinearity of reliability-based design optimization approaches. J Mech Des 126:403.  https://doi.org/10.1115/1.1701880 CrossRefGoogle Scholar
  274. Youn BD, Wang P (2008) Bayesian reliability-based design optimization using eigenvector dimension reduction (EDR) method. Struct Multidiscip Optim 36:107–123CrossRefGoogle Scholar
  275. Youn BD, Wang P (2009) Complementary intersection method for system reliability analysis. J Mech Des 131:041004CrossRefGoogle Scholar
  276. Youn BD, Xi Z (2009) Reliability-based robust design optimization using the eigenvector dimension reduction (EDR) method. Struct Multidiscip Optim 37:475–492CrossRefGoogle Scholar
  277. Youn BD, Xi Z, Wang P (2008) Eigenvector dimension reduction (EDR) method for sensitivity-free probability analysis. Struct Multidiscip Optim 37:13–28CrossRefMathSciNetzbMATHGoogle Scholar
  278. Youn BD, Jung BC, Xi Z, Kim SB, Lee W (2011) A hierarchical framework for statistical model calibration in engineering product development. Comput Methods Appl Mech Eng 200:1421–1431CrossRefzbMATHGoogle Scholar
  279. Yuan J, Ng SH, Tsui KL (2013) Calibration of stochastic computer models using stochastic approximation methods. Autom Sci Eng IEEE Trans 10:171–186CrossRefGoogle Scholar
  280. Zambom AZ, Dias R (2012) A review of kernel density estimation with applications to econometrics arXiv preprint arXiv:12122812Google Scholar
  281. Zárate BA, Caicedo JM (2008) Finite element model updating: multiple alternatives. Eng Struct 30:3724–3730.  https://doi.org/10.1016/j.engstruct.2008.06.012 CrossRefGoogle Scholar
  282. Zhan Z, Fu Y, Yang R-J (2011a) Enhanced error assessment of response time histories (EEARTH) metric and calibration process. SAE 2011-01-0245Google Scholar
  283. Zhan Z, Fu Y, Yang R-J, Peng Y (2011b) An automatic model calibration method for occupant restraint systems. Struct Multidiscip Optim 44:815–822CrossRefGoogle Scholar
  284. Zhan Z, Fu Y, Yang R-J, Peng Y (2011c) An enhanced Bayesian based model validation method for dynamic systems. J Mech Des 133:041005CrossRefGoogle Scholar
  285. Zhan Z-f, Hu J, Fu Y, Yang R-J, Peng Y-h, Qi J (2012a) Multivariate error assessment of response time histories method for dynamic systems. J Zhejiang Univ Sci A 13:121–131CrossRefGoogle Scholar
  286. Zhan Z, Fu Y, Yang R-J, Peng Y (2012b) Bayesian based multivariate model validation method under uncertainty for dynamic systems. J Mech Des 134:034502CrossRefGoogle Scholar
  287. Zhang J, Du X (2010) A second-order reliability method with first-order efficiency. J Mech Des 132:101006CrossRefGoogle Scholar
  288. Zhang X, Pandey MD (2014) An effective approximation for variance-based global sensitivity analysis. Reliab Eng Syst Saf 121:164–174CrossRefGoogle Scholar
  289. Zhang S, Zhu P, Chen W, Arendt P (2013) Concurrent treatment of parametric uncertainty and metamodeling uncertainty in robust design. Struct Multidiscip Optim 47:63–76CrossRefMathSciNetzbMATHGoogle Scholar
  290. Zhao Y-G, Ono T (2000) New point estimates for probability moments. J Eng Mech 126:433–436CrossRefGoogle Scholar
  291. Zhu S-P, Huang H-Z, Peng W, Wang H-K, Mahadevan S (2016) Probabilistic physics of failure-based framework for fatigue life prediction of aircraft gas turbine discs under uncertainty. Reliab Eng Syst Saf 146:1–12.  https://doi.org/10.1016/j.ress.2015.10.002 CrossRefGoogle Scholar

Copyright information

© Springer-Verlag GmbH Germany, part of Springer Nature 2019

Authors and Affiliations

  1. 1.Department of Mechanical and Aerospace EngineeringSeoul National UniversitySeoulSouth Korea
  2. 2.School of Mechanical EngineeringGwangju Institute of Science and TechnologyGwangjuSouth Korea
  3. 3.Department of Mechanical and Aerospace Engineering & Institute of Advanced Machines and DesignSeoul National UniversitySeoulSouth Korea
  4. 4.OnePredict Inc.SeoulSouth Korea
  5. 5.Department of Mechanical and Aerospace EngineeringUniversity of FloridaGainesvilleUSA

Personalised recommendations