Multilevel Uncertainty Integration

  • Sankaran MahadevanEmail author
  • Shankar Sankararaman
  • Chenzhao Li
Reference work entry


This chapter discusses a Bayesian methodology to integrate model verification, validation, and calibration activities for the purpose of overall uncertainty quantification in model-based prediction. The methodology is first developed for single-level models and then extended to systems that are studied using multilevel models that interact with each other. Two types of interactions among multilevel models are considered: (1) Type-I, where the output of a lower-level model (component and/or subsystem) becomes an input to a higher-level system model, and (2) Type-II, where parameters of the system model are inferred using lower-level models and tests (that describe simplified components and/or isolated physics). The various models; their inputs, parameters, and outputs; experimental data; and various sources of model error are connected through a Bayesian network. The results of calibration, verification, and validation with respect to each individual model are integrated using the principles of conditional probability and total probability and propagated through the Bayesian network in order to quantify the overall system-level prediction uncertainty. For Type-II model, the relevance of each lower-level output to the system-level quantity of interest is quantified by comparing Sobol indices, thus measuring the extent to which a lower-level test represents the characteristics of the system so that the calibration results can be reliably used in the system level. The proposed methodology is illustrated with numerical examples that deal with heat conduction and structural dynamics.


Multilevel system Uncertainty quantification Bayesian network Calibration Validation Verification Relevance Sobol indices Bayes factor Model reliability metric 


  1. 1.
    Urbina, A., Mahadevan, S., Paez, T.L.: Quantification of margins and uncertainties of complex systems in the presence of aleatoric and epistemic uncertainty. Reliab. Eng. Syst. Saf. 96(9), 1114–1125 (2011)CrossRefGoogle Scholar
  2. 2.
    Sankararaman, S., Ling, Y., Mahadevan, S.: Uncertainty quantification and model validation of fatigue crack growth prediction. Eng. Fract. Mech. 78(7), 1487–1504 (2011)CrossRefGoogle Scholar
  3. 3.
    Sankararaman, S., Mahadevan, S.: Model parameter estimation with imprecise and unpaired data. Inverse Probl. Sci. Eng. 20(7), 1017–1041 (2012)MathSciNetCrossRefGoogle Scholar
  4. 4.
    Sankararaman, S., Mahadevan, S.: Model validation under epistemic uncertainty. Reliab. Eng. Syst. Saf. 96(9), 1232–1241 (2011)CrossRefGoogle Scholar
  5. 5.
    Ling, Y., Mahadevan, S.: Quantitative model validation techniques: new insights. Reliab. Eng. Syst. Saf. 111, 217–231 (2013)CrossRefGoogle Scholar
  6. 6.
    Sankararaman, S., Mahadevan, S.: Likelihood-based representation of epistemic uncertainty due to sparse point data and/or interval data. Reliab. Eng. Syst. Saf. 96(7), 814–824 (2011)CrossRefGoogle Scholar
  7. 7.
    Jeffrey, H.: Theory of Probability. Oxford University Press, Oxford (1998)Google Scholar
  8. 8.
    Sankararaman, S., Ling, Y., Shantz, C., Mahadevan, S.: Uncertainty quantification in fatigue crack growth prognosis. Int. J. Progn. Heal. Manag. 2(1), 15 (2011)Google Scholar
  9. 9.
    Sankararaman, S., Ling, Y., Shantz, C., Mahadevan, S.: Inference of equivalent initial flaw size under multiple sources of uncertainty. Int. J. Fatigue 33(2), 75–89 (2011)CrossRefGoogle Scholar
  10. 10.
    Sankararaman, S., Ling, Y., Mahadevan, S.: Statistical inference of equivalent initial flaw size with complicated structural geometry and multi-axial variable amplitude loading. Int. J. Fatigue 32(10), 1689–1700 (2010)CrossRefGoogle Scholar
  11. 11.
    Sankararaman, S., McLemore, K., Mahadevan, S., Bradford, S.C., Peterson, L.D.: Test resource allocation in hierarchical systems using bayesian networks. AIAA J. 51(3), 537–550 (2013)CrossRefGoogle Scholar
  12. 12.
    Mullins, J., Li, C., Mahadevan, S., Urbina, A.: Optimal Selection of Calibration and Validation Test Samples under Uncertainty. In: IMAC XXXII, Orlando, pp. 391–401 (2014)Google Scholar
  13. 13.
    Li, C., Mahadevan, S.: Sensitivity Analysis for Test Resource Allocation. In: IMAC XXXIII, Orlando (2015)CrossRefGoogle Scholar
  14. 14.
    Mullins, J., Li, C., Sankararaman, S., Mahadevan, S.: Probabilistic integration of validation and calibration results for prediction level uncertainty quantification: application to structural dynamics. In: 54th AIAA/ASME/ASCE/AHS/ASC Structures, Structural Dynamics, and Materials Conference, Boston (2013)Google Scholar
  15. 15.
    Kennedy, M.C., O’Hagan, A.: Bayesian calibration of computer models. J. R. Stat. Soc. 63(3), 425–464 (2001)MathSciNetzbMATHCrossRefGoogle Scholar
  16. 16.
    Sankararaman, S., Mahadevan, S.: Comprehensive framework for integration of calibration, verification and validation. In: 53rd AIAA/ASME/ASCE/AHS/ASC Structures, Structural Dynamics and Materials Conference, Honolulu, pp. 1–12 (2012)Google Scholar
  17. 17.
    Li, C., Mahadevan, S.: Uncertainty quantification and output prediction in multi-level problems. In: 16th AIAA Non-Deterministic Approaches Conference, National Harbor (2014)Google Scholar
  18. 18.
    Li, C., Mahadevan, S.: Role of calibration, validation, and relevance in multi-level uncertainty integration. Reliab. Eng. Syst. Saf. 148, 32–43 (2016)CrossRefGoogle Scholar
  19. 19.
    Sankararaman, S., Mahadevan, S.: Likelihood-based approach to multidisciplinary analysis under uncertainty. J. Mech. Des. 134(3), 031008 (2012)CrossRefGoogle Scholar
  20. 20.
    Babuska, I., Oden, J.T.T.: Verification and validation in computational engineering and science: basic concepts. Comput. Methods Appl. Mech. Eng. 193(36–38), 4057–4066 (2004)MathSciNetzbMATHCrossRefGoogle Scholar
  21. 21.
    Roy, C.J.: Review of code and solution verification procedures for computational simulation. J. Comput. Phys. 205(1), 131–156 (2005)zbMATHCrossRefGoogle Scholar
  22. 22.
    AIAA: Guide for the verification and validation of computational fluid dynamics simulations. American Institute of Aeronautics and Astronautics (AIAA), no. AIAA G-077-1998 (1998)Google Scholar
  23. 23.
    Defense Modelling and Simulation Office, Verification, Validation, and accreditation (VV & A) recommend practices guide, Alexandia (1998)Google Scholar
  24. 24.
    Oberkampf, W.L., Blottner, F.G.: Issues in computational fluid dynamics code verification and validation. AIAA J 36(5), 687–695 (1998)CrossRefGoogle Scholar
  25. 25.
    Oberkampf, W.L., Trucano, T.G.G.: Verification and validation in computational fluid dynamics. Prog. Aerosp. Sci. 38(3), 209–272 (2002)CrossRefGoogle Scholar
  26. 26.
    Benay, R., Chanetz, B., Delery, J.: Code verification/validation with respect to experimental data banks. Aerosp. Sci. Technol. 7(4), 239–262 (2003)CrossRefGoogle Scholar
  27. 27.
    Roy, C.J., Oberkampf, W.L.: A comprehensive framework for verification, validation, and uncertainty quantification in scientific computing. Comput. Methods Appl. Mech. Eng. 200(25), 2131–2144 (2011)MathSciNetzbMATHCrossRefGoogle Scholar
  28. 28.
    Roache, P.J.: Verification of codes and calculations. Aiaa J. 36(5), 696–702 (1998)CrossRefGoogle Scholar
  29. 29.
    Roache, P.J.: Verification and Validation in Computational Science and Engineering. Hermosa Publishers, Albuquerque (1998)Google Scholar
  30. 30.
    Oberkampf, W.L., Trucano, T.G., Hirsch, C.: Verification, validation, and predictive capability in computational engineering and physics. Appl. Mech. Rev. 57(5), 345–384 (2004)CrossRefGoogle Scholar
  31. 31.
    Roy, C.J., McWherter-Payne, M.A., Oberkampf, W.L.: Verification and validation for laminar hypersonic flowfields, part 1: verification. Aiaa J. 41(10), 1934–1943 (2003)CrossRefGoogle Scholar
  32. 32.
    Rebba, R., Mahadevan, S., Huang, S.: Validation and error estimation of computational models. Reliab. Eng. Syst. Saf. 91(10–11), 1390–1397 (2006)CrossRefGoogle Scholar
  33. 33.
    Liang, B., Mahadevan, S.: Error and uncertainty quantification and sensitivity analysis in mechanics computational models. Int. J. Uncertain. Quantif. 1(2), 147–161 (2011)MathSciNetCrossRefGoogle Scholar
  34. 34.
    Rangavajhala, S., Sura, V.S., Hombal, V.K., Mahadevan, S.: Discretization error estimation in multidisciplinary simulations. AIAA J. 49(12), 2673–2683 (2011)CrossRefGoogle Scholar
  35. 35.
    Ferziger, J., Peric, M.: Computational Methods for Fluid Dynamics. Springer, Berlin (1996)zbMATHCrossRefGoogle Scholar
  36. 36.
    Ainsworth, M., Oden, J.T.T.: A posteriori error estimation in finite element analysis. Comput. Methods Appl. Mech. Eng. 142(1–2), 1–88 (1997)MathSciNetzbMATHCrossRefGoogle Scholar
  37. 37.
    Oberkampf, W.L., DeLand, S.M., Rutherford, B.M., Diegert, K.V., Alvin, K.F.: Error and uncertainty in modeling and simulation. Reliab. Eng. Syst. Saf. 75(3), 333–357 (2002)CrossRefGoogle Scholar
  38. 38.
    Haldar, A., Mahadevan, S.: Probability, Reliability, and Statistical Methods in Engineering Design. John Wiley, New York (2000)Google Scholar
  39. 39.
    Ghanem, R., Spanos, P.D.: Polynomial chaos in stochastic finite elements. J. Appl. Mech. 57(1)(89), 197–202 (1990)Google Scholar
  40. 40.
    Buhmann, M.D.: Radial Basis Functions: Theory and Implementations, vol. 12. Cambridge university press, Cambridge/New York (2003)zbMATHCrossRefGoogle Scholar
  41. 41.
    Rasmussen, C.E., Williams, C.K.I.: Gaussian Processes for Machine Learning. MIT, Cambridge (2006)zbMATHGoogle Scholar
  42. 42.
    Richards, S.A.: Completed Richardson extrapolation in space and time. Commun. Numer. Methods Eng. 13(7), 573–582 (1997)MathSciNetzbMATHCrossRefGoogle Scholar
  43. 43.
    Xu, P., Su, X., Mahadevan, S., Li, C., Deng, Y.: A non-parametric method to determine basic probability assignment for classification problems. Appl. Intell. 41(3), 681–693 (2014)CrossRefGoogle Scholar
  44. 44.
    Babuska, I., Rheinboldt, W.C.: A posteriori error estimates for the finite element method. Int. J. Numer. Methods Eng. 12(10), 1597–1615 (1978)zbMATHCrossRefGoogle Scholar
  45. 45.
    Demkowicz, L., Oden, J.T., Strouboulis, T.: Adaptive finite elements for flow problems with moving boundaries. part I: variational principles and a posteriori estimates. Comput. Methods Appl. Mech. Eng. 46(2), 217–251 (1984)Google Scholar
  46. 46.
    Rasmussen, C.E.: Evaluation of Gaussian processes and other methods for non-linear regression. PhD dissertation, University of Toronto, 1996Google Scholar
  47. 47.
    Rasmussen, C.E.: The infinite Gaussian mixture model. In: NIPS, Denver, vol. 12, pp. 554–560 (1999)Google Scholar
  48. 48.
    Rasmussen, C.E.: Gaussian processes in machine learning. In: Bousquet, O., VonLuxburg, U., Ratsch, G. (eds.) Advanced Lectures on Machine Learning, vol. 3176, pp. 63–71 (2004)Google Scholar
  49. 49.
    Santner, T.J., Williams, B.J., Notz, W.I.: The design and analysis of computer experiments. Springer, Dordrecht/New York (2013)zbMATHGoogle Scholar
  50. 50.
    Bichon, B.J., Eldred, M.S., Swiler, L.P., Mahadevan, S., McFarland, J.M.: Efficient global reliability analysis for nonlinear implicit performance functions. Aiaa J. 46(10), 2459–2468 (2008)CrossRefGoogle Scholar
  51. 51.
    McFarland, J.M.: Uncertainty Analysis for Computer Simulations throuth Validation and Calibraion. Vanderbilt University, Nashville (2008)Google Scholar
  52. 52.
    Cressie, N.: Spatial Statistics. John Wiley, New York (1991)zbMATHGoogle Scholar
  53. 53.
    Chiles, J.-P., Delfiner, P.: Geostatistics: Modeling Spatial Uncertainty, vol. 344. Wiley-Interscience, New York (1999)zbMATHCrossRefGoogle Scholar
  54. 54.
    Wackernagel, H.: Multivariate Geostatistics: An Introduction with Applications. Springer, Berlin/New York (2003)zbMATHCrossRefGoogle Scholar
  55. 55.
    Trucano, T.G., Swiler, L.P., Igusa, T., Oberkampf, W.L., Pilch, M.: Calibration, validation, and sensitivity analysis: what’s what. Reliab. Eng. Syst. Saf. 91(10–11), 1331–1357 (2006)CrossRefGoogle Scholar
  56. 56.
    Seber, G.A.F., Wild, C.J.: Nonlinear Regression. Wiley, New York (1989)zbMATHCrossRefGoogle Scholar
  57. 57.
    Edwards, A.W.F.: Likelihood. Cambridge University Press, Cambridge, UK (1972)zbMATHGoogle Scholar
  58. 58.
    Pawitan, Y.: In all likelihood: statistical modelling and inference using likelihood. Oxford University Press, Oxford/New York (2001)zbMATHGoogle Scholar
  59. 59.
    Leonard, T., Hsu, J.: Bayesian Methods. Cambridge University Press, Cambridge (2001)zbMATHGoogle Scholar
  60. 60.
    Lee, P.: Bayesian Statistics, 3rd edn. Arnold, London (2004)zbMATHGoogle Scholar
  61. 61.
    Malinverno, A., Briggs, V.A.: Expanded uncertainty quantification in inverse problems: hierarchical Bayes and empirical Bayes. Geophysics 69(4), 1005–1016 (2004)CrossRefGoogle Scholar
  62. 62.
    Park, I., Amarchinta, H.K., Grandhi, R.V.: A Bayesian approach for quantification of model uncertainty. Reliab. Eng. Syst. Saf. 95(7), 777–785 (2010)CrossRefGoogle Scholar
  63. 63.
    Oliver, T.A., Moser, R.D.: Accounting for uncertainty in the analysis of overlap layer mean velocity models. Phys. Fluids 24(7), 075108 (2012)CrossRefGoogle Scholar
  64. 64.
    Arendt, P.D., Apley, D.W., Chen, W.: Quantification of model uncertainty: calibration, model discrepancy, and identifiability. J. Mech. Des. 134(10), 100908 (2012)CrossRefGoogle Scholar
  65. 65.
    Ling, Y., Mullins, J.G., Mahadevan, S.: Options for the inclusion of model discrepancy in Bayesian calibration. In: 16th AIAA Non-Deterministic Approaches Conference, National Harbor. American Institute of Aeronautics and Astronautics (2014)Google Scholar
  66. 66.
    Liu, F., Bayarri, M.J., Berger, J.O.: Modularization in Bayesian analysis, with emphasis on analysis of computer models. Bayesian Anal. 4(1), 119–150 (2009)MathSciNetzbMATHCrossRefGoogle Scholar
  67. 67.
    Sankararaman, S.: Uncertainty Quantification and Integration in Engineering Systems. Vanderbilt University, Nashville (2012)Google Scholar
  68. 68.
    Gilks, W.R., Richardson, S., Spiegelhalter, D.J.: Markov chain Monte Carlo in practice. Chapman and Hall, London (1996)zbMATHGoogle Scholar
  69. 69.
    Metropolis, N., Rosenbluth, A.W., Rosenbluth, M.N., Teller, A.H., Teller, E.: Equation of state calculations by fast computing machines. J. Chem. Phys. 21(6), 1087–1092 (1953)CrossRefGoogle Scholar
  70. 70.
    Gilks, W.R., Wild, P.: Adaptive rejection sampling for Gibbs sampling. Appl. Stat. 41(2), 337–348 (1992)zbMATHCrossRefGoogle Scholar
  71. 71.
    Neal, R.M.: Slice sampling. Ann. Stat. 31(3), 705–767 (2003)MathSciNetzbMATHCrossRefGoogle Scholar
  72. 72.
    American Society of Mechanical Engineers: Guide for Verification and Validation in Computational Solid Mechanics, p. 53. American Society of Mechanical Engineers, New York (2006)Google Scholar
  73. 73.
    Coleman, H.W., Stern, F.: Uncertainties and CFD code validation. J. Fluids Eng. Asme 119(4), 795–803 (1997)CrossRefGoogle Scholar
  74. 74.
    Oberkampf, W.L., Barone, M.F.: Measures of agreement between computation and experiment: validation metrics. J. Comput. Phys. 217(1), 5–36 (2006)zbMATHCrossRefGoogle Scholar
  75. 75.
    Ferson, S., Oberkampf, W.L., Ginzburg, L.: Model validation and predictive capability for the thermal challenge problem. Comput. Methods Appl. Mech. Eng. 197(29–32), 2408–2430 (2008)zbMATHCrossRefGoogle Scholar
  76. 76.
    Hills, R.G., Leslie, I.H.: Statistical validation of engineering and scientific models: validation experiments to application. Sandia National Labs., Albuquerque/Livermore (2003).CrossRefGoogle Scholar
  77. 77.
    Urbina, A., Paez, T.L., Hasselman, T.K., Wathugala, G.W., Yap, K.: Assessment of model accuracy relative to stochastic system behavior. In: Proceedings of 44th AIAA Structures, Structural Dynamics, Materials Conference, Norfolk, pp. 7–10 (2003)Google Scholar
  78. 78.
    Gelfand, A.E., Dey, D.K.: Bayesian model choice: asymptotics and exact calculations. J. R. Stat. Soc. Ser. B-Methodol. 56(3), 501–514 (1994)MathSciNetzbMATHGoogle Scholar
  79. 79.
    Geweke, J.: Bayesian model comparison and validation. Am. Econ. Rev. 97(2), 60–64 (2007)MathSciNetCrossRefGoogle Scholar
  80. 80.
    Zhang, R.X., Mahadevan, S.: Bayesian methodology for reliability model acceptance. Reliab. Eng. Syst. Saf. 80(1), 95–103 (2003)CrossRefGoogle Scholar
  81. 81.
    Mahadevan, S., Rebba, R.: Validation of reliability computational models using Bayes networks. Reliab. Eng. Syst. Saf. 87(2), 223–232 (2005)CrossRefGoogle Scholar
  82. 82.
    Rebba, R., Mahadevan, S.: Computational methods for model reliability assessment. Reliab. Eng. Syst. Saf. 93(8), 1197–1207 (2008)CrossRefGoogle Scholar
  83. 83.
    Sankararaman, S., Mahadevan, S.: Assessing the reliability of computational models under uncertainty. In: 54th AIAA/ASME/ASCE/AHS/ASC Structures, Structural Dynamics, and Materials Conference, Boston, pp. 1–8 (2013)Google Scholar
  84. 84.
    Thacker, B.H., Paez, T.L.: A simple probabilistic validation metric for the comparison of uncertain model and test results. In: ASME Verification and Validation Symposium, Las Vegas (2013)Google Scholar
  85. 85.
    Liu, Y., Chen, W., Arendt, P., Huang, H.-Z.: Toward a better understanding of model validation metrics. J. Mech. Des. 133(7), 071005 (2011)CrossRefGoogle Scholar
  86. 86.
    Roache, P.J.: Fundamentals of Verification and Validation. Hermosa Press, Socorro (2009)Google Scholar
  87. 87.
    Oberkampf, W.L., Roy, C.C.J.: Verification and Validation in Scientific Computing. Cambridge University Press, New York (2010)zbMATHCrossRefGoogle Scholar
  88. 88.
    O’Hagan, A.: Fractional Bayes Factors for Model Comparison. J. R. Stat. Soc. 57(1), 99–138 (1995)MathSciNetzbMATHGoogle Scholar
  89. 89.
    Jiang, X., Mahadevan, S.: Bayesian risk-based decision method for model validation under uncertainty. Reliab. Eng. Syst. Saf. 92(6), 707–718 (2007)CrossRefGoogle Scholar
  90. 90.
    Cha, S.: Comprehensive survey on distance/similarity measures between probability density functions. Int. J. Math. Model. METHODS Appl. Sci. 1(4) (2007)Google Scholar
  91. 91.
    De Maesschalck, R., Jouan-Rimbaud, D., Massart, D.L.: The Mahalanobis distance. Chemom. Intell. Lab. Syst. 50(1), 1–18 (2000)CrossRefGoogle Scholar
  92. 92.
    Saltelli, A., Ratto, M., Andres, T., Campolongo, F., Cariboni, J., Gatelli, D., Saisana, M., Tarantola, S.: Global Sensitivity Analysis: The Primer. John Wiley, Chichester (2008)zbMATHGoogle Scholar
  93. 93.
    Sobol’, I.M.: Global sensitivity indices for nonlinear mathematical models and their Monte Carlo estimates. Math. Comput. Simul. 55(1–3), 271–280 (2001)Google Scholar
  94. 94.
    Li, C., Mahadevan, S.: Global sensitivity analysis for system response prediction using auxiliary variable method. In: 17th AIAA Non-Deterministic Approaches Conference, Kissimmee (2015)Google Scholar
  95. 95.
    Li, C., Mahadevan, S.: Relative contributions of aleatory and epistemic uncertainty sources in time series prediction. Int. J. Fatigue 82, 474–486 (2016)CrossRefGoogle Scholar
  96. 96.
    Singhal, A.: Modern information retrieval: a brief overview. IEEE Data Eng. Bull. 24(4), 35–43 (2001)Google Scholar
  97. 97.
    Van Horn, K.S.: Constructing a logic of plausible inference: a guide to Cox’s theorem. Int. J. Approx. Reason. 34(1), 3–24 (2003)MathSciNetzbMATHCrossRefGoogle Scholar
  98. 98.
    Sankararaman, S., Mahadevan, S.: Integration of model verification, validation, and calibration for uncertainty quantification in engineering systems. Reliab. Eng. Syst. Saf. 138, 194–209 (2015)zbMATHCrossRefGoogle Scholar
  99. 99.
    Li, C., Mahadevan, S.: Uncertainty quantification and integration in multi-level problems. In: IMAC XXXII, Orlando, vol. 3 (2014)Google Scholar
  100. 100.
    Rosenblatt, M.: Remarks on some nonparametric estimates of a density function. Ann. Math. Stat. 27(3), 832–837 (1956)MathSciNetzbMATHCrossRefGoogle Scholar
  101. 101.
    Red-Horse, J.R.R., Paez, T.L.L.: Sandia National Laboratories validation workshop: Structural dynamics application. Comput. Methods Appl. Mech. Eng. 197(29–32), 2578–2584 (2008)zbMATHCrossRefGoogle Scholar
  102. 102.
    Chopra, A.K.: Dynamics of Structures: Theory and Applications to Earthquake Engineering, 4th edn. Prentice Hall, Englewood Cliffs (2011)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2017

Authors and Affiliations

  • Sankaran Mahadevan
    • 1
    Email author
  • Shankar Sankararaman
    • 2
  • Chenzhao Li
    • 1
  1. 1.Department of Civil and Environmental EngineeringVanderbilt UniversityNashvilleUSA
  2. 2.NASA Ames Research CenterSGT Inc.Moffett FieldUSA

Personalised recommendations