Abstract
As the field of computational modeling continues to mature and simulation results are used to inform more critical decisions, validation of the physical models that form the basis of these simulations takes on increasing importance. While model validation is not a new concept, traditional techniques such as visual comparison of model outputs and experimental observations without accounting for uncertainties are insufficient for assessing model validity, particularly for the case where the intended purpose of the model is to make extrapolative predictions. This work provides an overview of validation of physical models in the presence of uncertainty. In particular, two issues are discussed: comparison of model outputs and observational data when both the model and observations are uncertain, and the process of building confidence in extrapolative predictions. For comparing uncertain model outputs and data, a Bayesian probabilistic perspective is adopted in which the problem of assessing the consistency of the model and the observations becomes one of Bayesian model checking. A broadly applicable approach to Bayesian model checking for physical models is described. For validating extrapolative predictions, a recently developed process termed predictive validation is discussed. This process relies on the ideas of Bayesian model checking but goes beyond comparison of model and data to assess the conditions necessary for reliable extrapolation using physics-based models.
References
Adams, B.M., Ebeida, M.S., Eldred, M.S., Others: Dakota, A Multilevel Parallel Object-Oriented Framework for Design Optimization, Parameter Estimation, Uncertainty Quantification, and Sensitivity Analysis: Version 6.2 User’s Manual. Sandia National Laboratories, Albuquerque (2014). https://dakota.sandia.gov/documentation.html
AIAA Computational Fluid Dynamics Committee on Standards: AIAA Guide for Verification and Validation of Computational Fluid Dynamics Simulations. AIAA Paper number G-077-1999 (1998)
Arlot, S., Celisse, A.: A survey of cross-validation procedures for model selection. Stat. Surv. 4, 40–79 (2010). doi:10.1214/09-SS054, http://dx.doi.org/10.1214/09-SS054
ASME Committee V&V 10: Standard for Verification and Validation in Computational Solid Mechanics. ASME (2006)
Babuška, I., Nobile, F., Tempone, R.: Reliability of computational science. Numer. Methods Partial Differ. Equ. 23(4), 753–784 (2007). doi:10.1002/num.20263
Box, G.E.P.: Sampling and Bayes’ inference in scientific modeling and robustness. R. Stat. Soc. Ser. A 143, 383–430 (1980)
Box, G., Tiao, G.C.: Bayesian Inference in Statistical Analysis. Wiley Classics, New York (1973)
Cox, R.T.: The Algebra of Probable Inference. Johns Hopkins University Press, Baltimore (1961)
Cui, T., Martin, J., Marzouk, Y.M., Solonen, A., Spantini, A.: Likelihood-informed dimension reduction for nonlinear inverse problems. Inverse Probl. 30(11), 114015 (2014)
Dubois, D., Prade, H.: Possibility Theory: An Approach to Computerized Processing of Uncertainty. Plenum Press, New York (1988)
Ferson, S., Ginzburg, L.R.: Different methods are needed to propagate ignorance and variability. Reliab. Eng. Syst. Saf. 54, 133–144 (1996)
Fine, T.L.: Theories of Probability. Academic, New York (1973)
Firm uses doe’s fastest supercomputer to streamline long-haul trucks. Office of Science, U.S. Department of Energy, Stories of Discovery and Innovation (2011). http://science.energy.gov/discovery-and-innovation/stories/2011/127008/
Gelman, A.: Comment: ‘Bayesian checking of the second levels of hierarchical models’. Stat. Sci. 22, 349–352 (2007). doi:doi:10.1214/07-STS235A
Gelman, A., Rubin, D.B.: Avoiding model selection in Bayesian social research. Sociol. Methodol. 25, 165–173 (1995)
Gelman, A., Shalizi, C.R.: Philosophy and the practice of Bayesian statistics. Br. J. Math. Stat. Psychol. 66(1), 8–38 (2013)
Gelman, A., Meng, X.L., Stern, H.: Posterior predictive assessment of medel fitness via realized discrepancies. Statistica Sinica 6, 733–807 (1996)
Gelman, A., Carlin, J.B., Stern, H.S., Dunson, D.B., Vehtari, A., Rubin, D.B.: Bayesian Data Analysis, 3rd edn. CRC Press, Boca Raton (2014)
Hsu, J.: Multiple Comparisons: Theory and Methods. Chapman and Hall/CRC, London (1996)
Hyndman, R.J.: Computing and graphing highest density regions. Am. Stat. 50(2), 120–126 (1996)
Jaynes, E.T.: Probability Theory: The Logic of Science. Cambridge University Press, Cambridge/New York (2003)
Kanji, G.K.: 100 Statistical Tests, 3rd edn. Sage Publications, London/Thousand Oaks (2006)
Le Maıtre, O., Knio, O., Najm, H., Ghanem, R.: Uncertainty propagation using wiener–haar expansions. J. Comput. Phys. 197(1), 28–57 (2004)
Li, J., Marzouk, Y.M.: Adaptive construction of surrogates for the Bayesian solution of inverse problems. SIAM J. Sci. Comput. 36(3), A1163–A1186 (2014)
Miller, R.G.J.: Simultaneous Statistical Inference, 2nd edn. Springer, New York (1981)
Miller, L.K.: Simulation-based engineering for industrial competitive advantage. Comput. Sci. Eng. 12(3), 14–21 (2010). doi:10.1109/MCSE.2010.71
Najm, H.N.: Uncertainty quantification and polynomial chaos techniques in computational fluid dynamics. Annu. Rev. Fluid Mech. 41, 35–52 (2009)
Oberkampf, W.L., Helton, J.C., Sentz, K.: Mathematical representation of uncertainty. AIAA 2001-1645
Oden, J.T., Belytschko, T., Fish, J., Hughes, T.J.R., Johnson, C., Keyes, D., Laub, A., Petzold, L., Srolovitz, D., Yip, S.: Revolutionizing engineering science through simulation: a report of the National Science Foundation blue ribbon panel on simulation-based engineering science (2006). http://www.nsf.gov/pubs/reports/sbes_final_report.pdf
Oliver, T.A., Terejanu, G., Simmons, C.S., Moser, R.D.: Validating predictions of unobserved quantities. Comput. Methods Appl. Mech. Eng. 283, 1310–1335 (2015). doi:http://dx.doi.org/10.1016/j.cma.2014.08.023, http://www.sciencedirect.com/science/article/pii/S004578251400293X
Petra, N., Martin, J., Stadler, G., Ghattas, O.: A computational framework for infinite-dimensional Bayesian inverse problems, Part II: stochastic Newton mcmc with application to ice sheet flow inverse problems. SIAM J. Sci. Comput. 36(4), A1525–A1555 (2014)
Rubin, D.B.: Bayesianly justifiable and relevant frequency calculations for the applied statistician. Ann. Stat. 12, 1151–1172 (1984)
Sentz, K., Ferson, S.: Combination of evidence in Dempster–Shafer theory. Technical report SAND 2002-0835, Sandia National Laboratory (2002)
Shafer, G.: A Mathematical Theory of Evidence. Princeton University Press, Princeton (1976)
Van Horn, K.S.: Constructing a logic of plausible inference: a guide to Cox’s theorem. Int. J. Approx. Reason. 34(1), 3–24 (2003)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2015 Springer International Publishing Switzerland
About this entry
Cite this entry
Moser, R., Oliver, T. (2015). Validation of Physical Models in the Presence of Uncertainty. In: Ghanem, R., Higdon, D., Owhadi, H. (eds) Handbook of Uncertainty Quantification. Springer, Cham. https://doi.org/10.1007/978-3-319-11259-6_2-1
Download citation
DOI: https://doi.org/10.1007/978-3-319-11259-6_2-1
Received:
Accepted:
Published:
Publisher Name: Springer, Cham
Online ISBN: 978-3-319-11259-6
eBook Packages: Springer Reference MathematicsReference Module Computer Science and Engineering