Advertisement

Simulation Accuracy, Uncertainty, and Predictive Capability: A Physical Sciences Perspective

  • William L. OberkampfEmail author
Chapter
Part of the Simulation Foundations, Methods and Applications book series (SFMA)

Abstract

Most computational analysts, as well as most governmental policy-makers and the public, view computational simulation accuracy as a good agreement of simulation results with empirical measurements. However, decision-makers, such as business managers and safety regulators who rely on simulation for decision support, view computational simulation accuracy as much more than agreement of simulation results with experimental data. Decision-makers’ concept of accuracy is better captured by the term predictive capability of the simulation. Predictive capability meaning the use of a computational model to foretell or forecast the response of a system to conditions without available experimental data, even for system responses that have never occurred in nature. This chapter makes this important distinction by discussing the crucial ingredients needed for predictive capability: code verification, solution (or calculation) verification, model validation, model calibration, and predictive uncertainty estimation. Each of these ingredients is required, whether the simulation results are used in the generation of new knowledge, or for decision support by business managers, government policy-makers, or safety regulators.

Keywords

Verification Validation Uncertainty quantification Predictive capability Model calibration 

Acronyms

CFD

Computational fluid dynamics

MFE

Model form error

MMS

Method of manufactured solutions

PDE

Partial differential equation

SRQ

System response quantity

SQA

Software quality assurance

Notes

Acknowledgements

I thank Drs. Timothy Trucano, Patrick Roache, and Theodore Kneupper for carefully reviewing an earlier version of this chapter and providing many helpful suggestions for improvements and clarifications.

References

  1. Abanto, J., Pelletier, D., Garon, A., Trepanier, J.-Y., & Reggio, M. (2005). Verification of some commercial CFD codes on atypical CFD problems. Paper presented at the 43rd AIAA Aerospace Sciences Meeting and Exhibit, Reno, NV.Google Scholar
  2. Aeschliman, D. P., & Oberkampf, W. L. (1998). Experimental methodology for computational fluid dynamics code validation. AIAA Journal, 36(5), 733–741.CrossRefGoogle Scholar
  3. AIAA. (1998). Guide for the verification and validation of computational fluid dynamics simulations (AIAA-G-077-1998). Retrieved from Reston, VA.Google Scholar
  4. Ainsworth, M., & Oden, J. T. (2000). A posteriori error estimation in finite element analysis. New York: Wiley.zbMATHCrossRefGoogle Scholar
  5. Anderson, M. G., & Bates, P. D. (Eds.). (2001). Model validation: Perspectives in hydrological science. New York, NY: Wiley.Google Scholar
  6. ASME. (2006). Guide for verification and validation in computational solid mechanics (ASME Standard V&V 10-2006). Retrieved from New York, NY.Google Scholar
  7. ASME. (2009). Standard for verification and validation in computational fluid dynamics and heat transfer (ASME Standard V&V 20-2009). Retrieved from New York, NY.Google Scholar
  8. ASME. (2012). An illustration of the concepts of verification and validation in computational solid mechanics (ASME Standard V&V 10.1-2012). Retrieved from New York, NY.Google Scholar
  9. Augustin, T., Coolen, F. P. A., de Cooman, G., & Troffaes, M. C. M. (Eds.). (2014). Introduction to imprecise probabilities. Chichester, UK: Wiley.zbMATHGoogle Scholar
  10. Babuska, I., & Strouboulis, T. (2001). The finite element method and its reliability. Oxford, U.K.: Oxford University Press.zbMATHGoogle Scholar
  11. Babuska, I., Nobile, F., & Tempone, R. (2008). A systematic approach to model validation based on bayesian updates and prediction related rejection criteria. Computer Methods in Applied Mechanics and Engineering, 197(29–32), 2517–2539.zbMATHCrossRefGoogle Scholar
  12. Bayarri, M. J., Berger, J. O., Paulo, R., Sacks, J., Cafeo, J. A., Cavendish, J., et al. (2007). A framework for validation of computer models. Technometrics, 49(2), 138–154.MathSciNetCrossRefGoogle Scholar
  13. Bernardini, A., & Tonon, F. (2010). Bounding uncertainty in civil engineering. Berlin: Springer.zbMATHCrossRefGoogle Scholar
  14. Beven, K. (2002). Towards a coherent philosophy of modelling the environment. Proceedings of the Royal Society of London Series A, 458(2026), 2465–2484.MathSciNetzbMATHCrossRefGoogle Scholar
  15. Bossel, H. (1994). Modeling and simulation (1st ed.). Wellesley, MA: A. K. Peters.zbMATHCrossRefGoogle Scholar
  16. Chen, W., Xiong, Y., Tsui, K.-L., & Wang, S. (2008). A design-driven validation approach using bayesian prediction models. Journal of Mechanical Design, 130(2), 021101–021112.Google Scholar
  17. Chiles, J.-P., & Delfiner, P. (1999). Geostatistics: Modeling spatial uncertainty. New York: Wiley.zbMATHCrossRefGoogle Scholar
  18. Cooke, R. (2004). The antomy of the squizzel: The role of operational definitions in representing uncertainty. Reliability Engineering and System Safety, 85(1–3), 313–319.CrossRefGoogle Scholar
  19. Cullen, A. C., & Frey, H. C. (1999). Probabilistic techniques in exposure assessment: A handbook for dealing with variability and uncertainty in models and inputs. New York: Plenum Press.Google Scholar
  20. Davey, S., Gordon, N., Holland, I., Rutten, M., & Williams, J. (2016). Bayesian methods in the search for MH370. Springer Nature (Open Access).Google Scholar
  21. Donoho, D. L., Maleki, A., Shahram, M., Rahman, I. U., & Stodden, V. (2009). Reproducible research in computational harmonic analysis. Computing in Science & Engineering, 11(1), 8–18.CrossRefGoogle Scholar
  22. Duggirala, R. K., Roy, C. J., Saeidi, S. M., Khodadadi, J. M., Cahela, D. R., & Tatarchuk, B. J. (2008). Pressure drop predictions in microfibrous materials using computational fluid dynamics. Journal of Fluids Engineering, 130(7), 071302–071313.Google Scholar
  23. Eca, L., & Hoekstra, M. (2002). An evaluation of verification procedures for CFD applications. Paper presented at the Proceedings of the 24th Symposium on Naval Hydrodynamics, Fukuoka, Japan.Google Scholar
  24. Ferson, S., & Oberkampf, W. L. (2009). Validation of imprecise probability models. International Journal of Reliability and Safety, 3(1–3), 3–22.CrossRefGoogle Scholar
  25. Ferson, S., Oberkampf, W. L., & Ginzburg, L. (2008). Model validation and predictive capability for the thermal challenge problem. Computer Methods in Applied Mechanics and Engineering, 197(29–32), 2408–2430.zbMATHCrossRefGoogle Scholar
  26. Ferziger, J. H., & Peric, M. (2002). Computational methods for fluid dynamics (3rd ed.). New York: Springer.zbMATHCrossRefGoogle Scholar
  27. Fomel, S., & Claerbout, J. F. (2009). Guest editors’ introduction: Reproducible research. Computing in Science & Engineering, 11(1), 5–7.CrossRefGoogle Scholar
  28. Golub, G. H., & Van Loan, C. F. (2013). Matrix computations (4th ed.). Baltimore, MD: The Johns Hopkins University Press.zbMATHGoogle Scholar
  29. Haimes, Y. Y. (2009). Risk modeling, assessment, and management (3rd ed.). New York: Wiley.zbMATHGoogle Scholar
  30. Hatton, L. (1997). The T experiments: Errors in scientific software. IEEE Computational Science & Engineering, 4(2), 27–38.CrossRefGoogle Scholar
  31. Higdon, D., Nakhleh, C., Gattiker, J., & Williams, B. (2008). A bayesian calibration approach to the thermal problem. Computer Methods in Applied Mechanics and Engineering, 197(29–32), 2431–2441.zbMATHCrossRefGoogle Scholar
  32. Jolliffe, I. T., & Stephenson, D. B. (Eds.). (2011). Forecast verification: A practitioner’s guide in atmospheric science (2nd ed.). Hoboken, NJ: Wiley.Google Scholar
  33. Kennedy, M. C., & O’Hagan, A. (2001). Bayesian calibration of computer models. Journal of the Royal Statistical Society Series B-Statistical Methodology, 63(3), 425–450.MathSciNetzbMATHCrossRefGoogle Scholar
  34. Knupp, P., & Salari, K. (2002). Verification of computer codes in computational science and engineering. Boca Raton, FL: Chapman & Hall/CRC.zbMATHCrossRefGoogle Scholar
  35. Lehmann, E. L., & Romano, J. P. (2005). Testing statistical hypotheses (3rd ed.). Berlin: Springer.zbMATHGoogle Scholar
  36. Leijnse, A., & Hassanizadeh, S. M. (1994). Model definition and model validation. Advances in Water Resources, 17, 197–200.CrossRefGoogle Scholar
  37. Lenhard, J., & Winsberg, E. (2010). Holism, entrenchment, and the future of climate model pluralism. Studies in History and Philosophy of Modern Physics, 41, 253–262.CrossRefGoogle Scholar
  38. LeVeque, R. J., Mitchell, I. M., & Stodden, V. (2012). Reproducible research for scientific computing: Tools and strategies for changing the culture. Computing in Science & Engineering, 14(4), 13–17.CrossRefGoogle Scholar
  39. Li, W., Chen, W., Jiang, Z., Lu, Z., & Liu, Y. (2014). New validation metrics for models with multiple correlated responses. Reliability Engineering and System Safety, 127, 1–11.CrossRefGoogle Scholar
  40. Li, W., Chen, S., Jiang, Z., Apley, D. W., Lu, Z., & Chen, W. (2016). Integrating bayesian calibration, bias correction, and machine learning for the 2014 sandia verification and validation challent problem. Journal of Verification, Validation and Uncertainty Quantification, 1(1), 011004–011012.Google Scholar
  41. Liu, F., Bayarri, M. J., Berger, J. O., Paulo, R., & Sacks, J. (2008). A bayesian analysis of the thermal challenge problem. Computer Methods in Applied Mechanics and Engineering, 197(29–32), 2457–2466.zbMATHCrossRefGoogle Scholar
  42. Liu, Y., Chen, W., Arendt, P., & Huang, H.-Z. (2011). Toward a better understanding of model validation metrics. Journal of Mechanical Design, 133(13), 071001–071013.Google Scholar
  43. Marvin, J. G. (1995). Perspective on computational fluid dynamics validation. AIAA Journal, 33(10), 1778–1787.zbMATHCrossRefGoogle Scholar
  44. McFarland, J., & Mahadevan, S. (2008). Multivariate significance testing and model calibration under uncertainty. Computer Methods in Applied Mechanics and Engineering, 197(29–32), 2467–2479.zbMATHCrossRefGoogle Scholar
  45. Morgan, M. G., & Henrion, M. (1990). Uncertainty: A guide to dealing with uncertainty in quantitative risk and policy analysis (1st ed.). Cambridge, UK: Cambridge University Press.CrossRefGoogle Scholar
  46. Morrison, J. H. (2014). Statistical analysis of the fifth drag prediction workshop computational fluid dynamics solutions. Journal of Aircraft, 51(4), 1214–1222.CrossRefGoogle Scholar
  47. Neumann, P. G. (1995). Computer-related risks. New York: ACM Press, Addison-Wesley Publishing Company.Google Scholar
  48. Oberkampf, W. L., & Aeschliman, D. P. (1992). Joint computational/experimental aerodynamics research on a hypersonic vehicle: Part 1, experimental results. AIAA Journal, 30(8), 2000–2009.CrossRefGoogle Scholar
  49. Oberkampf, W. L., & Barone, M. F. (2006). Measures of agreement between computation and experiment: Validation metrics. Journal of Computational Physics, 217(1), 5–36.zbMATHCrossRefGoogle Scholar
  50. Oberkampf, W. L., & Trucano, T. G. (2008). Verification and validation benchmarks. Nuclear Engineering and Design, 238(3), 716–743.CrossRefGoogle Scholar
  51. Oberkampf, W. L., & Roy, C. J. (2010). Verification and validation in scientific computing. Cambridge, UK: Cambridge University Press.zbMATHCrossRefGoogle Scholar
  52. O’Hagan, A. (2006). Bayesian analysis of computer code outputs: A tutorial. Reliability Engineering and System Safety, 91(10–11), 1290–1300.CrossRefGoogle Scholar
  53. Reason, J. (1997). Managing the risks of organizational accidents. Burlington, VT: Ashgate Publishing Limited.Google Scholar
  54. Reason, J. (2008). The human contribution: Unsafe acts, accidents and heroic recoveries. Burlington, VT: Ashgate Publishing Co.Google Scholar
  55. Refsgaard, J. C., & Henriksen, H. J. (2004). Modelling guidelines-terminology and guiding principles. Advances in Water Resources, 27(1), 71–82.CrossRefGoogle Scholar
  56. Roache, P. J. (1972). Computational fluid dynamics. Albuquerque, NM: Hermosa Publishers.zbMATHGoogle Scholar
  57. Roache, P. (2009). Fundamentals of verification and validation. Socorro, New Mexico: Hermosa Publishers.Google Scholar
  58. Rougier, J. (2007). Probabilistic inference for future climate using an ensemble of climate model evaluations. Climate Change, 81(3–4), 247–264.CrossRefGoogle Scholar
  59. Roy, C. J. (2010). Review of discretization error estimators in scientific computing. Paper presented at the 48th AIAA Aerospace Sciences Meeting, Orlando, FL.Google Scholar
  60. Rumsey, C. L., Reif, B. A. P., & Gatski, T. B. (2006). Arbitrary steady-state solutions with the k–ε model. AAIAA Journal, 44(7), 1586–1592.CrossRefGoogle Scholar
  61. Rykiel, E. J. (1996). Testing ecological models: The meaning of validation. Ecological Modelling, 90(3), 229–244.CrossRefGoogle Scholar
  62. Silver, N. (2012). The signal and the noise. New York, NY: Penguin Books.Google Scholar
  63. Stodden, V. (2012). Guest editor’s introduction: Reproducible research-tools and strategies for scientific computing. IEEE Computing in Science and Engineering, 14(4), 11–12.CrossRefGoogle Scholar
  64. Taleb, N. N. (2007). The black swan: The impact of the highly improbable. New York: Random House.Google Scholar
  65. Taleb, N. N. (2008). The fourth quadrant: A map of the limits of statistics. Retrieved September 14, 2008, from https://www.edge.org/conversation/the-fourth-quadrant-a-map-of-the-limits-of-statistics.
  66. Trucano, T. G., Easterling, R. G., Dowding, K. J., Paez, T. L., Urbina, A., Romero, V. J., … Hills, R. G. (2001). Description of the sandia validation metrics project (SAND2001-1339). Retrieved from Albuquerque, NM.Google Scholar
  67. Trucano, T. G., Pilch, M., & Oberkampf, W. L. (2002). General concepts for experimental validation of asci code applications (SAND2002-0341). Retrieved from Albuquerque, NM.Google Scholar
  68. Trucano, T. G., Post, D. E., Pilch, M., & Oberkampf, W. L. (2005). Software engineering intersection with verification and validation of higher performance computational science software: Some observations (SAND2005-3662P). Retrieved from Albuquerque, NM.Google Scholar
  69. Trucano, T. G., Swiler, L. P., Igusa, T., Oberkampf, W. L., & Pilch, M. (2006). Calibration, validation, and sensitivity analysis: What’s what. Reliability Engineering and System Safety, 91(10–11), 1331–1357.CrossRefGoogle Scholar
  70. Verfurth, R. (2013). A posteriori error estimation techniques for finite element methods. Oxford, UK: Oxford University Press.zbMATHCrossRefGoogle Scholar
  71. Vose, D. (2008). Risk analysis: A quantitative guide (3rd ed.). New York: Wiley.zbMATHGoogle Scholar
  72. Voyles, I. T., & Roy, C. J. (2015). Evaluation of model validation techniques in the presence of aleatory and epistemic input uncertainties. Paper presented at the American Institute of Aeronautics and Astronautics SciTech Conference, Kissimmee, FL.Google Scholar
  73. Wang, S., Chen, W., & Tsui, K.-L. (2009). Bayesian validation of computer models. Technometrics, 51(4), 439–451.MathSciNetCrossRefGoogle Scholar
  74. Wellek, S. (2010). Testing statistical hypotheses of equivalence and noninferiority (2nd ed.). Boca Raton, FL: Chapman & Hall/CRC.zbMATHCrossRefGoogle Scholar
  75. Wilks, D. S. (2011). Statistical methods in the atmospheric sciences (3rd ed.). Amsterdam: Elsevier.Google Scholar
  76. Wolpert, R. L. (2004). A conversation with James O. Berger. Statistical Science, 19(1), 205–218.MathSciNetzbMATHCrossRefGoogle Scholar
  77. Ziliak, S. T., & McCloskey, D. N. (2008). The cult of statistical significance: How the standard error costs us jobs, justice, and lives. Ann Arbor, Michigan: University of Michigan Press.zbMATHGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  1. 1.Sandia National Laboratories - retiredAlbuquerqueUSA

Personalised recommendations