Advertisement

Validation Using Bayesian Methods

  • Xiaomo JiangEmail author
  • Xueyu Cheng
  • Yong Yuan
Chapter
Part of the Simulation Foundations, Methods and Applications book series (SFMA)

Abstract

Quantitative model validation is playing an increasingly important role in performance and reliability assessment of a complicated system whenever simulation is involved. This chapter discusses model validation from a Bayesian perspective, considering in particular data uncertainty. First, Bayes’ theorem is defined, then the Bayesian risk rule method is introduced. Explicit expressions for the Bayesian interval hypothesis testing approach are presented in both univariate and multivariate cases. The problem of non-normal validation data is addressed by the Box–Cox transformation. A generalized procedure is presented to implement Bayesian validation methods. Classic hypothesis testing method is utilized to conduct a comparison study. The impact of the data normality assumption and of the variation of the threshold on model assessment accuracy is investigated by using both classical and Bayesian approaches. The Bayesian methodology is illustrated with a reliability model of rotor blades, a univariate stochastic damage accumulation model, and a multivariate heat conduction problem.

Keywords

Bayesian statistics Hypothesis testing Box–Cox transformation Bayes network Model validation Non-normality 

References

  1. AIAA. (1998). Guide for the Verification and Validation of Computational Fluid Dynamics Simulations, American Institute of Aeronautics and Astronautics, AIAA-G-077-1998, Reston, Virginia, USA.Google Scholar
  2. Andrews, D. F., Gnanadesikan, R., & Warner, J. L. (1971). Transformations of multivariate data. Biometrika, 27(4), 825–840.CrossRefGoogle Scholar
  3. Annis, C. (2002). Modeling high cycle fatigue with Markov chain Monte Carlo-a new look at an old idea. In: Proceedings of 43rd AIAA/ASME/ASCE/AHS/ASC Structure, Structural Dynamics and Materials Conference (AIAA-2002-1380) (pp. 1351–1361). Denver CO 2002.Google Scholar
  4. ASME. (2002). Verification & Validation (V&V) methodology and quantitative reliability at confidence: Basis for an investment strategy, PTC 60, UCRL-ID-150874, Livermore, CA: Lawrence Livermore National Laboratory.Google Scholar
  5. ASME. (2006). Guide for verification and validation in computational solid mechanics. ASME V&V 10-2006. New York, NY: American Society of Mechanical Engineers.Google Scholar
  6. Babuska, I., & Oden, J. T. (2004). Verification and validation in computational engineering and science: Basic concepts. Computer Methods in Applied Mechanics and Engineering, 193(36–38), 4057–4066.MathSciNetzbMATHCrossRefGoogle Scholar
  7. Balci, O., & Sargent, R. G. (1981). A methodology for cost-risk analysis in the statistical validation of simulation models. Communications of Association for Computing Machinery (ACM), 24(11), 190–197.MathSciNetCrossRefGoogle Scholar
  8. Berger, J. O., & Delampady, M. (1987). Testing precise hypotheses. Statistical Science, 2(3), 317–352.MathSciNetzbMATHCrossRefGoogle Scholar
  9. Box, G. E. P., & Cox, D. R. (1964). An analysis of transformations. Journal of the Royal Statistical Society, 26(2), 211–252.MathSciNetzbMATHGoogle Scholar
  10. Chen, W., Baghdasaryan, L., Buranathiti, T., & Cao, J. (2004). Model validation via uncertainty propagation and data transformations. AIAA Journal, 42(7), 1406–1415.CrossRefGoogle Scholar
  11. Chen, W., Xiong, Y., Tsui, K. -L., & Wang, S. (2008). A design-driven validation approach using Bayesian prediction models. Journal of Mechanical Design, 130(2), 021101-1-12.Google Scholar
  12. DOD. (1996). Verification, validation, and accreditation (VV&A) recommended practices guide. Alexandria, VA: Department of Defense.Google Scholar
  13. DOE. (2000). Accelerated strategic computing initiative (ASCI) program plan. DOE/DP-99-000010592, Washington, DC: Department of Energy.Google Scholar
  14. Dowding, K. J., Pilch, M., & Hills, R. G. (2008). Formulation of the thermal problem. Computer Methods in Applied Mechanics and Engineering (special issue), 197(29–32), 2385–2389.zbMATHCrossRefGoogle Scholar
  15. Genz, A. (1992). Numerical computation of multivariate normal probabilities. Journal of Computational and Graphical Statistics, 1(2), 141–149.Google Scholar
  16. Hills, R.G., & Trucano, T. G. (2002). Statistical validation of engineering and scientific models: A maximum likelihood based metric. Technical Report Sand. No 2001-1783, Albuquerque, NM: Sandia National Laboratories.Google Scholar
  17. Hwang, J. T., Casella, G., Robert, C., Wells, M. T., & Farrell, R. H. (1992). Estimation of accuracy in testing. The Annals of Statistics, 20(1), 490–509.MathSciNetzbMATHCrossRefGoogle Scholar
  18. Jeffreys, H. (1961). Theory of probability (3rd ed.). London, UK: Oxford University Press.zbMATHGoogle Scholar
  19. Jensen, F. V., & Jensen, F. B. (2001). Bayesian networks and decision graphs. New York: Springer.zbMATHCrossRefGoogle Scholar
  20. Jiang, X., & Mahadevan, S. (2007). Bayesian risk-based decision method for model validation under uncertainty. Reliability Engineering and System Safety, 92(6), 707–718.CrossRefGoogle Scholar
  21. Jiang, X., & Mahadevan, S. (2008a). Bayesian wavelet method for multivariate model assessment of dynamic systems. Journal of Sound and Vibration, 312(4–5), 694–712.CrossRefGoogle Scholar
  22. Jiang, X., & Mahadevan, S. (2008b). Bayesian validation assessment of multivariate computational models. Journal of Applied Statistics, 35(1), 49–65.MathSciNetzbMATHCrossRefGoogle Scholar
  23. Jiang, X., & Mahadevan, S. (2009a). Bayesian inference method for model validation and confidence extrapolation. Journal of Applied Statistics, 36(6), 659–677.MathSciNetzbMATHCrossRefGoogle Scholar
  24. Jiang, X., & Mahadevan, S. (2009b). Bayesian structural equation modelling method for hierarchical model validation. Reliability Engineering and System Safety, 94(4), 796–809.CrossRefGoogle Scholar
  25. Jiang, X., & Mahadevan, S. (2010). Bayesian nonlinear SEM approach for hierarchical validation of dynamical systems. Mechanical Systems and Signal Processing, 24(4), 957–975.CrossRefGoogle Scholar
  26. Jiang, X., Yuan, Y., Mahadevan, S., & Liu, X. (2013a). An Investigation of Bayesian inference approach to model validation with non-normal data. Journal of Statistical Computation and Simulation, 83(10), 1829–1851.MathSciNetCrossRefGoogle Scholar
  27. Jiang, X., Yuan, Y., & Liu, X. (2013b). Bayesian inference method for stochastic damage accumulation modeling. Reliability Engineering and System Safety, 111(3), 126–138.CrossRefGoogle Scholar
  28. Kass, R., & Raftery, A. (1995). Bayes factors. Journal of the American Statistical Association, 90(430), 773–795.MathSciNetzbMATHCrossRefGoogle Scholar
  29. Kennedy, M. C., & O’Hagan, A. (2001). Bayesian calibration of computer experiments. Journal of the Royal Statistical Society Series B (Statistical Methodology), 63(3), 425–464.MathSciNetzbMATHCrossRefGoogle Scholar
  30. Mahadevan, S., & Rebba, R. (2005). Validation of reliability computational models using Bayes networks. Reliability Engineering and System Safety, 87(2), 223–232.CrossRefGoogle Scholar
  31. Migon H. S., & Gamerman, D. (1999). Statistical inference-an integrated approach. London: Arnold, a Member of the Holder Headline Group.Google Scholar
  32. Nataf, A. (1962). Détermination des distributions de probalités dont les marges sont données. Comptes Rendus de l’Académie des Sciences, 225, 42–43.zbMATHGoogle Scholar
  33. NIST/SEMATECH (2005). e-Handbook of statistical methods, National Institute of Standards and Technology. Retrieved from http://www.itl.nist.gov/div898/handbook.
  34. Nowak, R., & Scott, C. (2004). The Bayes risk criterion in hypothesis testing. Connexions, Retrieved from http://cnx.rice.edu/content/m11533/1.6/.
  35. Oberkampf, W. L., & Barone, M. F. (2006). Measures of agreement between computation and experiment: Validation metrics. Journal of Computational Physics, 217(1), 5–36.zbMATHCrossRefGoogle Scholar
  36. Oberkampf, W. L., & Trucano, T. G. (2002). Verification and validation in computational fluid dynamics. Progress in Aerospace Sciences, 38(3), 209–272.CrossRefGoogle Scholar
  37. Pericchi, L. R. (1981). A Bayesian approach to transformations to normality. Biometrika, 68(1), 35–43.MathSciNetzbMATHCrossRefGoogle Scholar
  38. Rebba, R., & Mahadevan, S. (2006a). Model predictive capability assessment under uncertainty. AIAA Journal, 44(10), 2376–2384.CrossRefGoogle Scholar
  39. Rebba, R., & Mahadevan, S. (2006b). Validation of models with multivariate outputs. Reliability Engineering and System Safety, 91(8), 861–871.CrossRefGoogle Scholar
  40. Rebba, R., & Mahadevan, S. (2008). Computational methods for model reliability assessment. Reliability Engineering and System Safety, 93(8), 1197–1207.CrossRefGoogle Scholar
  41. Roache, P. J. (1998). Verification and validation in computational science and engineering in: Science and engineering. Albuquerque, NM: Hermosa Publishers.Google Scholar
  42. Rosenblatt, M. (1952). Remarks on multivariate transformation. Annals of Mathematical Statistics, 23(3), 470–472.MathSciNetzbMATHCrossRefGoogle Scholar
  43. Schlesinger, S. (1979). Terminology for model credibility. Simulation, 32(3), 103–104.CrossRefGoogle Scholar
  44. Schervish, M. J. (1995). Theory of statistics. New York: Springer.zbMATHCrossRefGoogle Scholar
  45. Schwer, L. E. (2007). Validation metrics for response histories: Perspectives and case studies. Engineering with Computers, 23(4), 295–309.CrossRefGoogle Scholar
  46. Stephens, M. A. (1974). EDF statistics for goodness of fit and some comparisons. Journal of the American Statistical Association, 69(347), 730–737.CrossRefGoogle Scholar
  47. Zhang, R., & Mahadevan, S. (2003). Bayesian methodology for reliability model acceptance. Reliability Engineering and System Safety, 80(1), 95–103.CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  1. 1.Tongji UniversityShanghaiChina
  2. 2.Clayton State UniversityMorrowUSA

Personalised recommendations