Advertisement

Simulation Validation from a Bayesian Perspective

  • Claus BeisbartEmail author
Chapter
Part of the Simulation Foundations, Methods and Applications book series (SFMA)

Abstract

Bayesian epistemology offers a powerful framework for characterizing scientific inference. Its basic idea is that rational belief comes in degrees that can be measured in terms of probabilities. The axioms of the probability calculus and a rule for updating (e.g., Bayesian conditionalization) emerge as constraints on the formation of rational belief. Bayesian epistemology has led to useful explications of notions such as confirmation. It thus is natural to ask whether Bayesian epistemology offers a useful framework for thinking about the inferences implicit in the validation of computer simulations. The aim of this chapter is to answer this question. Bayesian epistemology is briefly summarized and then applied to validation. Updating is shown to form a viable method for data-driven validation. Bayesians can also express how a simulation obtains prior credibility because the underlying conceptual model is credible. But the impact of this prior credibility is indirect since simulations at best provide partial and approximate solutions to the conceptual model. Fortunately, this gap between the simulations and the conceptual model can be addressed using what we call Bayesian verification. The final part of the chapter systematically evaluates the use of Bayesian epistemology in validation, e.g., by comparing it to a falsificationist approach. It is argued that Bayesian epistemology goes beyond mere calibration and that it can provide the foundations for a sound evaluation of computer simulations.

Keywords

Degree of belief Bayesian updating Problem of old evidence Error Uncertainty Confirmation Verification 

Notes

Acknowledgements

This chapter has benefited from helpful comments by Seamus Bradley, interesting critical suggestions and comments by Nicole J. Saam, and an extensive and extremely valuable commentary by Stephan Poppe. Thanks to all of them!

References

  1. Albert, M. (1999). Bayesian learning when chaos looms large. Economics Letters, 65, 1–7.MathSciNetCrossRefGoogle Scholar
  2. Bartelborth, T. (2013). Sollten wir klassische Überzeugungssysteme durch Bayesianische ersetzen? Logos, 3, 2–68.Google Scholar
  3. Bayarri, M. J., Berger, J. O., Higdon, D., Kennedy, M. C., Kottas, A., Paulo, R., Sacks, J., Cafeo, J. A., Cavendish, J., Lin, C. H., & Tu, J. (2002). A framework for validation of computer models. Tech. rep. Obtained from http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.125.7166&rank=5.
  4. Bayarri, M. J., Berger, J. O., Paulo, R., Sacks, J., Cafeo, J. A., James Cavendish, C.-H. L., et al. (2005). A framework for validation of computer models. NISS, Technical Report Number 162.Google Scholar
  5. Bayarri, M. J., Berger, J. O., Paulo, R., Sacks, J., Cafeo, J. A., Cavendish, J., et al. (2007). A framework for validation of computer models. Technometrics, 49(2), 138–154.MathSciNetCrossRefGoogle Scholar
  6. Beisbart, C. (2011). A rational approach to risk? Bayesian decision theory. In S. Roeser, R. Hillerbrand, P. Sandin, & M. Peterson (Eds.), Handbook of risk theory (pp. 375–404). Berlin: Springer.Google Scholar
  7. Beisbart, C. (2012). How can computer simulations produce new knowledge? European Journal for Philosophy of Science, 2, 395–434.CrossRefGoogle Scholar
  8. Blackwell, D., & Dubins, L. (1962). Merging of opinions with increasing information. Annals of Statistical Mathematics, 33, 882–886.MathSciNetCrossRefGoogle Scholar
  9. Bogen, J., & Woodward, J. (1988). Saving the phenomena. Philosophical Review, 97, 303–352.CrossRefGoogle Scholar
  10. Bovens, L., & Hartmann, S. (2004). Bayesian epistemology. Oxford: Oxford University Press.CrossRefGoogle Scholar
  11. Bradley, S. (2016). Imprecise probabilities. In E. N. Zalta (Ed.), The Stanford encyclopedia of philosophy. Metaphysics Research Lab, Stanford University, winter 2016 ed.Google Scholar
  12. Chen, W., Xiong, Y., Tsui, K.-L., & Wang, S. (2006). Some metric and Bayesian procedure for validating predictive models in engineering design. In Proceedings of IDETC, CIE 2006 ASME 2006 International Design Engineering Technical Conferences & Computers and Information in Engineering Conference September 10–13, 2006. Philadelphia: Pennsylvania, USA.Google Scholar
  13. Chen, W., Xiong, Y., Tsui, K.-L., & Wang, S. (2008). A design-driven validation approach using Bayesian prediction models. Journal of Mechanical Design, 130(2), 021101.CrossRefGoogle Scholar
  14. de Finetti, B. (1931a). Probabilismo. Logos, 14, 163–219. Translated as de Finetti, B. (1989). Probabilism. A critical essay on the theory of probability and on the value of science. Erkenntnis, 31, 169–223.Google Scholar
  15. de Finetti, B. (1931b). Sul significato soggetivo della probabilità. Fundamenta Mathematica, 17, 298–329.Google Scholar
  16. de Finetti, B. (1937). La prévision: ses lois logiques, ses sources subjectives. Annales de l’Institut Henri Poincaré, 7, 1–68. Here quoted from the English translation: de Finetti, B. (1964). Foresight: Its logical laws, its subjective sources. In H. E. Kyburg, & H. E. Smokler (Eds.), Studies in subjective probability (pp. 53–118). Wiley.Google Scholar
  17. Döring, F. (2000). Conditional probability and Dutch books. Philosophy of Science, 67(3), 391–409. http://www.jstor.org/stable/188624MathSciNetCrossRefGoogle Scholar
  18. Earman, J. (1992). Bayes or bust. Cambridge (MA): MIT Press.Google Scholar
  19. Easwaran, K. (2011a). Bayesianism I: Introduction and arguments in favor. Philosophy Compass, 6(5), 312–320.CrossRefGoogle Scholar
  20. Easwaran, K. (2011b). Bayesianism II: Applications and criticisms. Philosophy Compass, 6(5), 321–332.CrossRefGoogle Scholar
  21. Frigg, R., Bradley, S., Du, H., & Smith, L. A. (2014). Laplace’s demon and the adventures of his apprentices. Philosophy of Science, 81(1), 31–59.MathSciNetCrossRefGoogle Scholar
  22. Gettier, E. (1963). Is justified true belief knowledge? Analysis, 23(6), 121–123.CrossRefGoogle Scholar
  23. Gilbert, M. (1987). Modeling collective belief. Synthese, 73, 185–204.CrossRefGoogle Scholar
  24. Gillies, D. (2000). Philosophical theories of probability. London and New York: Routledge.Google Scholar
  25. Glymour, C. N. (1980). Theory and evidence. Princeton: Princeton University Press.Google Scholar
  26. Hájek, A. (2012). Interpretations of probability. In E. N. Zalta (Ed.), The Stanford encyclopedia of philosophy. (Winter 2012 ed).Google Scholar
  27. Hájek, A., & Hartmann, S. (2010a). Bayesian epistemology. In J. D. et al. (Ed.) A companion to epistemology, (pp. 93–106). Oxford: Blackwell.Google Scholar
  28. Hájek, A., & Hartmann, S. (2010b). Bayesian epistemology. A companion to epistemology (pp. 93–105). Oxford: Blackwell.Google Scholar
  29. Hartmann, S., & Sprenger, J. (2010). Bayesian epistemology. In D. Pritchard & S. Bernecker (Eds.), Routledge companion to epistemology (pp. 609–620). London: Routledge.Google Scholar
  30. Hempel, C. G. (1945). Studies in the logic of confirmation (I.). Mind, 54, 1–26. reprinted in L. Sklar, (2000). Philosophy of science. Probability and confirmation (pp. 245–270). Garland, New York.Google Scholar
  31. Howson, C., & Urbach, P. (2006). Scientific reasoning: The Bayesian approach (3rd ed.). Open Court: La Salle.Google Scholar
  32. Huber, F. (2007). Confirmation and induction. In J. Fieser, & B. Dowden (Eds.), Internet encyclopedia of philosophy.Google Scholar
  33. Huber, F. (2009). Belief and degrees of belief. In F. Huber, & C. Schmidt-Petri (Eds.), Degrees of belief. Springer.Google Scholar
  34. Jaynes, E. T. (1968). Prior probabilities. IEEE Transactions on Systems Science and Cybernetics, 4/3, 227–241. http://bayes.wustl.edu/etj/articles/prior.pdf.CrossRefGoogle Scholar
  35. Jaynes, E. T. (1957). Information theory and statistical mechanics. Physical Review, 106, 620–630.MathSciNetCrossRefGoogle Scholar
  36. Jaynes, E. T. (1979). Where do we stand on maximum entropy? In R. D. Levine & M. Tribus (Eds.), The maximum entropy formalism (pp. 15–118). Cambridge (MA): M. I. T. Press.Google Scholar
  37. Jeffrey, R. (1967). The logic of decision (2nd ed.). New York: McGraw-Hill.Google Scholar
  38. Jensen, K. K. (2011). A philosophical assessment of decision theory. In S. Roeser, R. Hillerbrand, P. Sandin, & M. Peterson (Eds.), Handbook of risk theory (pp. 405–439). Berlin: Springer.Google Scholar
  39. Joyce, J. (2016). Bayes’ theorem. In E. N. Zalta (Ed.), The Stanford encyclopedia of philosophy. Metaphysics Research Lab, Stanford University (winter 2016 ed.).Google Scholar
  40. Kennedy, M. C., & O’Hagan, A. (2000). Predicting the output from a complex computer code when fast approximations are available. Biometrika, 87(1), 1–13.MathSciNetCrossRefGoogle Scholar
  41. Kennedy, M. C., & O’Hagan, A. (2001). Bayesian calibration of computer models. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 63(3), 425–464.MathSciNetCrossRefGoogle Scholar
  42. Keuth, H. (2000). Die Philosophie Karl Poppers. Tübingen: UTB, Mohr und Siebeck.Google Scholar
  43. Keynes, J. M. (1921). A treatise on probability. London: Macmillan. (Reprint 1973).Google Scholar
  44. Knutti, R. (2008). Should we believe model predictions of future climate change? Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, 366(1885), 4647–4664.  https://doi.org/10.1098/rsta.2008.0169.MathSciNetCrossRefGoogle Scholar
  45. Kolmogorov, A. N. (1956). Foundations of the theory of probability (Second English edition). Chelsea.Google Scholar
  46. Lee, P. M. (2012). Bayesian statistics. An introduction (4th ed.). Chichester: Wiley.Google Scholar
  47. Lewis, D. (1980). A subjectivist’s guide to objective chance. In Studies in inductive logic and probability (vol. II, pp. 263–293). Berkeley and Los Angeles: University of California Press. (Here quoted from Lewis (1986) pp. 83–113).Google Scholar
  48. Lewis, D. (1986). Philosophical papers (Vol. II). New York: Oxford University Press.zbMATHGoogle Scholar
  49. Lewis, D. (1994). Humean supervenience debugged. Mind, 103, 473–490. Reprinted in Lewis, D. (1999). Papers in metaphysics and epistemology. Cambridge: Cambridge University Press.Google Scholar
  50. Lewis, D. (1997). Why conditionalize? In D. Lewis (Ed.), Papers in metaphysics and epistemology (pp. 403–407). Cambridge: Cambridge University Press.Google Scholar
  51. Mayo, D. (1996). Error and the growth of experimental knowledge. Chicago: University of Chicago Press.CrossRefGoogle Scholar
  52. Mellor, H. D. (2005). Probability. A philosophical introduction. London and New York: Routledge.Google Scholar
  53. Myrvold, W. C. (2003). A Bayesian account of the virtue of unification. Philosophy of science, 70(2), 399–423.MathSciNetCrossRefGoogle Scholar
  54. Neapolitan, R. E. (2004). Learning Bayesian networks. Upper Saddle River, NJ: Pearson Prentice Hall.Google Scholar
  55. Oakley, J. E., & O’Hagan, A. (2004). Probabilistic sensitivity analysis of complex models: A Bayesian approach. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 66(3), 751–769.MathSciNetCrossRefGoogle Scholar
  56. Oberkampf, W. L., & Barone, M. F. (2006). Measures of agreement between computation and experiment: Validation metrics. Journal of Computational Physics, 217, 5–36. http://portal.acm.org/citation.cfm?id=1167051.1167053
  57. Oberkampf, W. L., & Roy, C. J. (2010). Verification and validation in scientific computing. Cambridge: Cambridge University Press.CrossRefGoogle Scholar
  58. O’Hagan, A. (2006). Bayesian analysis of computer code outputs: A tutorial. Reliability Engineering & System Safety, 91(10), 1290–1300.CrossRefGoogle Scholar
  59. Papoulis, A., & Pillai, S. U. (2002). Probability, random variables, and stochastic processes (4th ed.). Boston etc.: McGraw-Hill.Google Scholar
  60. Parker, W. S. (2009). Confirmation and adequacy-for-purpose in climate modelling. Aristotelian Society Supplementary, 83(1), 233–249.CrossRefGoogle Scholar
  61. Plato. (2015). Theatetus and sophist. Cambridge: Cambridge University Press. (Edited by C. Rowe).Google Scholar
  62. Popper, K. R. (1934). Die Logik der Forschung. Zur Erkenntnistheorie der modernen Naturwissenschaft. Wien: Julius Springer. (English version: The logic of scientific discovery, London: Hutchinson, 1959).Google Scholar
  63. Popper, K. R. (1969). Conjectures and refutations (3rd ed.). London: Routlege and Kegan Paul.Google Scholar
  64. Putnam, H. (1974). The ‘corroboration’ of theories. In P. A. Schilpp (Ed.), The philosophy of karl popper (pp. 221–240). La Salle (IL): Open Court.Google Scholar
  65. Ramsey, F. (1926). Truth and probability. First printed in: Braithwaite, R. B. (Ed.), Foundations of mathematics and other essays (pp. 156–198). Routledge and P. Kegan, London 1931. Reprinted in Ramsey, F. (1990). Philosophical papers. In D. H. Mellor (Ed.). (pp. 52–94). Cambridge: University Press.Google Scholar
  66. Rawls, J. (1971). A theory of justice. Harvard University Press: Cambridge (MA). (Quoted from the revised edition 1999).Google Scholar
  67. Roy, C. J., & Oberkampf, W. L. (2010). A complete framework for verification, validation, and uncertainty quantification in scientific computing. In AIAA 2010-124, 48th AIAA Aerospace Sciences Meeting Including the New Horizons Forum and Aerospace Exposition. Published by American Institute of Aeronautics and Astronautics, Inc.Google Scholar
  68. Salmon, W. (1981). Rational prediction. British Journal for the Philosophy of Science, 32, 115–125.CrossRefGoogle Scholar
  69. Savage, L. J. (1972). The foundations of statistics (2nd ed.). New York: Dover. (1st ed.) Wiley: New York (1954).Google Scholar
  70. Sprenger, J. (2015). A novel solution to the problem of old evidence. Philosophy of Science, 82(3), 383–401.MathSciNetCrossRefGoogle Scholar
  71. Strevens, M. (2006). The Bayesian approach to the philosophy of science. In D. M. Borchert (Ed.), Encyclopedia of philosophy (2nd ed., pp. 495–502). Macmillan Reference.Google Scholar
  72. Talbott, W. (2016). Bayesian epistemology. In E. N. Zalta (Ed.), The Stanford encyclopedia of philosophy. Metaphysics Research Lab, Stanford University (winter 2016 ed.)Google Scholar
  73. Teller, P. (1973). Conditionalization and observation. Synthese, 26, 218–258.CrossRefGoogle Scholar
  74. von Plato, J. (1989). De Finetti’s earliest works on the foundations of probability. Erkenntnis, 31, 263–282.CrossRefGoogle Scholar
  75. von Plato, J. (1994). Creating modern probability. Cambridge: Cambridge University Press.CrossRefGoogle Scholar
  76. Wagner, C. G. (1997). Old evidence and new explanation. Philosophy of Science, 64(4), 677–691.MathSciNetCrossRefGoogle Scholar
  77. Wang, S., Chen, W., & Tsui, K.-L. (2009). Bayesian validation of computer models. Technometrics, 51(4), 439–451.MathSciNetCrossRefGoogle Scholar
  78. Wenmackers, S., & Romeijn, J. (2016). New theory about old evidence. Synthese, 193(4).Google Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  1. 1.Institute of PhilosophyUniversity of BernBernSwitzerland

Personalised recommendations