Confidence, credibility and prediction

Abstract

This paper examines, from a historical and model-based Bayesian perspective, two inferential issues: (1) the relation between confidence coverage and credibility of interval statements about model parameters; (2) the prediction of new values of a random variable. Confidence and credible intervals have different properties. This worries some statisticians, who want them to have the same properties, in frequentist repeated sampling coverage and Bayesian credibility content. Research continues on the conditions under which intervals have the same repeated sampling coverage and credibility content. We conclude that these two inferential approaches generally have incommensurable properties, which converge only asymptotically in sample size, and in a restricted class of samples. The repeated confusion, by new students of statistics, of the coverage of a confidence interval with the credibility of the observed interval, suggests that in general the credibility of the interval is the more important inferential aim. We show how the credibility of a confidence interval can be assessed generally. Bayesian prediction of new values is well established using the posterior predictive distribution. This has some curious features, known since Laplace but not well understood. The prediction of new Bernoullis highlights these features. We suggest that the credibility of predictive intervals needs to be reassessed.

This is a preview of subscription content, log in to check access.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Notes

  1. 1.

    The small expected frequencies in the extreme bins affected the validity of the asymptotic distribution, and so he suggested pooling of the extreme bins.

References

  1. 1.

    Agresti, A., Caffo, B.: Simple and effective confidence intervals for proportions and differences of proportions. Am. Stat. 54, 280–288 (2000)

    MATH  Google Scholar 

  2. 2.

    Anscombe, F.J.: Normal likelihood functions. Ann. Inst. Stat. Math. 16, 1–19 (1964)

    MathSciNet  Article  MATH  Google Scholar 

  3. 3.

    Aitchison, J., Dunsmore, I.R.: Statistical Prediction Analysis. Cambridge University Press, Cambridge (1975)

    Google Scholar 

  4. 4.

    Bayes, T.: An essay towards solving a problem in the doctrine of chances. Philos. Trans. R. Soc. Lond. 53, 370–418 (1764)

    MathSciNet  Article  MATH  Google Scholar 

  5. 5.

    Berger, J.O., Bernardo, J.M., Sun, D.: The formal definition of reference priors. Ann. Stat. 37, 905–938 (2009)

    MathSciNet  Article  MATH  Google Scholar 

  6. 6.

    Bjornstad, J.F.: Predictive likelihood: a review. Stat. Sci. 5, 242–265 (1990)

    MathSciNet  Article  MATH  Google Scholar 

  7. 7.

    Cox, D.R.: Principles of Statistical Inference. Cambridge University Press, Cambridge (2006)

    Google Scholar 

  8. 8.

    Datta, G.S., Sweeting, T.J.: Probability matching priors. In: Dey, D.K., Rao, C.R. (eds.) Handbook of Statistics, 25: Bayesian Thinking: Modeling and Computation, pp. 91–114. Elsevier, Amsterdam (2005)

    Google Scholar 

  9. 9.

    Dempster, A.P., Laird, N.M., Rubin, D.B.: Maximum likelihood from incomplete data via the EM algorithm. J. R. Stat. Soc. B 39, 1–38 (1977)

    MathSciNet  MATH  Google Scholar 

  10. 10.

    Diaz-Francés, E.: Simple estimation intervals for Poisson, exponential and inverse Gaussian means obtained by symmetrizing the likelihood function. Am. Stat. 70, 171–180 (2016)

    MathSciNet  Article  Google Scholar 

  11. 11.

    Fisher, R.A.: On an absolute criterion for fitting frequency curves. Messenger Math. 41, 155–160 (1912)

    Google Scholar 

  12. 12.

    Fisher, R.A.: On the mathematical foundations of theoretical statistics. Philos. Trans. R. Soc. Lond. A 222, 309–368 (1922)

    Article  MATH  Google Scholar 

  13. 13.

    Fisher, R.A.: Theory of statistical estimation. Proc. Camb. Philos. Soc. 22, 700–725 (1925)

    Article  MATH  Google Scholar 

  14. 14.

    Fraser, D.A.S.: Is Bayes just quick and dirty confidence? (with discussion). Stat. Sci. 26, 299–316 (2011)

    Article  MATH  Google Scholar 

  15. 15.

    Gelman, A., Meng, X.L., Stern, H.S.: Posterior predictive assessment of model fitness via realised discrepancies (with discussion). Stat. Sin. 6, 733–807 (1996)

    MATH  Google Scholar 

  16. 16.

    Gelman, A., Carlin, J.B., Stern, H.S., Dunson, D.B., Vehtari, A., Rubin, D.B.: Bayesian date analysis, 3rd edn. CRC Press, Boca Raton (2014)

    Google Scholar 

  17. 17.

    Jeffreys, H.: Theory of Probability (3rd edn. 1961, reissued 1998). Oxford University Press, Oxford (1939)

    Google Scholar 

  18. 18.

    Laplace, P.-S.: Essai philosophique sur les probabilités (1814). English translation A Philosophical Essay on Probabilities. Wiley (1902) and Dover, New York (1950)

  19. 19.

    Little, R.J.A.: Calibrated Bayes: a Bayes/frequentist roadmap. Am. Stat. 60, 213–223 (2006)

    MathSciNet  Article  Google Scholar 

  20. 20.

    Little, R.J.A.: Calibrated Bayes, for statistics in general, and missing data in particular. Stat. Sci. 26, 162–174 (2011)

    MathSciNet  Article  MATH  Google Scholar 

  21. 21.

    Little, R.J.A.: Calibrated Bayes, an inferential paradigm for official statistics in the era of big data. Stat. J. IAOS 31, 555–563 (2015)

    Article  Google Scholar 

  22. 22.

    Müller, U.K., Noretz, A.: Coverage inducing priors in nonstandard inference problems. J. Am. Stat. Assoc. 111, 1233–1241 (2016)

    MathSciNet  Article  Google Scholar 

  23. 23.

    Pearson, K.: Contribution to the mathematical theory of evolution. Philos. Trans. R. Soc. Lond. A 185, 71–110 (1894)

    Article  MATH  Google Scholar 

  24. 24.

    Pearson, K.: On the criterion that a given system of deviations from the probable in the case of a correlated system of variables is such that it can be reasonably supposed to have arisen from random sampling. Philos. Mag. 50, 157–175 (1900)

    Article  MATH  Google Scholar 

  25. 25.

    Pearson, K.: On the systematic fitting of curves to observations and measurements, Parts I. Biometrika I, 265–303 (1902)

    Article  Google Scholar 

  26. 26.

    Pearson, K.: On the systematic fitting of curves to observations and measurements, Parts II. Biometrika II, 1–23 (1902)

    Google Scholar 

  27. 27.

    Raiffa, H., Schlaifer, R.: Applied statistical decision theory. Harward Business School, Boston (1961)

    Google Scholar 

  28. 28.

    Rubin, D.B.: Bayesianly justifiable and relevant frequency calculations for the applied statistician. Ann. Stat. 12, 1151–1172 (1984)

    MathSciNet  Article  MATH  Google Scholar 

  29. 29.

    Tanner, M., Wong, W.: The calculation of posterior distributions by data augmentation. J. Am. Stat. Assoc. 82, 528–550 (1987)

    MathSciNet  Article  MATH  Google Scholar 

Download references

Author information

Affiliations

Authors

Corresponding author

Correspondence to Murray Aitkin.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Aitkin, M., Liu, C. Confidence, credibility and prediction. METRON 76, 251–268 (2018). https://doi.org/10.1007/s40300-018-0139-1

Download citation

Keywords

  • Likelihood
  • Confidence
  • Credible
  • Prediction intervals