Quality & Quantity

, Volume 41, Issue 1, pp 73–91 | Cite as

Comparison of Hypothesis Testing and Bayesian Model Selection

Article

Abstract

The main goal of both Bayesian model selection and classical hypotheses testing is to make inferences with respect to the state of affairs in a population of interest. The main differences between both approaches are the explicit use of prior information by Bayesians, and the explicit use of null distributions by the classicists. Formalization of prior information in prior distributions is often difficult. In this paper two practical approaches (encompassing priors and training data) to specify prior distributions will be presented. The computation of null distributions is relatively easy. However, as will be illustrated, a straightforward interpretation of the resulting p-values is not always easy. Bayesian model selection can be used to compute posterior probabilities for each of a number of competing models. This provides an alternative for the currently prevalent testing of hypotheses using p-values. Both approaches will be compared and illustrated using case studies. Each case study fits in the framework of the normal linear model, that is, analysis of variance and multiple regression.

Keywords

Bayesian model selection encompassing prior posterior model probability p-value training data 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Bayarri, M.J., Berger, J.O. 2000P-values for composite null modelsJournal of the American Statistical Association9511271142CrossRefGoogle Scholar
  2. Berger, J.O., Perricchi, L. 1996The intrinsic bayes factor for model selection predictionJournal of the American Statistical Association9109122CrossRefGoogle Scholar
  3. Berger, J.O., Sellke, T. 1987Testing a point null hypothesis: the irreconcilability of p-values and evidenceJournal of the American Statistical Association82112122CrossRefGoogle Scholar
  4. Camstra, A., Boomsma, A. 1992Cross-validation in regression and covariance structure analysisSociological Methods and Research218995Google Scholar
  5. Carlin, B.P., Chib, S. 1995Bayesian model choice via Markov Chain Monte Carlo methodsJournal of the Royal Statistical Society, B57473484Google Scholar
  6. Congdon, P. 2001Bayesian Statistical ModellingJohn Wiley and SonsNew YorkGoogle Scholar
  7. Cohen, J. 1994The earth is round (p  <  0.05)American Psychologist129971003CrossRefGoogle Scholar
  8. Dayton, C.M. 2003Information criteria for pairwise comparisonsPsychological Methods86171CrossRefGoogle Scholar
  9. Frick, R.W. 1996The appropriate use of null hypothesis testingPsychological Methods1379390CrossRefGoogle Scholar
  10. Gelman, A., Carlin, J.B., Stern, H.S., Rubin, D.B. 1995Bayesian Data AnalysisChapmann and HallLondonGoogle Scholar
  11. Howson, C. 2002

    Bayesianism in statistics

    Swinburne, R. eds. Bayes TheoremOxford University PressOxford3969
    Google Scholar
  12. Kass, R.E., Raftery, A.E. 1995Bayes factorsJournal of the American Statistical Association90773795CrossRefGoogle Scholar
  13. Klugkist, I., Kato, B., Hoijtink, H. 2005Bayesian model selection using encompassing priorsStatistica Neerlandica595769CrossRefGoogle Scholar
  14. Newton, M.A., Raftery, A.E. 1994Approximate Bayesian inference by the weighted likelihood bootstrapJournal of the Royal Statistical Society B56348Google Scholar
  15. Ramsey, P.H. 2002Comparison of closed testing procedures for pairwise testing of meansPsychological Methods7504523CrossRefGoogle Scholar
  16. Robertson, T., Wright, F.T., Dykstra, R.L. 1988Order Restricted Statistical InferenceJohn Wiley and SonsNew YorkGoogle Scholar
  17. Sellke, T., Bayarri, M.J., Berger, J.O. 2001Calibration of p values for testing precise null hypothesesThe American Statistician556271CrossRefGoogle Scholar
  18. Sober, E. 2002

    Bayesianism–its scope and limits

    Swinburne, R. eds. Bayes TheoremOxford University PressOxford2138
    Google Scholar
  19. Stevens, J. 1992Applied Multivariate Statistics for the Social SciencesLawrence ErlbaumLondonGoogle Scholar
  20. Tabachnick, B.G., Fidell, L.S. 2001Using Multivariate StatisticsAllyn and BaconLondonGoogle Scholar
  21. Toothaker, L.E. 1993Multiple Comparison ProceduresSAGELondonGoogle Scholar
  22. Wainer, H. 1999One cheer for null hypothesis significance testingPsychological Methods4212213CrossRefGoogle Scholar

Copyright information

© Springer 2007

Authors and Affiliations

  1. 1.Department of Methodology and StatisticsUniversity of UtrechtUtrechtThe Netherlands

Personalised recommendations