Download PDF

How to quantify support for and against the null hypothesis: A flexible WinBUGS implementation of a default Bayesian t test

Psychonomic Bulletin & Review volume 16, pages752–760(2009)Cite this article

Abstract

We propose a sampling-based Bayesian t test that allows researchers to quantify the statistical evidence in favor of the null hypothesis. This Savage—Dickey (SD) t test is inspired by the Jeffreys—Zellner—Siow (JZS) t test recently proposed by Rouder, Speckman, Sun, Morey, and Iverson (2009). The SD test retains the key concepts of the JZS test but is applicable to a wider range of statistical problems. The SD test allows researchers to test order restrictions and applies to two-sample situations in which the different groups do not share the same variance.

References

  1. Batchelder, W. H., & Riefer, D. M. (1999). Theoretical and empirical review of multinomial process tree modeling. Psychonomic Bulletin & Review, 6, 57–86.

    Article  Google Scholar 

  2. Berger, J. O., & Jefferys, W. H. (1992). The application of robust Bayesian analysis to hypothesis testing and Occam’s razor. Statistical Methods & Applications, 1, 17–32.

    Google Scholar 

  3. Berger, J. O., & Pericchi, L. R. (1996). The intrinsic Bayes factor for model selection and prediction. Journal of the American Statistical Association, 91, 109–122.

    Article  Google Scholar 

  4. Box, G. E. P., & Tiao, G. C. (1973). Bayesian inference in statistical analysis. Reading, MA: Addison-Wesley.

    Google Scholar 

  5. Busemeyer, J. R., & Stout, J. C. (2002). A contribution of cognitive decision models to clinical assessment: Decomposing performance on the Bechara gambling task. Psychological Assessment, 14, 253–262.

    Article  PubMed  Google Scholar 

  6. Carlin, B. P., & Louis, T. A. (2000). Bayes and empirical Bayes methods for data analysis (2nd ed.). London: Chapman & Hall.

    Google Scholar 

  7. Dickey, J. M., & Lientz, B. P. (1970). The weighted likelihood ratio, sharp hypotheses about chances, the order of a Markov chain. Annals of Mathematical Statistics, 41, 214–226.

    Article  Google Scholar 

  8. Edwards, W., Lindman, H., & Savage, L. J. (1963). Bayesian statistical inference for psychological research. Psychological Review, 70, 193–242.

    Article  Google Scholar 

  9. Gamerman, D., & Lopes, H. F. (2006). Markov chain Monte Carlo: Stochastic simulation for Bayesian inference. Boca Raton, FL: Chapman & Hall/CRC.

    Google Scholar 

  10. Gelman, A., & Hill, J. (2007). Data analysis using regression and multilevel/hierarchical models. Cambridge: Cambridge University Press.

    Google Scholar 

  11. Gilks, W. R., Thomas, A., & Spiegelhalter, D. J. (1994). A language and program for complex Bayesian modelling. The Statistician, 43, 169–177.

    Article  Google Scholar 

  12. Hedges, L. (1981). Distribution theory for Glass’s estimator of effect size and related estimators. Journal of Educational & Behavioral Statistics, 6, 107.

    Article  Google Scholar 

  13. Hoijtink, H., Klugkist, I., & Boelen, P. (2008). Bayesian evaluation of informative hypotheses that are of practical value for social scientists. New York: Springer.

    Google Scholar 

  14. Jeffreys, H. (1961). Theory of probability. Oxford: Oxford University Press.

    Google Scholar 

  15. Kass, R. E., & Raftery, A. E. (1995). Bayes factors. Journal of the American Statistical Association, 90, 377–395.

    Google Scholar 

  16. Killeen, P. R. (2007). Replication statistics as a replacement for significance testing: Best practices in scientific decision-making. In J. W. Osborne (Ed.), Best practices in quantitative methods (pp. 103–124). Thousand Oaks, CA: Sage.

    Google Scholar 

  17. Kim, S., & Cohen, A. (1998). On the Behrens-Fisher problem: A review. Journal of Educational & Behavioral Statistics, 23, 356–377.

    Google Scholar 

  18. Klugkist, I., Laudy, O., & Hoijtink, H. (2005). Inequality constrained analysis of variance: A Bayesian approach. Psychological Methods, 10, 477–493.

    Article  PubMed  Google Scholar 

  19. Kruschke, J. K. (1992). ALCOVE: An exemplar-based connectionist model of category learning. Psychological Review, 99, 22–44.

    Article  PubMed  Google Scholar 

  20. Lauritzen, S. L. (1996). Graphical models. Oxford: Oxford University Press, Clarendon Press.

    Google Scholar 

  21. Lee, M. D. (2008). Three case studies in the Bayesian analysis of cognitive models. Psychonomic Bulletin & Review, 15, 1–15.

    Article  Google Scholar 

  22. Liang, F., Paulo, R., Molina, G., Clyde, M., & Berger, J. (2008). Mixtures of g priors for Bayesian variable selection. Journal of the American Statistical Association, 103, 410.

    Article  Google Scholar 

  23. Lunn, D. J., Thomas, A., Best, N., & Spiegelhalter, D. (2000). WinBUGS—a Bayesian modelling framework: Concepts, structure, and extensibility. Statistics & Computing, 10, 325–337.

    Article  Google Scholar 

  24. Moreno, E., Bertolino, F., & Racugno, W. (1999). Default Bayesian analysis of the Behrens-Fisher problem. Journal of Statistical Planning & Inference, 81, 323–333.

    Article  Google Scholar 

  25. Myung, I. J., & Pitt, M. A. (1997). Applying Occam’s razor in modeling cognition: A Bayesian approach. Psychonomic Bulletin & Review, 4, 79–95.

    Article  Google Scholar 

  26. Nosofsky, R. (1986). Attention, similarity, and the identification-categorization relationship. Journal of Experimental Psychology: General, 115, 39–57.

    Article  Google Scholar 

  27. O’Hagan, A., & Forster, J. (2004). Kendall’s advanced theory of statistics: Vol. 2B. Bayesian inference (2nd ed.). London: Arnold.

    Google Scholar 

  28. Raftery, A. E. (1995). Bayesian model selection in social research. In P. V. Marsden (Ed.), Sociological methodology (pp. 111–196). Cambridge: Blackwells.

    Google Scholar 

  29. Richard, F. D., Bond, C. F. J., & Stokes-Zoota, J. J. (2003). One hundred years of social psychology quantitatively described. Review of General Psychology, 7, 331–363.

    Article  Google Scholar 

  30. Rouder, J. N., & Lu, J. (2005). An introduction to Bayesian hierarchical models with an application in the theory of signal detection. Psychonomic Bulletin & Review, 12, 573–604.

    Article  Google Scholar 

  31. Rouder, J. N., Lu, J., Morey, R. D., Sun, D., & Speckman, P. L. (2008). A hierarchical process dissociation model. Journal of Experimental Psychology: General, 137, 370–389.

    Article  Google Scholar 

  32. Rouder, J. N., Lu, J., Sun, D., Speckman, P., Morey, R., & Naveh-Benjamin, M. (2007). Signal detection models with random participant and item effects. Psychometrika, 72, 621–642.

    Article  Google Scholar 

  33. Rouder, J. N., Speckman, P. L., Sun, D., Morey, R. D., & Iverson, G. (2009). Bayesian t tests for accepting and rejecting the null hypothesis. Psychonomic Bulletin & Review, 16, 225–237.

    Article  Google Scholar 

  34. Shiffrin, R. M., Lee, M. D., Kim, W., & Wagenmakers, E.-J. (2008). A survey of model evaluation approaches with a tutorial on hierarchical Bayesian methods. Cognitive Science, 32, 1248–1284.

    Article  Google Scholar 

  35. Spiegelhalter, D. J. (1998). Bayesian graphical modelling: A case-study in monitoring health outcomes. Applied Statistics, 47, 115–133.

    Google Scholar 

  36. Stone, C. J., Hansen, M. H., Kooperberg, C., & Truong, Y. K. (1997). Polynomial splines and their tensor products in extended linear modeling (with discussion). Annals of Statistics, 25, 1371–1470.

    Article  Google Scholar 

  37. Vandekerckhove, J., Tuerlinckx, F., & Lee, M. D. (2008). A Bayesian approach to diffusion process models of decision-making. In V. Sloutsky, B. Love, & K. McRae (Eds.), Proceedings of the 30th Annual Conference of the Cognitive Science Society (pp. 1429–1434). Austin, TX: Cognitive Science Society.

    Google Scholar 

  38. Verdinelli, I., & Wasserman, L. (1995). Computing Bayes factors using a generalization of the Savage-Dickey density ratio. Journal of the American Statistical Association, 90, 614–618.

    Article  Google Scholar 

  39. Wagenmakers, E.-J. (2007). A practical solution to the pervasive problems of p values. Psychonomic Bulletin & Review, 14, 779–804.

    Article  Google Scholar 

  40. Wagenmakers, E.-J. (2009). Methodological and empirical developments for the Ratcliff diffusion model of response times and accuracy. European Journal of Cognitive Psychology, 21, 641–671.

    Article  Google Scholar 

  41. Wagenmakers, E.-J., Lee, M. D., Lodewyckx, T., & Iverson, G. (2008). Bayesian versus frequentist inference. In H. Hoijtink, I. Klugkist, & P. A. Boelen (Eds.), Bayesian evaluation of informative hypotheses (pp. 181–207). New York: Springer.

    Google Scholar 

  42. Wasserman, L. (2000). Bayesian model selection and model averaging. Journal of Mathematical Psychology, 44, 92–107.

    Article  PubMed  Google Scholar 

  43. Weaver, R. (2008). Parameters, predictions, and evidence in computational modeling: A statistical view informed by ACT-R. Cognitive Science, 32, 1349–1375.

    Article  Google Scholar 

  44. Wetzels, R., Vandekerckhove, J., Tuerlinckx, F., & Wagenmakers, E.-J. (in press). Bayesian parameter estimation in the expectancy valence model of the Iowa gambling task. Journal of Mathematical Psychology.

  45. Wilkinson, L., & the Task Force on Statistical Inference (1999). Statistical methods in psychology journals: Guidelines and explanations. American Psychologist, 54, 594–604.

    Article  Google Scholar 

  46. Zellner, A. (1986). On assessing prior distributions and Bayesian regression analysis with g-prior distributions. In P. K. Goel & A. Zellner (Eds.), Bayesian inference and decision techniques: Essays in honor of Bruno de Finetti (pp. 233–243). Amsterdam: North-Holland.

    Google Scholar 

  47. Zellner, A., & Siow, A. (1980). Posterior odds ratios for selected regression hypotheses. In J. M. Bernardo, M. H. DeGroot, D. V. Lindley, & A. F. M. Smith (Eds.), Bayesian statistics (pp. 585–603). Valencia: Valencia University Press.

    Google Scholar 

Download references

Author information

Affiliations

  1. Department of Psychology, University of Amsterdam, Roetersstraat 15, 1018 WB, Amsterdam, The Netherlands

    Ruud Wetzels, Jeroen G. W. Raaijmakers, Emöke Jakab & Eric-Jan Wagenmakers

Authors
  1. Ruud Wetzels
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jeroen G. W. Raaijmakers
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Emöke Jakab
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Eric-Jan Wagenmakers
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ruud Wetzels.

Additional information

This research was supported by a Vidi grant from the Dutch Organization for Scientific Research (NWO). The SD computer code and the data of Dr. Smith’s hypothetical experiment can be found on the first author’s Web site, www.ruudwetzels.com.

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Wetzels, R., Raaijmakers, J.G.W., Jakab, E. et al. How to quantify support for and against the null hypothesis: A flexible WinBUGS implementation of a default Bayesian t test. Psychonomic Bulletin & Review 16, 752–760 (2009). https://doi.org/10.3758/PBR.16.4.752

Download citation

Keywords

  • Posterior Distribution
  • Psychonomic Bulletin
  • Order Restriction
  • Iowa Gambling Task
  • Posterior Odds