The three examples Gronau and Wagenmakers (Computational Brain and Behavior, 2018; hereafter denoted G&W) use to demonstrate the limitations of Bayesian forms of leave-one-out cross validation (let us term this LOOCV) for model selection have several important properties: The true model instance is among the model classes being compared; the smaller, simpler model is a point hypothesis that in fact generates the data; the larger class contains the smaller. As G&W admit, there is a good deal of prior history pointing to the limitations of cross validation and LOOCV when used in such situations (e.g., Bernardo and Smith 1994). We do not wish to rehash this literature trail, but rather give a conceptual overview of methodology that allows discussion of the ways that various methods of model selection align with scientific practice and scientific inference, and give our recommendation for the simplest approach that matches statistical inference to the needs of science. The methods include minimum description length (MDL) as reported by Grünwald (2007); Bayesian model selection (BMS) as reported by Kass and Raftery (Journal of the American Statistical Association, 90, 773–795, 1995); and LOOCV as reported by Browne (Journal of Mathematical Psychology, 44, 108–132, 2000) and Gelman et al. (Statistics and Computing, 24, 997–1016, 2014). In this commentary, we shall restrict the focus to forms of BMS and LOOCV. In addition, in these days of “Big Data,” one wants inference procedures that will give reasonable answers as the amount of data grows large, one focus of the article by G&W. We discuss how the various inference procedures fare when the data grow large.
This is a preview of subscription content, log in to check access.
Buy single article
Instant access to the full article PDF.
Price includes VAT for USA
Subscribe to journal
Immediate online access to all issues from 2019. Subscription will auto renew annually.
This is the net price. Taxes to be calculated in checkout.
Baribault, B., Donkin, C., Little, D. R., Trueblood, J., Oravecz, Z., van Ravenzwaaij, D., White, C. N., De Boeck, P., & Vandekerckhove, J. (2018). Metastudies for robust tests of theory. Proceedings of the National Academy of Sciences, 115, 2607–2612.
Bernardo, J. M., & Smith, A. F. (1994). Bayesian theory. 1994. John Willey and Sons. Valencia (España).
Browne, M. (2000). Cross-validation methods. Journal of Mathematical Psychology, 44, 108–132.
Chandramouli, S. H., & Shiffrin, R. M. (2016). Extending Bayesian induction. Journal of Mathematical Psychology., 72, 38–42.
Gelman, A., & Carlin, J. (2017). Some natural solutions to the p-value communication problem—and why they won’t work. Journal of the American Statistical Association, 112(519), 899–901.
Gelman, A., Carlin, J. B., Stern, H. S., & Rubin, D. B. (2004). Bayesian data analysis, 2nd edn. Chapman and Hall.
Gelman, A., Hwang, J., & Vehtari, A. (2014). Understanding predictive information criteria for Bayesian models. Statistics and Computing, 24, 997–1016.
Gronau, F. Q., & Wagenmakers, E.J. (2018). Limitations of Bayesian leave-one-out validation for model selection. Computational Brain and Behavior.
Grünwald, P. (2007). The minimum description length principle. Cambridge: MIT Press.
Kass, R. E., & Raftery, A. E. (1995). Bayes factors. Journal of the American Statistical Association, 90, 773–795.
Kruschke, J. (2014). Doing Bayesian data analysis. In A tutorial with R, JAGS, and Stan, 2nd edn. Academic Press.
Shiffrin, R. M., & Chandramouli, S. H. (2016). Model selection, data distributions, and reproducibility. In H. Atmanspacher & S. Maasen (Eds.), Reproducibility: principles, problems, and practices (pp. 115–140). New York: John Wiley.
Shiffrin, R. M., Chandramouli, S. H., & Grunwald, P. G. (2016). Bayes factors, relations to minimum description length, and overlapping model classes. Journal of Mathematical Psychology., 72, 56–77.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Chandramouli, S.H., Shiffrin, R.M. Commentary on Gronau and Wagenmakers. Comput Brain Behav 2, 12–21 (2019). https://doi.org/10.1007/s42113-018-0017-1
- Model Selection
- Overlapping Model Classes
- Bayes Factor