Commentary on Gronau and Wagenmakers

Abstract

The three examples Gronau and Wagenmakers (Computational Brain and Behavior, 2018; hereafter denoted G&W) use to demonstrate the limitations of Bayesian forms of leave-one-out cross validation (let us term this LOOCV) for model selection have several important properties: The true model instance is among the model classes being compared; the smaller, simpler model is a point hypothesis that in fact generates the data; the larger class contains the smaller. As G&W admit, there is a good deal of prior history pointing to the limitations of cross validation and LOOCV when used in such situations (e.g., Bernardo and Smith 1994). We do not wish to rehash this literature trail, but rather give a conceptual overview of methodology that allows discussion of the ways that various methods of model selection align with scientific practice and scientific inference, and give our recommendation for the simplest approach that matches statistical inference to the needs of science. The methods include minimum description length (MDL) as reported by Grünwald (2007); Bayesian model selection (BMS) as reported by Kass and Raftery (Journal of the American Statistical Association, 90, 773–795, 1995); and LOOCV as reported by Browne (Journal of Mathematical Psychology, 44, 108–132, 2000) and Gelman et al. (Statistics and Computing, 24, 997–1016, 2014). In this commentary, we shall restrict the focus to forms of BMS and LOOCV. In addition, in these days of “Big Data,” one wants inference procedures that will give reasonable answers as the amount of data grows large, one focus of the article by G&W. We discuss how the various inference procedures fare when the data grow large.

This is a preview of subscription content, log in to check access.

Fig. 1
Fig. 2
Fig. 3
Fig. 4

References

  1. Baribault, B., Donkin, C., Little, D. R., Trueblood, J., Oravecz, Z., van Ravenzwaaij, D., White, C. N., De Boeck, P., & Vandekerckhove, J. (2018). Metastudies for robust tests of theory. Proceedings of the National Academy of Sciences, 115, 2607–2612.

    Article  Google Scholar 

  2. Bernardo, J. M., & Smith, A. F. (1994). Bayesian theory. 1994. John Willey and Sons. Valencia (España).

  3. Browne, M. (2000). Cross-validation methods. Journal of Mathematical Psychology, 44, 108–132.

    Article  Google Scholar 

  4. Chandramouli, S. H., & Shiffrin, R. M. (2016). Extending Bayesian induction. Journal of Mathematical Psychology., 72, 38–42.

    Article  Google Scholar 

  5. Gelman, A., & Carlin, J. (2017). Some natural solutions to the p-value communication problem—and why they won’t work. Journal of the American Statistical Association, 112(519), 899–901.

    Article  Google Scholar 

  6. Gelman, A., Carlin, J. B., Stern, H. S., & Rubin, D. B. (2004). Bayesian data analysis, 2nd edn. Chapman and Hall.

  7. Gelman, A., Hwang, J., & Vehtari, A. (2014). Understanding predictive information criteria for Bayesian models. Statistics and Computing, 24, 997–1016.

    Article  Google Scholar 

  8. Gronau, F. Q., & Wagenmakers, E.J. (2018). Limitations of Bayesian leave-one-out validation for model selection. Computational Brain and Behavior.

  9. Grünwald, P. (2007). The minimum description length principle. Cambridge: MIT Press.

    Google Scholar 

  10. Kass, R. E., & Raftery, A. E. (1995). Bayes factors. Journal of the American Statistical Association, 90, 773–795.

    Article  Google Scholar 

  11. Kruschke, J. (2014). Doing Bayesian data analysis. In A tutorial with R, JAGS, and Stan, 2nd edn. Academic Press.

  12. Shiffrin, R. M., & Chandramouli, S. H. (2016). Model selection, data distributions, and reproducibility. In H. Atmanspacher & S. Maasen (Eds.), Reproducibility: principles, problems, and practices (pp. 115–140). New York: John Wiley.

    Google Scholar 

  13. Shiffrin, R. M., Chandramouli, S. H., & Grunwald, P. G. (2016). Bayes factors, relations to minimum description length, and overlapping model classes. Journal of Mathematical Psychology., 72, 56–77.

    Article  Google Scholar 

Download references

Author information

Affiliations

Authors

Corresponding author

Correspondence to Suyog H. Chandramouli.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Chandramouli, S.H., Shiffrin, R.M. Commentary on Gronau and Wagenmakers. Comput Brain Behav 2, 12–21 (2019). https://doi.org/10.1007/s42113-018-0017-1

Download citation

Keywords

  • Model Selection
  • Overlapping Model Classes
  • Cross-Validation
  • Bayes Factor