Bayesian Hypothesis Testing in Machine Learning
Most hypothesis testing in machine learning is done using the frequentist null-hypothesis significance test, which has severe drawbacks. We review recent Bayesian tests which overcome the drawbacks of the frequentist ones.
KeywordsBayesian hypothesis testing Null hypothesis significance testing
Unable to display preview. Download preview PDF.
- 1.Demšar, J.: Statistical comparisons of classifiers over multiple data sets. Journal of Machine Learning Research 7, 1–30 (2006)Google Scholar
- 2.Garcıa, S., Herrera, F.: An extension on statistical comparisons of classifiers over multiple data sets for all pairwise comparisons. Journal of Machine Learning Research 9, 2677–2694 (2008)Google Scholar
- 3.Benavoli, A., Corani, G., Mangili, F.: Should we really use post-hoc tests based on mean-ranks? Journal of Machine Learning Research (2015) (in press)Google Scholar
- 4.Kruschke, J.: Doing Bayesian data analysis: A tutorial introduction with R. Academic Press (2010)Google Scholar
- 6.Benavoli, A., Mangili, F., Corani, G., Zaffalon, M., Ruggeri, F.: A Bayesian Wilcoxon signed-rank test based on the Dirichlet process. In: Proceedings of the 31st International Conference on Machine Learning (ICML 2014), pp. 1026–1034 (2014)Google Scholar
- 7.Elkan, C.: The foundations of cost-sensitive learning. In: International Joint Conference on Artificial Intelligence, vol. 17, pp. 973–978 (2001)Google Scholar
- 8.Corani, G., Benavoli, A.: A Bayesian approach for comparing cross-validated algorithms on multiple data sets. Machine Learning (2015) (in press)Google Scholar
- 10.Lacoste, A., Laviolette, F., Marchand, M.: Bayesian comparison of machine learning algorithms on single and multiple datasets. In: Proc. of the Fifteenth Int. Conf. on Artificial Intelligence and Statistics (AISTATS 2012), pp. 665–675 (2012)Google Scholar
- 11.Benavoli, A., Mangili, F., Corani, G., Zaffalon, M.: A Bayesian nonparametric procedure for comparing algorithms. In: Proceedings of the 32nd International Conference on Machine Learning (ICML 2015) (2015) (in press)Google Scholar