Joint European Conference on Machine Learning and Knowledge Discovery in Databases

ECML PKDD 2015: Machine Learning and Knowledge Discovery in Databases pp 199-202 | Cite as

Bayesian Hypothesis Testing in Machine Learning

  • Giorgio Corani
  • Alessio Benavoli
  • Francesca Mangili
  • Marco Zaffalon
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9286)

Abstract

Most hypothesis testing in machine learning is done using the frequentist null-hypothesis significance test, which has severe drawbacks. We review recent Bayesian tests which overcome the drawbacks of the frequentist ones.

Keywords

Bayesian hypothesis testing Null hypothesis significance testing 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Demšar, J.: Statistical comparisons of classifiers over multiple data sets. Journal of Machine Learning Research 7, 1–30 (2006)Google Scholar
  2. 2.
    Garcıa, S., Herrera, F.: An extension on statistical comparisons of classifiers over multiple data sets for all pairwise comparisons. Journal of Machine Learning Research 9, 2677–2694 (2008)Google Scholar
  3. 3.
    Benavoli, A., Corani, G., Mangili, F.: Should we really use post-hoc tests based on mean-ranks? Journal of Machine Learning Research (2015) (in press)Google Scholar
  4. 4.
    Kruschke, J.: Doing Bayesian data analysis: A tutorial introduction with R. Academic Press (2010)Google Scholar
  5. 5.
    Müller, P., Parmigiani, G., Robert, C., Rousseau, J.: Optimal sample size for multiple testing: the case of gene expression microarrays. Journal of the American Statistical Association 99(468), 990–1001 (2004)MathSciNetCrossRefMATHGoogle Scholar
  6. 6.
    Benavoli, A., Mangili, F., Corani, G., Zaffalon, M., Ruggeri, F.: A Bayesian Wilcoxon signed-rank test based on the Dirichlet process. In: Proceedings of the 31st International Conference on Machine Learning (ICML 2014), pp. 1026–1034 (2014)Google Scholar
  7. 7.
    Elkan, C.: The foundations of cost-sensitive learning. In: International Joint Conference on Artificial Intelligence, vol. 17, pp. 973–978 (2001)Google Scholar
  8. 8.
    Corani, G., Benavoli, A.: A Bayesian approach for comparing cross-validated algorithms on multiple data sets. Machine Learning (2015) (in press)Google Scholar
  9. 9.
    Nadeau, C., Bengio, Y.: Inference for the generalization error. Machine Learning 52(3), 239–281 (2003)CrossRefMATHGoogle Scholar
  10. 10.
    Lacoste, A., Laviolette, F., Marchand, M.: Bayesian comparison of machine learning algorithms on single and multiple datasets. In: Proc. of the Fifteenth Int. Conf. on Artificial Intelligence and Statistics (AISTATS 2012), pp. 665–675 (2012)Google Scholar
  11. 11.
    Benavoli, A., Mangili, F., Corani, G., Zaffalon, M.: A Bayesian nonparametric procedure for comparing algorithms. In: Proceedings of the 32nd International Conference on Machine Learning (ICML 2015) (2015) (in press)Google Scholar
  12. 12.
    Ferguson, T.S.: A Bayesian analysis of some nonparametric problems. The Annals of Statistics 1(2), 209–230 (1973)MathSciNetCrossRefGoogle Scholar

Copyright information

© Springer International Publishing Switzerland 2015

Authors and Affiliations

  • Giorgio Corani
    • 1
  • Alessio Benavoli
    • 1
  • Francesca Mangili
    • 1
  • Marco Zaffalon
    • 1
  1. 1.Istituto Dalle Molle di Studi sull’Intelligenza Artificiale (IDSIA), USI - SUPSIMannoSwitzerland

Personalised recommendations