Evaluating the Replicability of Significance Tests for Comparing Learning Algorithms

  • Remco R. Bouckaert
  • Eibe Frank
Part of the Lecture Notes in Computer Science book series (LNCS, volume 3056)


Empirical research in learning algorithms for classification tasks generally requires the use of significance tests. The quality of a test is typically judged on Type I error (how often the test indicates a difference when it should not) and Type II error (how often it indicates no difference when it should). In this paper we argue that the replicability of a test is also of importance. We say that a test has low replicability if its outcome strongly depends on the particular random partitioning of the data that is used to perform it. We present empirical measures of replicability and use them to compare the performance of several popular tests in a realistic setting involving standard learning algorithms and benchmark datasets. Based on our results we give recommendations on which test to use.


Cross Validation High Type Random Subsampling Random Number Seed Popular Test 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Witten, I.H., Frank, E.: Data Mining: Practical Machine Learning Tools and Techniques with Java Implementations. Morgan Kaufmann, San Francisco (2000)Google Scholar
  2. 2.
    Bouckaert, R.R.: Choosing between two learning algorithms based on calibrated tests. In: Proc. 20th Int. Conf. on Machine Learning, Morgan Kaufmann, San Francisco (2003)Google Scholar
  3. 3.
    Bouckaert, R.R.: Choosing learning algorithms using sign tests with high replicability. In: Proc. 16th Australian Joint Conference on Artificial Intelligence, Springer, Heidelberg (2003)Google Scholar
  4. 4.
    Blake, C.L., Merz, C.J.: UCI Repository of machine learning databases. Irvine, CA (1998),
  5. 5.
    Dietterich, T.G.: Approximate Statistical Tests for Comparing Supervised Classification Learning Algorithms. Neural Computation 10(7), 1895–1924 (1998)CrossRefGoogle Scholar
  6. 6.
    Nadeau, C.: andY. Bengio. Inference for the generalization error. Machine Learning 52, 239–281 (2003)zbMATHCrossRefGoogle Scholar
  7. 7.
    Quinlan, R.: C4.5: Programs for Machine Learning. Morgan Kaufmann, San Francisco (1993)Google Scholar
  8. 8.
    Friedman, N., Geiger, D., Goldszmidt, M.: Bayesian Network Classifiers. Machine Learning 29, 131–163 (1997)zbMATHCrossRefGoogle Scholar
  9. 9.
    Pearl, J.: Probabilistic Reasoning in Intelligent Systems. Morgan Kaufmann, San Francisco (1988)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2004

Authors and Affiliations

  • Remco R. Bouckaert
    • 1
    • 2
  • Eibe Frank
    • 2
  1. 1.Xtal Mountain Information TechnologyAucklandNew Zealand
  2. 2.Computer Science DepartmentUniversity of WaikatoHamiltonNew Zealand

Personalised recommendations