Skip to main content
SpringerLink
Log in

Evaluating the Replicability of Significance Tests for Comparing Learning Algorithms

  • Conference paper
Advances in Knowledge Discovery and Data Mining (PAKDD 2004)

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 3056))

Included in the following conference series:

Abstract

Empirical research in learning algorithms for classification tasks generally requires the use of significance tests. The quality of a test is typically judged on Type I error (how often the test indicates a difference when it should not) and Type II error (how often it indicates no difference when it should). In this paper we argue that the replicability of a test is also of importance. We say that a test has low replicability if its outcome strongly depends on the particular random partitioning of the data that is used to perform it. We present empirical measures of replicability and use them to compare the performance of several popular tests in a realistic setting involving standard learning algorithms and benchmark datasets. Based on our results we give recommendations on which test to use.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Witten, I.H., Frank, E.: Data Mining: Practical Machine Learning Tools and Techniques with Java Implementations. Morgan Kaufmann, San Francisco (2000)

    Google Scholar 

  2. Bouckaert, R.R.: Choosing between two learning algorithms based on calibrated tests. In: Proc. 20th Int. Conf. on Machine Learning, Morgan Kaufmann, San Francisco (2003)

    Google Scholar 

  3. Bouckaert, R.R.: Choosing learning algorithms using sign tests with high replicability. In: Proc. 16th Australian Joint Conference on Artificial Intelligence, Springer, Heidelberg (2003)

    Google Scholar 

  4. Blake, C.L., Merz, C.J.: UCI Repository of machine learning databases. Irvine, CA (1998), http://www.ics.uci.edu/~mlearn/MLRepository.html

  5. Dietterich, T.G.: Approximate Statistical Tests for Comparing Supervised Classification Learning Algorithms. Neural Computation 10(7), 1895–1924 (1998)

    Article  Google Scholar 

  6. Nadeau, C.: andY. Bengio. Inference for the generalization error. Machine Learning 52, 239–281 (2003)

    Article  MATH  Google Scholar 

  7. Quinlan, R.: C4.5: Programs for Machine Learning. Morgan Kaufmann, San Francisco (1993)

    Google Scholar 

  8. Friedman, N., Geiger, D., Goldszmidt, M.: Bayesian Network Classifiers. Machine Learning 29, 131–163 (1997)

    Article  MATH  Google Scholar 

  9. Pearl, J.: Probabilistic Reasoning in Intelligent Systems. Morgan Kaufmann, San Francisco (1988)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Xtal Mountain Information Technology, 215 Three Oaks Drive, Dairy Flat, Auckland, New Zealand

    Remco R. Bouckaert

  2. Computer Science Department, University of Waikato, Private Bag 3105, Hamilton, New Zealand

    Remco R. Bouckaert & Eibe Frank

Authors
  1. Remco R. Bouckaert
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Eibe Frank
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. School of Engineering and Information Technology, Deakin University, VIC 3125, Australia

    Honghua Dai

  2. University of Illinois at Urbana-Champaign, 61801, Urbana, IL, USA

    Ramakrishnan Srikant

  3. Faculty of Engineering and Information Technology, Centre for Quantum Computation and Intelligent Systems, and Australian ACS National Committee for Artificial Intelligence, University of Technology, Sydney, Australia

    Chengqi Zhang

Rights and permissions

Reprints and permissions

Copyright information

© 2004 Springer-Verlag Berlin Heidelberg

About this paper

Cite this paper

Bouckaert, R.R., Frank, E. (2004). Evaluating the Replicability of Significance Tests for Comparing Learning Algorithms. In: Dai, H., Srikant, R., Zhang, C. (eds) Advances in Knowledge Discovery and Data Mining. PAKDD 2004. Lecture Notes in Computer Science(), vol 3056. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-24775-3_3

Download citation

  • DOI: https://doi.org/10.1007/978-3-540-24775-3_3

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-540-22064-0

  • Online ISBN: 978-3-540-24775-3

  • eBook Packages: Springer Book Archive

Publish with us

Policies and ethics

Search

Navigation