On Comparing Classifiers: Pitfalls to Avoid and a Recommended Approach
- Steven L. Salzberg
- … show all 1 hide
Rent the article at a discountRent now
* Final gross prices may vary according to local VAT.Get Access
An important component of many data mining projects is finding a good classification algorithm, a process that requires very careful thought about experimental design. If not done very carefully, comparative studies of classification and other types of algorithms can easily result in statistically invalid conclusions. This is especially true when one is using data mining techniques to analyze very large databases, which inevitably contain some statistically unlikely data. This paper describes several phenomena that can, if ignored, invalidate an experimental comparison. These phenomena and the conclusions that follow apply not only to classification, but to computational experiments in almost any aspect of data mining. The paper also discusses why comparative analysis is more important in evaluating some types of algorithms than for others, and provides some suggestions about how to avoid the pitfalls suffered by many experimental studies.
- Aha, D. Generalizing from case studies: A case study. In Proc. Ninth Intl. Workshop on Machine Learning, pages 1–10, San Mateo, CA, 1992. Morgan Kaufmann.
- Cochran, W. and Cox, G. Experimental Designs. Wiley, 2nd edition, 1957.
- Cohen, P.R. and Jensen, D. Overfitting explained. In Prelim. Papers Sixth Intl. Workshop on Artificial Intelligence and Statistics, pages 115–122, January 1997.
- Denton, F. (1985) Data mining as an industry. Review of Economics and Statistics 67: pp. 124-127
- Dietterich, T. Statistical tests for comparing supervised learning algorithms. Technical report, Oregon State University, Corvallis, OR, 1996.
- Everitt, B. The Analysis of Contingency Tables. Chapman and Hall, London., 1977.
- Fayyad, U.M. and Irani, K.B. Multi-interval discretization of continuous valued attributes for classification learning. In Proc. 13th Intl. Joint Conf. on Artificial Intelligence, pages 1022–1027, Chambery, France, 1993. Morgan Kaufmann.
- Feelders, A. and Verkooijen,W. Which method learns most from the data? In Prelim. Papers Fifth Intl. Workshop on Artificial Intelligence and Statistics, pages 219–225, Fort Lauderdale, Florida, 1995.
- Flexer, A. Statistical evaluation of neural network experiments: Minimum requirements and current practice. In R. Trappl, editor, Cybernetics and Systems '96: Proc. 13th European Meeting on Cybernetics and Systems Res., pages 1005–1008. Austrian Society for Cybernetic Studies, 1996.
- Gascuel, O. and Caraux, G. Statistical significance in inductive learning. In Proc. of the European Conf. on Artificial Intelligence (ECAI), pages 435–439, New York, 1992. Wiley.
- Hildebrand, D. (1986) Statistical Thinking for Behavioral Scientists. Duxbury Press, Boston, MA
- Holte, R. (1993) Very simple classification rules perform well on most commonly used datasets. Machine Learning 11: pp. 63-90
- Jensen, D. Knowledge discovery through induction with randomization testing. In G. Piatetsky-Shapiro, editor, Proc. 1991 Knowledge Discovery in DatabasesWorkshop, pages 148–159, Menlo Park, CA, 1991. AAAI Press.
- Jensen, D. (1995) Labeling space: A tool for thinking about significance testing in knowledge discovery. Office of Technology Assessment, U.S. Congress
- Kibler, D. and Langley, P. Machine learning as an experimental science. In Proc. of 1988 Euro. Working Session on Learning, pages 81–92, 1988.
- Kohavi, R. and Sommerfield, D. Oblivious decision trees, graphs, and top-down pruning. In Proc. 14th Intl. Joint Conf. on Artificial Intelligence, pages 1071–1077, Montreal, 1995. Morgan Kaufmann.
- Murphy, P.M. UCI repository of machine learning databases-a machine-readable data repository. Maintained at the Department of Information and Computer Science, University of California, Irvine. Anonymous FTP from ics.uci.edu in the directory pub/machine-learning-databases, 1995.
- Prechelt, L. A quantitative study of experimental evaluations of neural network algorithms: Current research practice. Neural Networks, 9, 1996.
- Qian, N., Sejnowski, T. (1988) Predicting the secondary structure of globular proteins using neural network models. Journal of Molecular Biology 202: pp. 65-884
- Raftery, A. Bayesian model selection in social research (with discussion by Andrew Gelman, Donald B. Rubin, and Robert M. Hauser). In: Marsden, P. eds. (1995) Sociological Methodology 1995. Blackwells, Oxford, UK, pp. 111-196
- Sejnowski, T., Rosenberg, C. (1987) Parallel networks that learn to pronounce English text. Complex Systems 1: pp. 145-168
- Shavlik, J., Mooney, R., Towell, G. (1991) Symbolic and neural learning algorithms: An experimental comparison. Machine Learning 6: pp. 111-143
- Wettschereck, D., Dietterich, T. (1995) An experimental comparison of the nearest-neighbor and nearest-hyperrectangle algorithms. Machine Learning 19: pp. 5-28
- Wolpert, D. (1992) On the connection between in-sample testing and generalization error. Complex Systems 6: pp. 47-94
- On Comparing Classifiers: Pitfalls to Avoid and a Recommended Approach
Data Mining and Knowledge Discovery
Volume 1, Issue 3 , pp 317-328
- Cover Date
- Print ISSN
- Online ISSN
- Kluwer Academic Publishers
- Additional Links
- comparative studies
- statistical methods
- Industry Sectors
- Author Affiliations
- 1. Department of Computer Science, Johns Hopkins University, Baltimore, MD, 21218, USA