Abstract
Evaluating classifier performance with ROC curves is popular in the machine learning community. To date, the only method to assess confidence of ROC curves is to construct ROC bands. In the case of severe class imbalance with few instances of the minority class, ROC bands become unreliable. We propose a generic framework for classifier evaluation to identify a segment of an ROC curve in which misclassifications are balanced. Confidence is measured by Tango’s 95%-confidence interval for the difference in misclassification in both classes. We test our method with severe class imbalance in a two-class problem. Our evaluation favors classifiers with low numbers of misclassifications in both classes. Our results show that the proposed evaluation method is more confident than ROC bands.
Chapter PDF
Similar content being viewed by others
References
Ling, C.X., Huang, J., Zang, H.: Auc: a better measure than accuracy in comparing learning algorithms. In: Canadian Conference on AI, pp. 329–341 (2003)
Provost, F., Fawcett, T.: Analysis and visualization f classifier performance: Comparison under imprecise class and cost distributions. In: The Third International Conference on Knowledge Discovery and Data Mining, pp. 34–48 (1997)
Cohen, W.W., Schapire, R.E., Singer, Y.: Learning to order things. Journal of Artificial Intelligence Research (10), 243–270 (1999)
Swets, J.: Measuring the accuracy of diagnostic systems. Science (240), 1285–1293 (1988)
Drummond, C., Holte, R.C.: Explicitly representing expected cost: An alternative to roc representation. In: The Sixth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 198–207 (2000)
Drummond, C., Holte, R.C.: What roc curves can’t do (and cost curves can). In: ECAI 2004 Workshop on ROC Analysis in AI (2004)
Macskassy, S.A., Provost, F., Rosset, S.: Roc confidence bands: An empirical evaluation. In: Proceedings of the 22nd International Conference on Machine Learning (ICML 2005), pp. 537–544 (2005)
Macskassy, S.A., Provost, F.: Confidence bands for roc curves: Methods and empirical study. In: Proceedings of the 1st Workshop on ROC Analasis in AI (ROCAI-2004) at ECAI-2004 (2004)
Drummond, C., Holte, R.C.: Severe class imbalance: Why better algorithms aren’t the answer. In: Proceedings of the 16th European Conference of Machine Learning, pp. 539–546 (2005)
Motulsky, H.: Intuitive Biostatistics. Oxford University Press, Oxford (1995)
Tango, T.: Equivalence test and confidence interval for the difference in proportions for the paired-sample design. Statistics in Medicine 17, 891–908 (1998)
Newcombe, R.G.: Improved confidence intervals for the difference between binomial proportions based on paired data. Statistics in Medicine 17, 2635–2650 (1998)
Newman, D.J., Hettich, S., Blake, C.L., Merz, C.J.: UCI repository of machine learning databases, University of California, Irvine, Dept. of Information and Computer Sciences (1998), http://www.ics.uci.edu/~mlearn/MLRepository.html
Witten, I.H., Frank, E.: Data Mining: Practical machine learning tools and techniques, 2nd edn. Morgan Kaufmann, San Francisco (2005)
Dietterich, T.G.: Approximate statistical test for comparing supervised classification learning algorithms. Neural Computation 10(7), 1895–1923 (1998)
Newcombe, R.G.: Two-sided confidence intervals for the single proportion: comparison of seven methods. Statistics in Medicine 17, 857–872 (1998)
Everitt, B.S.: The analysis of contingency tables. Chapman-Hall, Boca Raton (1992)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2006 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Elazmeh, W., Japkowicz, N., Matwin, S. (2006). Evaluating Misclassifications in Imbalanced Data. In: Fürnkranz, J., Scheffer, T., Spiliopoulou, M. (eds) Machine Learning: ECML 2006. ECML 2006. Lecture Notes in Computer Science(), vol 4212. Springer, Berlin, Heidelberg. https://doi.org/10.1007/11871842_16
Download citation
DOI: https://doi.org/10.1007/11871842_16
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-45375-8
Online ISBN: 978-3-540-46056-5
eBook Packages: Computer ScienceComputer Science (R0)