Smooth Receiver Operating Characteristics (smROC) Curves

  • William Klement
  • Peter Flach
  • Nathalie Japkowicz
  • Stan Matwin
Part of the Lecture Notes in Computer Science book series (LNCS, volume 6912)

Abstract

Supervised learning algorithms perform common tasks including classification, ranking, scoring, and probability estimation. We investigate how scoring information, often produced by these models, is utilized by an evaluation measure. The ROC curve represents a visualization of the ranking performance of classifiers. However, they ignore the scores which can be quite informative. While this ignored information is less precise than that given by probabilities, it is much more detailed than that conveyed by ranking. This paper presents a novel method to weight the ROC curve by these scores. We call it the Smooth ROC (smROC) curve, and we demonstrate how it can be used to visualize the performance of learning models. We report experimental results to show that the smROC is appropriate for measuring performance similarities and differences between learning models, and is more sensitive to performance characteristics than the standard ROC curve.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Asuncion, A., Newman, D.J.: UCI Machine Learning Repository (2007), http://www.ics.uci.edu/~mlearn/MLRepository.html
  2. 2.
    Bennett, P.N.: Using Asymmetric Distributions to Improve Text Classifier Probability Estimates. In: Proceedings of ACM SIGIR 2003, pp. 111–118 (2003)Google Scholar
  3. 3.
    Brier, G.: Verification of Forecasts Expressed in Terms of Probabilities. Monthly Weather Review 78, 1–3 (1950)CrossRefGoogle Scholar
  4. 4.
    DeGroot, M., Fienberg, S.: The Comparison and Evalution of Forecasters. The statistician 32, 12–22 (1983)CrossRefGoogle Scholar
  5. 5.
    Fawcett, T., Niculescu-Mizil, A.: PAV and the ROC Convex Hull. Machine Learning 68(1), 97–106 (2007)CrossRefGoogle Scholar
  6. 6.
    Ferri, C., Flach, P., Hernandez-Orallo, J.: Improving the AUC of Probabilistic Estimation Trees. In: Lavrač, N., Gamberger, D., Todorovski, L., Blockeel, H. (eds.) ECML 2003. LNCS (LNAI), vol. 2837, pp. 121–132. Springer, Heidelberg (2003)CrossRefGoogle Scholar
  7. 7.
    Fawcett, T.: ROC Graphs: Notes and Practical Considerations for Data Mining Researchers. Technical Report HPL-2003-4, HP Labs (2003)Google Scholar
  8. 8.
    Forman, G.: Counting Positives Accurately Despite Inaccurate Classification. In: Gama, J., Camacho, R., Brazdil, P.B., Jorge, A.M., Torgo, L. (eds.) ECML 2005. LNCS (LNAI), vol. 3720, pp. 564–575. Springer, Heidelberg (2005)CrossRefGoogle Scholar
  9. 9.
    Greiner, R., Su, X., Shen, B., Zhou, W.: Structural Extension to Logistic Regression: Discriminative Parameter Learning of Belief Net Classifiers. Machine Learning 59(3), 213–235 (2005)CrossRefMATHGoogle Scholar
  10. 10.
    Grossman, D., Domingos, P.: Learning Bayesian Network Classifiers by Maximizing Conditional Likelihood. In: Proceedings of ICML 2004, pp. 361–368 (2004)Google Scholar
  11. 11.
    Ling, C.X., Huang, J., Zhang, H.: AUC: A Better Measure than Accuracy in Comparing Learning Algorithms. In: Proceedings of Canadian AI 2003, pp. 329–341 (2003)Google Scholar
  12. 12.
    Margineantu, D.D., Dietterich, T.G.: Improved Class Probability Estimates from Decision Tree Models. Nonlinear Estimation and Classification 171, 169–184 (2002)MATHGoogle Scholar
  13. 13.
    Provost, F., Domingos, P.: Tree Induction for Probability-Based Ranking. Machine Learning 52, 199–215 (2003)CrossRefMATHGoogle Scholar
  14. 14.
    Vanderlooy, S., Hullermeier, E.: A Critical Analysis of Variants of the AUC. Machine Learning 72(3), 247–262 (2008)CrossRefGoogle Scholar
  15. 15.
    Wu, S., Flach, P.A., Ferri, C.: An Improved Model Selection Heuristic for AUC. In: Kok, J.N., Koronacki, J., Lopez de Mantaras, R., Matwin, S., Mladenič, D., Skowron, A. (eds.) ECML 2007. LNCS (LNAI), vol. 4701, pp. 478–489. Springer, Heidelberg (2007)CrossRefGoogle Scholar
  16. 16.
    Zhang, H., Su, J.: Learning Probability Decision Trees for AUC. Pattern Recognition Letters 27, 892–899 (2006)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2011

Authors and Affiliations

  • William Klement
    • 1
  • Peter Flach
    • 2
  • Nathalie Japkowicz
    • 1
  • Stan Matwin
    • 1
    • 3
  1. 1.School of Electrical Engineering and Computer ScienceUniversity of OttawaCanada
  2. 2.Computer ScienceBristol UniversityUnited Kingdom
  3. 3.Institute of Computer SciencePolish Academy of SciencesWarsawPoland

Personalised recommendations