Reinventing Machine Learning with ROC Analysis

  • Peter A. Flach
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 4140)

Abstract

Receiver Operating Characteristics (ROC) Analysis originated from signal detection theory, as a model of how well a receiver is able to detect a signal in the presence of noise [1,9]. Its key feature is the distinction between hit rate (or true positive rate) and false alarm rate (or false positive rate) as two separate performance measures. ROC analysis has also widely been used in medical data analysis to study the effect of varying the threshold on the numerical outcome of a diagnostic test. It has been introduced to machine learning relatively recently, in response to classification tasks with skewed class distributions or misclassification costs [11,12,5].

References

  1. 1.
    Egan, J.P.: Signal detection theory and ROC analysis. Academic Press, New York (1975)Google Scholar
  2. 2.
    Ferri, C., Flach, P.A., Hernández-Orallo, J.: Learning decision trees using the area under the ROC curve. In: Proceedings of the Nineteenth International Conference on Machine Learning (ICML 2002), pp. 139–146 (2002)Google Scholar
  3. 3.
    Ferri, C., Flach, P.A., Hernández-Orallo, J., Senad, A.: Modifying ROC curves to incorporate predicted probabilities. In: Proceedings of the second workshop on ROC analysis in machine learning, ICML 2005 workshop proceedings (2005)Google Scholar
  4. 4.
    Flach, P.A.: The geometry of ROC space: Understanding machine learning metrics through ROC isometrics. In: Proceedings of the Twentieth International Conference on Machine Learning (ICML 2003), pp. 194–201 (2003)Google Scholar
  5. 5.
    Flach, P.A.: The many faces of ROC analysis in machine learning, ICML 2004 tutorial notes (2004), http://www.cs.bris.ac.uk/~flach/ICML04tutorial/
  6. 6.
    Flach, P.A., Wu, S.: Repairing concavities in ROC curves. In: Proceedings of the 19th International Joint Conference on Artificial Intelligence (IJCAI 2005), pp. 702–707 (2005)Google Scholar
  7. 7.
    Fürnkranz, J., Flach, P.A.: An analysis of rule evaluation metrics. In: Proceedings of the Twentieth International Conference on Machine Learning (ICML 2003), pp. 202–209 (2003)Google Scholar
  8. 8.
    Fürnkranz, J., Flach, P.A.: ROC ’n’ rule learning-towards a better understanding of covering algorithms. Machine Learning 58(1), 39–77 (2005)MATHCrossRefGoogle Scholar
  9. 9.
    Green, D.M., Swets, J.A.: Signal detection theory and psychophysics. Wiley, New York (1966)Google Scholar
  10. 10.
    Prati, R.C., Flach, P.A.: ROCCER: An algorithm for rule learning based on ROC analysis. In: Proceedings of the 19th International Joint Conference on Artificial Intelligence (IJCAI 2005), pp. 823–828 (2005)Google Scholar
  11. 11.
    Provost, F.J., Fawcett, T.: Analysis and visualization of classifier performance: Comparison under imprecise class and cost distributions. In: Proceedings of the Third International Conference on Knowledge Discovery and Data Mining (KDD 1997), pp. 43–48 (1997)Google Scholar
  12. 12.
    Provost, F.J., Fawcett, T.: Robust classification for imprecise environments. Machine Learning 42(3), 203–231 (2001)MATHCrossRefGoogle Scholar
  13. 13.
    Wu, S., Flach, P.A.: A scored AUC metric for model evaluation and selection. In: Proceedings of the second workshop on ROC analysis in machine learning. In: ICML 2005 workshop proceedings (2005)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2006

Authors and Affiliations

  • Peter A. Flach
    • 1
  1. 1.Department of Computer ScienceUniversity of BristolUnited Kingdom

Personalised recommendations