Machine Learning

, Volume 77, Issue 1, pp 103–123

Measuring classifier performance: a coherent alternative to the area under the ROC curve


DOI: 10.1007/s10994-009-5119-5

Cite this article as:
Hand, D.J. Mach Learn (2009) 77: 103. doi:10.1007/s10994-009-5119-5


The area under the ROC curve (AUC) is a very widely used measure of performance for classification and diagnostic rules. It has the appealing property of being objective, requiring no subjective input from the user. On the other hand, the AUC has disadvantages, some of which are well known. For example, the AUC can give potentially misleading results if ROC curves cross. However, the AUC also has a much more serious deficiency, and one which appears not to have been previously recognised. This is that it is fundamentally incoherent in terms of misclassification costs: the AUC uses different misclassification cost distributions for different classifiers. This means that using the AUC is equivalent to using different metrics to evaluate different classification rules. It is equivalent to saying that, using one classifier, misclassifying a class 1 point is p times as serious as misclassifying a class 0 point, but, using another classifier, misclassifying a class 1 point is P times as serious, where pP. This is nonsensical because the relative severities of different kinds of misclassifications of individual points is a property of the problem, not the classifiers which happen to have been chosen. This property is explored in detail, and a simple valid alternative to the AUC is proposed.


ROC curves Classification AUC Specificity Sensitivity Misclassification rate Cost Loss Error rate 

Copyright information

© Springer Science+Business Media, LLC 2009

Authors and Affiliations

  1. 1.Department of MathematicsImperial College LondonLondonUK
  2. 2.Institute for Mathematical SciencesImperial College LondonLondonUK

Personalised recommendations