Volume under the ROC Surface for Multi-class Problems

  • César Ferri
  • José Hernández-Orallo
  • Miguel Angel Salido
Part of the Lecture Notes in Computer Science book series (LNCS, volume 2837)


Operating Characteristic (ROC) analysis has been successfully applied to classifier problems with two classes. The Area Under the ROC Curve (AUC) has been elected as a better way to evaluate classifiers than predictive accuracy or error and has also recently used for evaluating probability estimators. However, the extension of the Area Under the ROC Curve for more than two classes has not been addressed to date, because of the complexity and elusiveness of its precise definition. Some approximations to the real AUC are used without an exact appraisal of their quality. In this paper, we present the real extension to the Area Under the ROC Curve in the form of the Volume Under the ROC Surface (VUS), showing how to compute the polytope that corresponds to the absence of classifiers (given only by the trivial classifiers), to the best classifier and to whatever set of classifiers. We compare the real VUS with ”approximations” or ”extensions” of the AUC for more than two classes.


Receiver Operating Characteristic Receiver Operating Characteristic Curve Receiver Operating Characteristic Analysis Minority Class Cost Matrix 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. 1.
    Adams, N.M., Hand, D.J.: Comparing classifiers when the misallocation costs are uncertain. Pattern Recognition 32(7), 1139–1147 (1999)CrossRefGoogle Scholar
  2. 2.
    Barber, C.B., Huhdanpaa, H.: “QHull”, The Geometry Center, University of Minnesota,
  3. 3.
    Boissonat, J.D., Yvinec, M.: Algorithmic Geometry. Cambridge University Press, Cambridge (1998)Google Scholar
  4. 4.
    Ferri, C., Hernández-Orallo, J., Salido, M.A.: Volume Under the ROC Surface for Multiclass Problems. Exact Computation and Evaluation of Approximations. Technical Report DSIC. Univ. Politèc. València (2003),
  5. 5.
    Flach, P., Blockeel, H., Ferri, C., Hernández-Orallo, J., Struyf, J.: Decision support for data mining; Introduction to ROC analysis and its applications. In: Data Mining and Decision Support: Integration and Collaboration, Kluwer Publishers, Dordrecht (2003) (to appear)Google Scholar
  6. 6.
    Hand, D.J., Till, R.J.: A Simple Generalisation of the Area Under the ROC Curve for Multiple Class Classification Problems. Machine Learning 45, 171–186 (2001)zbMATHCrossRefGoogle Scholar
  7. 7.
    Hanley, J.A., McNeil, B.J.: The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology 143, 29–36 (1982)Google Scholar
  8. 8.
    Lane, T.: Extensions of ROC Analysis to Multi-Class Domains. In: ICML 2000 Workshop on cost-sensitive learning (2000)Google Scholar
  9. 9.
    Provost, F., Fawcett, T.: Analysis and visualization of classifier performance: Comparison under imprecise class and cost distribution. In: Proc. of The Third International Conference on Knowledge Discovery and Data Mining (KDD 1997), pp. 43–48. AAAI Press, Menlo Park (1997)Google Scholar
  10. 10.
    Provost, F., Domingos, P.: Tree Induction for Probability-based Ranking. Machine Learning 52(3), 199–215 (2003)zbMATHCrossRefGoogle Scholar
  11. 11.
    Salido, M.A., Giret, A., Barber, F.: Constraint Satisfaction by means of Dynamic Polyhedra. In: Operations Research Proceedings 2001, pp. 405–412. Springer, Heidelberg (2002)Google Scholar
  12. 12.
    Srinivasan, A.: Note on the Location of Optimal Classifiers in N-dimensional ROC Space. Technical Report PRG-TR-2-99, Oxford University Computing LaboratoryGoogle Scholar
  13. 13.
    Swets, J., Dawes, R., Monahan, J.: Better decisions through science. Scientific American, 82–87 (October 2000)Google Scholar
  14. 14.
    Turney, P.: Cost-sensitive classification: Empirical evaluation of a hybrid genetic decision tree induction algorithm. Journal of Artificial Intelligence Research 2, 369–409 (1995)Google Scholar
  15. 15.
    Zweig, M.H., Campbell, G.: Receiver-operating characteristic (ROC) plots: a fundamental evaluation tool in clinical medicine. Clin. Chem. 39, 561–577 (1993)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2003

Authors and Affiliations

  • César Ferri
    • 1
  • José Hernández-Orallo
    • 1
  • Miguel Angel Salido
    • 1
  1. 1.Dep. Sistemes Informàtics i ComputacióUniv. Politècnica de València(Spain)

Personalised recommendations