Abstract
Operating Characteristic (ROC) analysis has been successfully applied to classifier problems with two classes. The Area Under the ROC Curve (AUC) has been elected as a better way to evaluate classifiers than predictive accuracy or error and has also recently used for evaluating probability estimators. However, the extension of the Area Under the ROC Curve for more than two classes has not been addressed to date, because of the complexity and elusiveness of its precise definition. Some approximations to the real AUC are used without an exact appraisal of their quality. In this paper, we present the real extension to the Area Under the ROC Curve in the form of the Volume Under the ROC Surface (VUS), showing how to compute the polytope that corresponds to the absence of classifiers (given only by the trivial classifiers), to the best classifier and to whatever set of classifiers. We compare the real VUS with ”approximations” or ”extensions” of the AUC for more than two classes.
Chapter PDF
Similar content being viewed by others
Keywords
- Receiver Operating Characteristic
- Receiver Operating Characteristic Curve
- Receiver Operating Characteristic Analysis
- Minority Class
- Cost Matrix
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.
References
Adams, N.M., Hand, D.J.: Comparing classifiers when the misallocation costs are uncertain. Pattern Recognition 32(7), 1139–1147 (1999)
Barber, C.B., Huhdanpaa, H.: “QHull”, The Geometry Center, University of Minnesota, http://www.geom.umn.edu/software/qhull/
Boissonat, J.D., Yvinec, M.: Algorithmic Geometry. Cambridge University Press, Cambridge (1998)
Ferri, C., Hernández-Orallo, J., Salido, M.A.: Volume Under the ROC Surface for Multiclass Problems. Exact Computation and Evaluation of Approximations. Technical Report DSIC. Univ. Politèc. València (2003), http://www.dsic.upv.es/users/elp/cferri/VUS.pdf
Flach, P., Blockeel, H., Ferri, C., Hernández-Orallo, J., Struyf, J.: Decision support for data mining; Introduction to ROC analysis and its applications. In: Data Mining and Decision Support: Integration and Collaboration, Kluwer Publishers, Dordrecht (2003) (to appear)
Hand, D.J., Till, R.J.: A Simple Generalisation of the Area Under the ROC Curve for Multiple Class Classification Problems. Machine Learning 45, 171–186 (2001)
Hanley, J.A., McNeil, B.J.: The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology 143, 29–36 (1982)
Lane, T.: Extensions of ROC Analysis to Multi-Class Domains. In: ICML 2000 Workshop on cost-sensitive learning (2000)
Provost, F., Fawcett, T.: Analysis and visualization of classifier performance: Comparison under imprecise class and cost distribution. In: Proc. of The Third International Conference on Knowledge Discovery and Data Mining (KDD 1997), pp. 43–48. AAAI Press, Menlo Park (1997)
Provost, F., Domingos, P.: Tree Induction for Probability-based Ranking. Machine Learning 52(3), 199–215 (2003)
Salido, M.A., Giret, A., Barber, F.: Constraint Satisfaction by means of Dynamic Polyhedra. In: Operations Research Proceedings 2001, pp. 405–412. Springer, Heidelberg (2002)
Srinivasan, A.: Note on the Location of Optimal Classifiers in N-dimensional ROC Space. Technical Report PRG-TR-2-99, Oxford University Computing Laboratory
Swets, J., Dawes, R., Monahan, J.: Better decisions through science. Scientific American, 82–87 (October 2000)
Turney, P.: Cost-sensitive classification: Empirical evaluation of a hybrid genetic decision tree induction algorithm. Journal of Artificial Intelligence Research 2, 369–409 (1995)
Zweig, M.H., Campbell, G.: Receiver-operating characteristic (ROC) plots: a fundamental evaluation tool in clinical medicine. Clin. Chem. 39, 561–577 (1993)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2003 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Ferri, C., Hernández-Orallo, J., Salido, M.A. (2003). Volume under the ROC Surface for Multi-class Problems. In: Lavrač, N., Gamberger, D., Blockeel, H., Todorovski, L. (eds) Machine Learning: ECML 2003. ECML 2003. Lecture Notes in Computer Science(), vol 2837. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-39857-8_12
Download citation
DOI: https://doi.org/10.1007/978-3-540-39857-8_12
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-20121-2
Online ISBN: 978-3-540-39857-8
eBook Packages: Springer Book Archive