The ClasSi coefficient for the evaluation of ranking quality in the presence of class similarities
Evaluation measures play an important role in the design of new approaches, and often quality is measured by assessing the relevance of the obtained result set. While many evaluation measures based on precision/recall are based on a binary relevance model, ranking correlation coefficients are better suited for multi-class problems. State-of-the-art ranking correlation coefficients like Kendall’s τ and Spearman’s ρ do not allow the user to specify similarities between differing object classes and thus treat the transposition of objects from similar classes the same way as that of objects from dissimilar classes. We propose ClasSi, a new ranking correlation coefficient which deals with class label rankings and employs a class distance function to model the similarities between the classes. We also introduce a graphical representation of ClasSi which describes how the correlation evolves throughout the ranking.
Keywordsranking quality measure class similarity ClasSi
Unable to display preview. Download preview PDF.
- 1.van Rijsbergen C J. Information Retrieval. 2nd ed. London: Butterworth-Heinemann, 1979Google Scholar
- 3.Flach P A, Blockeel H, Ferri C, Hernández-Orallo J, Struyf J. Decision support for data mining; introduction to ROC analysis and its applications. In: Mladenic D, Lavračn, Bohanec M, Moyle S, eds. Data Mining and Decision Support: Integration and Collaboration. Boston: Kluwer Academic Publishers, 2003, 81–90CrossRefGoogle Scholar
- 5.Ferri C, Hernández-Orallo J, Salido M A. Volume under the ROC surface for multi-class problems. In: Proceedings of the 14th European Conference on Machine Learning. 2003, 108–120Google Scholar
- 6.Hassan M R, Ramamohanarao K, Karmakar C K, Hossain M M, Bailey J. A novel scalable multi-class ROC for effective visualization and computation. In: Proceedings of the 14th Pacific-Asia Conference on Knowledge Discovery and Data Mining, Part I. 2010, 107–120Google Scholar
- 12.Ivanescu A, Wichterich M, Seidl T. ClasSi: measuring ranking quality in the presence of object classes with similarity information. In: Proceedings of PAKDD 2011 Quality Issues, Measures of Interestingness and Evaluation of Data Mining Models Workshop. 2011, 185–196Google Scholar
- 13.Beecks C, Uysal M S, Seidl T. Signature quadratic form distance. In: Proceedings of the 2010 ACM International Conference on Image and Video Retrieval. 2010, 438–445Google Scholar