On the Noise Resilience of Ranking Measures

Conference paper

DOI: 10.1007/978-3-319-46672-9_6

Part of the Lecture Notes in Computer Science book series (LNCS, volume 9948)
Cite this paper as:
Berrar D. (2016) On the Noise Resilience of Ranking Measures. In: Hirose A., Ozawa S., Doya K., Ikeda K., Lee M., Liu D. (eds) Neural Information Processing. ICONIP 2016. Lecture Notes in Computer Science, vol 9948. Springer, Cham

Abstract

Performance measures play a pivotal role in the evaluation and selection of machine learning models for a wide range of applications. Using both synthetic and real-world data sets, we investigated the resilience to noise of various ranking measures. Our experiments revealed that the area under the ROC curve (AUC) and a related measure, the truncated average Kolmogorov-Smirnov statistic (taKS), can reliably discriminate between models with truly different performance under various types and levels of noise. With increasing class skew, however, the H-measure and estimators of the area under the precision-recall curve become preferable measures. Because of its simple graphical interpretation and robustness, the lower trapezoid estimator of the area under the precision-recall curve is recommended for highly imbalanced data sets.

Keywords

Ranking Classification Noise Robustness ROC curve AUC H-measure taKS Precision-recall curve 

Copyright information

© Springer International Publishing AG 2016

Authors and Affiliations

  1. 1.School of Arts and Sciences, College of EngineeringShibaura Institute of TechnologyMinuma-kuJapan

Personalised recommendations