Article

Machine Learning

, Volume 6, Issue 1, pp 67-80

Information-Based Evaluation Criterion for Classifier's Performance

  • Igor KononenkoAffiliated withFaculty of Electrical and Computer Engineering
  • , Ivan BratkoAffiliated withFaculty of Electrical and Computer Engineering, Jozef Stefan Institute

Abstract

In the past few years many systems for learning decision rules from examples were developed. As different systems allow different types of answers when classifying new instances, it is difficult to appropriately evaluate the systems' classification power in comparison with other classification systems or in comparison with human experts. Classification accuracy is usually used as a measure of classification performance. This measure is, however, known to have several defects. A fair evaluation criterion should exclude the influence of the class probabilities which may enable a completely uninformed classifier to trivially achieve high classification accuracy. In this paper a method for evaluating the information score of a classifier's answers is proposed. It excludes the influence of prior probabilities, deals with various types of imperfect or probabilistic answers and can be used also for comparing the performance in different domains.

Classifier evaluation criteria machine learning information theory