The Quality of a Ranking Method

Abstract

The most commonly used performance measures represent the characteristics of a search up to a certain point in the retrieval process. These measures include precision, recall, and Van Rijsbergen’s and Shaw’s E measure. Swets’ E measure, on the other hand, addresses the performance of a particular ranking based on the characteristics of the distributions for relevant and non-relevant documents. It is therefore independent of the stopping point chosen in a particular search. The relationship between ordered sequences, such as lists of ranked documents, may be studied using a number of techniques (Levenstein, 1966; Myers & Miller, 1988), ranging from sophisticated cost-based methods to counting the number of swaps necessary in sorting one sequence so that it becomes the other sequence. How best to study the relationships between sequences is an open question.

There is [a]... great cause of the little advancement of the sciences, which is this: it is impossible to advance properly in the course when the goal is not properly fixed.

—Francis Bacon, 1620