Expected Reciprocal Rank

Definition

Expected reciprocal rank (ERR) is an information retrieval evaluation measure suitable for navigational queries, where the user requires only a small number of relevant documents. It generalizes reciprocal rank, which is based on binary relevance and considers only the first relevant document retrieved. Let Pr(r) denote the probability that the user is satisfied with a document at rank r. ERR assumes that the user stops scanning the ranked list as soon as she is satisfied with a document and that this satisfaction probability depends directly and solely on the relevance level of each document. For example, we can assume that Pr(r) = 0 if the document at r is nonrelevant; if we have partially relevant, relevant, and highly relevant documents (i.e., three relevance levels), we may let Pr(r) be (2¹ − 1)∕2³ = 1∕8, (2² − 1)∕2³ = 3∕8, and (2³ − 1)∕2³ = 7∕8, respectively. Under the linear traversalassumption (i.e., the user scans the list top down), the...