ERR-IA

Synonyms

Intent-aware ERR; Intent-aware expected reciprocal rank

Definition

ERR-IA, an intent-aware version of expected reciprocal rank (ERR), is an evaluation measure for diversified search. In diversified search evaluation, it is assumed that a topic q has multiple possible intents {i} and that intent probabilities Pr(i|q) and per-intent relevance assessments are available. Given the ERR score computed for each intent, denoted by ERR_i, ERR-IA is given by

$$ \displaystyle \begin{aligned} \mathrm {ERR}\mbox{-}\mathrm {IA}=\sum\limits_{i} \mathit{Pr}(i|q)\mathrm {ERR}_{i}. \end{aligned} $$

The per-intent ERR is computed as follows. For a particular intent, let Pr(r) denote the probability that the user is satisfied with a document at rank r. For example, we can assume that Pr(r) = 0 if the document at r is nonrelevant to the intent; if we have partially relevant, relevant, and highly relevant documents for the intent (i.e., three relevance levels), we may let Pr(r) be (2¹ − 1)∕2³=...