An Intrinsic Framework of Information Retrieval Evaluation Measures

Abstract

Information retrieval (IR) evaluation measures are cornerstones for determining the suitability and task performance efficiency of retrieval systems. Their metric and scale properties enable to compare one system against another to establish differences or similarities. Based on the representational theory of measurement, this paper determines these properties by exploiting the information contained in a retrieval measure itself. It establishes the intrinsic framework of a retrieval measure, which is the common scenario when the domain set is not explicitly specified. A method to determine the metric and scale properties of any retrieval measure is provided, requiring knowledge of only some of its attained values. The method establishes three main categories of retrieval measures according to their intrinsic properties. Some common user-oriented and system-oriented evaluation measures are classified according to the presented taxonomy.

A Appendix

1.1 A.1 Formal Proofs

Proof

(Proposition 1) Symmetry is trivially verified since $d_{f}(\mathbf {\hat{r}_1},\mathbf {\hat{r}_2})= \vert f(\mathbf {\hat{r}_1}) - f(\mathbf {\hat{r}_2}) \vert = \vert f(\mathbf {\hat{r}_2}) - f(\mathbf {\hat{r}_1}) \vert = d_{f}(\mathbf {\hat{r}_2},\mathbf {\hat{r}_1})$. Triangular inequality is also trivial, by considering the triangular inequality on the real numbers: $\vert f(\mathbf {\hat{r}_1}) - f(\mathbf {\hat{r}_2}) \vert \le \vert f(\mathbf {\hat{r}_1}) - f(\mathbf {\hat{r}_3}) \vert + \vert f(\mathbf {\hat{r}_3}) - f(\mathbf {\hat{r}_2}) \vert$.   $\square$

Proof

(Proposition 2) An interesting result about metric spaces [42] states the following: “Let $(\mathbf {R_2}, d_2)$ be a metric space and let $f:\mathbf {R_1} \longrightarrow \mathbf {R_2}$ an an injective or one-to-one function, then $(\mathbf {R_1}, d_1)$ is a metric space, where $d_1(\mathbf {\hat{r}_1}, \mathbf {\hat{r}_2}) = d_2(f(\mathbf {\hat{r}_1}), f(\mathbf {\hat{r}_2}))$, $\forall \mathbf {\hat{r}_1}$, $\mathbf {\hat{r}_2} \in \mathbf {R_1}$”.

In the retrieval scenario, $(\mathbf {R_2}, d_2) = (\mathbb {R}, \vert \cdot \vert )$, which is the metric space of the real line endowed with the usual norm (the absolute value). Let f be a one-to-one IR evaluation measure; from the previous result, it follows that $(\mathbf {R_1}, d_1) = (\textbf{R}, d_f)$ is a metric space, i.e., $d_f$ verifies the three postulates of a metric.   $\square$

Proof

(Proposition 3) It will be seen the implication from right to left. Consider a metric ordinal scale, f, where the attained values are equally spaced.

An interval is called prime if $[\mathbf {\hat{r}_1}, \mathbf {\hat{r}_2}] = \{\mathbf {\hat{r}_1}, \mathbf {\hat{r}_2}\}$. First, it will be seen that the function, $F(\textbf{x}, \textbf{y}) = \vert f(\textbf{x}) - f(\textbf{y}) \vert$, attains its minimum value on any prime interval.

Let $[\mathbf {\hat{r}_1}, \mathbf {\hat{r}_3}] = \{\mathbf {\hat{r}_1}, \mathbf {\hat{r}_2}, \mathbf {\hat{r}_3}\}$ be a non-prime interval, where $\mathbf {\hat{r}_1} \preceq _f \mathbf {\hat{r}_2} \preceq _f \mathbf {\hat{r}_3}$, then it holds that $f(\mathbf {\hat{r}_1}) \le f(\mathbf {\hat{r}_2}) \le f(\mathbf {\hat{r}_3})$ since f is an ordinal scale. It implies that $\vert f(\mathbf {\hat{r}_3}) - f(\mathbf {\hat{r}_1}) \vert \le \vert f(\mathbf {\hat{r}_3}) - f(\mathbf {\hat{r}_2}) \vert + \vert f(\mathbf {\hat{r}_2}) - f(\mathbf {\hat{r}_1}) \vert$, i.e., the minimum value of F is not attained at $[\mathbf {\hat{r}_1}, \mathbf {\hat{r}_3}]$. In addition, it holds that the function F assign the same value for every prime interval. Given a prime interval, $[\mathbf {\hat{r}_1}, \mathbf {\hat{r}_2}]$, it can be considered one of its consecutive prime intervals, $[\mathbf {\hat{r}_2}, \mathbf {\hat{r}_3}]$, since $\preceq _f$ is a weak order (every pair of elements is comparable). These two prime intervals verify that $f(\mathbf {\hat{r}_1}) < f(\mathbf {\hat{r}_2}) < f(\mathbf {\hat{r}_3})$ since f is a metric, and the attained values of f are equally spaced. Thus, it can be assumed that $F(\mathbf {\hat{r}_1},\mathbf {\hat{r}_2}) = k \in \mathbb {R}^{+}$ for any prime interval $[\mathbf {\hat{r}_1}, \mathbf {\hat{r}_2}]$.

Now, it will be seen that equally spaced intervals (not necessarily prime) are assigned equal differences. Consider any non-prime interval, $[\mathbf {\hat{r}_1}, \mathbf {\hat{r}_m}] = \{\mathbf {\hat{r}_1}, \mathbf {\hat{r}_2}, \ldots , \mathbf {\hat{r}_m}\}$. As f is a metric, then it attains different values for different elements. Thus, it can be assumed that $f(\mathbf {\hat{r}_1}) < f(\mathbf {\hat{r}_2}) < \cdots < f(\mathbf {\hat{r}_{m-1}}) < f(\mathbf {\hat{r}_m})$. Then, every interval $[\mathbf {\hat{r}_i}, \mathbf {\hat{r}_{i+1}}]$ are prime intervals for $i=1, \ldots m-1$ since F attain the minimum at these intervals. As $f(\mathbf {\hat{r}_m}) - f(\mathbf {\hat{r}_1}) = f(\mathbf {\hat{r}_m}) - f(\mathbf {\hat{r}_{m-1}}) + f(\mathbf {\hat{r}_{m-1}}) - \cdots - f(\mathbf {\hat{r}_2}) + f(\mathbf {\hat{r}_2}) - f(\mathbf {\hat{r}_1})$ and $f(\mathbf {\hat{r}_{i+1}}) - f(\mathbf {\hat{r}_i}) = k$ for $1 \le i \le m-1$, then $f(\mathbf {\hat{r}_1}) - f(\mathbf {\hat{r}_m}) = k \cdot m$, which only depends on the span of the interval, m, not on the considered elements. Therefore, equally spaced intervals are assigned equal differences, i.e., f is an interval scale.

Finally, it will be seen the other implication. Consider any prime interval, $[\mathbf {\hat{r}_1}, \mathbf {\hat{r}_2}]$, of $\textbf{R}$, as f is an interval scale, then equally spaced intervals are assigned to equal differences, i.e., the value $\vert f(\mathbf {\hat{r}_2}) - f(\mathbf {\hat{r}_1}) \vert$ is constant for every prime interval of $\textbf{R}$. In addition, it should be an strictly positive value. To see that the attained values are equally spaced, it is sufficient to check that different elements of $\textbf{R}$ are assigned different values of f, which is hold since f is a metric.   $\square$