Scoring in context

Abstract

A number of authors have recently put forward arguments pro or contra various rules for scoring probability estimates. In doing so, they have skipped over a potentially important consideration in making such assessments, to wit, that the hypotheses whose probabilities are estimated can approximate the truth to different degrees. Once this is recognized, it becomes apparent that the question of how to assess probability estimates depends heavily on context.

Appendices

Appendix A

Recall that the weights of a VS rule are all positive and add up to 1, and that they are said to reflect truthlikeness in a minimally adequate sense iff hypotheses are assigned weights as a function of their distance from the truth, with hypotheses farther from the truth being assigned larger weights than hypotheses closer to the truth.

Theorem 1

Every VS rule whose weights reflect truthlikeness in a minimally adequate sense is improper.

Proof

Without loss of generality, consider a hypothesis partition of three hypotheses, $H_1$, $H_2$, and $H_3$. Then, where $\mathcal {V}$ is some VS rule and $\mathbf {p}=(p_1,p_2,p_3)$ is a given person’s probability assignment to the aforementioned hypotheses, with $p_i$ the probability assigned to $H_i$, this person’s expected $\mathcal {V}$-score for a probability assignment $\mathbf {p}^*\!$ to the same hypotheses is given by the function

$$\begin{aligned} \mathbb {E}_{\mathbf {p}}[\mathcal {V}(\mathbf {p}^*)]= & {} p_1\bigl (w_{11}(1 - p^*_1)^2 + w_{21}(p^*_2)^2 + w_{31}(p^*_3)^2\bigr ) \\&+ p_2\bigl (w_{12}(p^*_1)^2 + w_{22}(1 - p^*_2)^2 + w_{32}(p^*_3)^2\bigr ) \\&+ p_3\bigl (w_{13}(p^*_1)^2 + w_{23}(p^*_2)^2 + w_{33}(1 - p^*_3)^2\bigr ). \end{aligned}$$

Again without loss of generality, assume that the hypotheses are ordered by their distances from each other, with $H_2$ being equally far from $H_1$ and $H_3$, and $H_1$ and $H_3$ being twice as far from each other as they are from $H_2$. Then $w_{11}=w_{33}$, $w_{21}=w_{23}$, $w_{31}=w_{13}$, and $w_{12}=w_{32}$, so that we can simplify notation by defining ; ; ; ; and . For $\mathcal {V}$ to be proper, it must hold that $\mathrm{arg\,min}_{\mathbf {p}^*}\!\mathbb {E}_{\mathbf {p}}[\mathcal {V}(\mathbf {p}^*)]=\mathbf {p}$, for any distribution $\mathbf {p}$ on $\{H_1,H_2,H_3\}$. To see whether this does hold, we use the method of Lagrange multipliers. Specifically, where $f(\mathbf {p}^*)=p^*_1+p^*_2+p^*_3$, we must find values for $p^*_1$, $p^*_2$, $p^*_3$, and $\lambda$ such that $\nabla \mathbb {E}_{\mathbf {p}}[\mathcal {V}(\mathbf {p}^*)] = \lambda \nabla f(\mathbf {p}^*)$ and $f(\mathbf {p}^*)=1$. Calculating the first-order partial derivatives of $\mathbb {E}_{\mathbf {p}}[\mathcal {V}(\mathbf {p}^*)]$, we find

$$\begin{aligned} (\partial /\partial p^*_1)\mathbb {E}_{\mathbf {p}}[\mathcal {V}(\mathbf {p}^*)]= & {} -2 w_1 p_1 (1 - p^*_1) + 2 w_3 p_3 p^*_1 + 2 w_4 p_2 p^*_1; \\ (\partial /\partial p^*_2)\mathbb {E}_{\mathbf {p}}[\mathcal {V}(\mathbf {p}^*)]= & {} -2 w_5 p_2 (1 - p^*_2) + 2 w_2 p_1 p^*_2 + 2 w_2 p_3 p^*_2; \\ (\partial /\partial p^*_3)\mathbb {E}_{\mathbf {p}}[\mathcal {V}(\mathbf {p}^*)]= & {} -2 w_1 p_3 (1 - p^*_3) + 2 w_3 p_1 p^*_3 + 2 w_4 p_2 p^*_3. \end{aligned}$$

Because $\nabla f(\mathbf {p}^*)=\mathbf {1}$, we have $(\partial /\partial p^*_i)\mathbb {E}_{\mathbf {p}}[\mathcal {V}(\mathbf {p}^*)]=\lambda$ for all $i\leqslant 3$. So in particular, expanding the partial derivatives in $p^*_1$ and $p^*_3$ and dividing both by 2, we have

$$\begin{aligned} -w_1 p_1 + w_1 p_1 p^*_1 + w_3 p_3 p^*_1 + w_4 p_2 p^*_1 = -w_1 p_3 + w_1 p_3 p^*_3 + w_3 p_1 p^*_3 + w_4 p_2 p^*_3, \end{aligned}$$

and hence

$$\begin{aligned} w_1 p_1 p^*_1 + w_3 p_3 p^*_1 + w_4 p_2 p^*_1 - w_1 p_3 p^*_3 - w_3 p_1 p^*_3 - w_4 p_2 p^*_3 - w_1 p_1 + w_1 p_3 = 0. \end{aligned}$$

Suppose that $\mathcal {V}$ is proper, so that $\mathbb {E}_{\mathbf {p}}[\mathcal {V}(\mathbf {p}^*)]$ reaches its minimum if $p_1 = p^*_1$, $p_2 = p^*_2$, and $p_3 = p^*_3$. Then there must be values for the $w_i$ such that

$$\begin{aligned} w_1 (p_1)^2 + w_3 p_3 p_1 + w_4 p_2 p_1 - w_1 (p_3)^2 - w_3 p_1 p_3 - w_4 p_2 p_3 - w_1 p_1 + w_1 p_3 \,\, = \,\, 0. \end{aligned}$$

However, factoring the left-hand side yields

$$\begin{aligned} (p_1 - p_3) (-w_1 + w_1 p_1 + w_4 p_2 + w_1 p_3). \end{aligned}$$

This equals 0 iff either (i) $p_1=p_3$ or (ii) $w_1=w_4$, where the latter follows from the fact that the condition that the right-hand factor equals 0 can be rewritten as $w_1(1-p_1-p_3)=w_4 p_2$, in conjunction with the fact that the $p_i$ sum to 1. Because, as said, for $\mathcal {V}$ to be proper, it must hold for all $\mathbf {p}$ that $\mathrm{arg\,min}_{\mathbf {p}^*}\!\mathbb {E}_{\mathbf {p}}[\mathcal {V}(\mathbf {p}^*)]=\mathbf {p}$, we may pick a $\mathbf {p}$ such that $p_1\ne p_3$, thereby violating (i). As for (ii), note that whichever precise values the $w_i$ assume, $w_1$ must be smaller than 1/3 (given that it is assigned to the supposed truth) and $w_4$ must be greater than 1/3 (given that it is assigned to the two hypotheses supposed false). Consequently, on the supposition that $\mathcal {V}$ is proper, we can minimize $\mathbb {E}_{\mathbf {p}}[\mathcal {V}(\mathbf {p}^*)]$ subject to the given constraint iff the truthlikeness weights assigned by the rule do not reflect truthlikeness in a minimally adequate sense. By assumption, the weights do reflect truthlikeness in a minimally adequate sense. Given that we made no further assumptions about $\mathcal {V}$, it follows that every VS rule is improper if it assigns truthlikeness weights in a minimally adequate fashion. $\square$

Remark

The above proof proceeds by constructing a specific counterexample involving three hypotheses that are assumed to stand in specific relations of truthlikeness to each other. To see that this assumption does not undermine the generality of the proof, we note that the said relations are perfectly possible according to all modern measures of truthlikeness (see page 5 for references). As a matter of fact, one can think of our earlier example concerning the possible grades (A, B, or C) a given student may receive as instantiating exactly the relations of truthlikeness that are assumed to hold in the counterexample. It is also to be noted, however, that not all known measures of truthlikeness will do for the purposes of the proof. Most famously, Tichý (1974) discovered that on Popper’s (1963) measure all false theories are equally far from the truth, contrary to what Popper had hoped to achieve with his measure.

Appendix B

In this appendix we prove

Theorem 2

Let S be the standard unit $(n-1)$-simplex, let $\mathbf {p}$ and $\mathbf {p}^*\!$ range over vectors in S, and let $m:S\rightarrow S$ be defined as follows:

with $\delta _{ij}$ the Kronecker delta, and with $w_{ij}>0$ for all i, j, and $\sum _{i=1}^n\sum _{j=1}^n w_{ij} = 1$. Then there is a $\mathbf {p}^+\!\in S$ such that (i) $m(\mathbf {p}^+)=\mathbf {p}^+\!$, (ii) $\mathbf {p}^+$ is unique, and (iii) $\mathbf {p}^+$ depends only on the $w_{ij}$.

Proof

Clause (i) follows from Brouwer’s (1911) fixed-point theorem, which (in one version) states that every continuous function from a simplex onto itself has a fixed point. It does not follow from Brouwer’s theorem that the fixed point is unique.

To prove clause (ii), then, one first verifies that the function that is being minimized at each step on the way to the fixed point has the Hessian

$$\begin{aligned} \begin{bmatrix} \phantom {.} 2 (p_1 w_{11} + \cdots + p_n w_{1n})&\quad 0&\quad \phantom {.}\cdots \phantom {.}&\quad 0 \\ 0&\quad 2 (p_1 w_{21} + \cdots p_n w_{2n})&\quad \cdots&\quad 0 \\ \vdots&\quad \vdots&\quad \ddots&\quad \vdots \\ 0&\quad 0&\quad \cdots&\quad 2 (p_1 w_{n1} + \cdots + p_n w_{nn}) \phantom {.} \\ \end{bmatrix} \end{aligned}$$

This is a diagonal matrix, so its eigenvalues are the diagonal elements, which, given the constraints on the $p_i$ and $w_{ij}$, can be seen to be all necessarily positive. Therefore, the Hessian is positive definite everywhere, and given that a simplex is a convex set, it follows that the function that is minimized is strictly convex, and hence the minimum it reaches is unique. So, at each step toward the fixed point, a unique minimum is reached. As a result, the minimum reached at the fixed point is unique as well.

For clause (iii), finally, note that at the fixed point the function that is being minimized is of the form

$$\begin{aligned} m^+(\mathbf {p}) = \sum _{i=1}^n \sum _{j=1}^n p_i w_{ij} (\delta _{ij} - p_j)^2. \end{aligned}$$

Because the fixed point $\mathbf {p}^+$ is a minimum, it holds that $\nabla m^+(\mathbf {p}^+)=\mathbf {0}$. We obtain a system of n polynomial equations with n variables and with the $w_{ij}$ as coefficients by setting $(\partial / \partial p_i)m^+(\mathbf {p}^+)=0$, for all $i\leqslant n$. This system has a unique solution (in virtue of the first two clauses), which is bound to be strictly in terms of the coefficients. $\square$