Judgement and ranking: living with hidden bias

Abstract

The complexity and subjectivity of the judgement task conceals the existence of biases that undermines the quality of the process. This paper presents a weighted aggregation function that attempts to reduce the influence of biased judgements on the final score. We also discuss a set of desirable properties. The proposed weighted aggregation function is able to correct the “nationalism bias” found by Emerson et al. (Am Stat 63(2):124–131, 2009) in the 2000 Olympic Games diving competition and suggest the possibility of a “reputation bias”. Our results can be applied to judgement sports and other activities that require the aggregation of several personal evaluations.

Appendix

Proof of Property 1

In order to show the first statement just note that if $s_{i,j}-\overline{s} _{i}=\overline{s}_{i}-s_{i,k}$ by the definition of absolute value function we must have $\left| s_{i,j}-\overline{s}_{i}\right| ^{\alpha }=\left| s_{i,k}-\overline{s}_{i}\right| ^{\alpha }$. Since all other terms in $w_{i,j}$ and $w_{i,k}$ are similar, we must have $w_{i,j}=w_{i,k}.$ The second statement implies that if for every $s_{i,j}$ exist another $s_{i,k}$ at the same distance from the mean, say $s_{i,j}- \overline{s}_{i}=\overline{s}_{i}-s_{i,n+1-j}$ which can be rewritten as $s_{i,j}+s_{i,n+1-j}=2\overline{s}_{i},$ then by the first statement $w_{i,j}=w_{i,n+1-j},$ and we have:

$$\begin{aligned} \overline{s}_{i}^{w}=\sum \limits _{j=1}^{n}w_{i,j}s_{i,j}=\sum \limits _{j=1}^{n/2}w_{i,j}(s_{i,j}+s_{i,n+1-j})=\sum \limits _{j=1}^{n/2}2w_{i,j}\overline{s}_{i}=\overline{s}_{i}, \end{aligned}$$

because $\sum \nolimits _{j=1}^{n/2}w_{i,j}=1/2$. In the case that the total number of grades is odd, then there must exist a odd number of grades equal to the mean in order to exist symmetry. $\square$

Proof of Property 2

In order to show scaling and translation invariance, apply the affine transformation function $\phi (s_{i,j})=a+bs_{i,j},$ where $a\in \mathbb {R}$ and $b\in \mathbb {R} _{+},$ to each grade $s_{i,j}$ in the weight function $w_{i,j},$ given by expression (1), to obtain:

$$\begin{aligned} w_{i,j}(\phi (\mathbf {s}_{i}))= & {} \frac{\sum \nolimits _{l\ne j}^{n}\left| a+bs_{i,l}-\frac{1}{n}\sum \nolimits _{k=1}^{n}(a+bs_{i,k}) \right| ^{\alpha }}{(n-1)\sum \nolimits _{l=1}^{n}\left| a+bs_{i,l}- \frac{1}{n}\sum \nolimits _{k=1}^{n}(a+bs_{i,k})\right| ^{\alpha }} \\= & {} \frac{\sum \nolimits _{l\ne j}^{n}\left| a+bs_{i,l}-a-\frac{1}{n} b\sum \nolimits _{k=1}^{n}s_{i,k}\right| ^{\alpha }}{(n-1)\sum \nolimits _{l=1}^{n}\left| a+bs_{i,l}-a-\frac{1}{n}b\sum \nolimits _{k=1}^{n}s_{i,k}\right| ^{\alpha }} \\= & {} \frac{\sum \nolimits _{l\ne j}^{n}\left| s_{i,l}-\overline{s} _{i}\right| ^{\alpha }}{(n-1)\sum \nolimits _{l=1}^{n}\left| s_{i,l}- \overline{s}_{i}\right| ^{\alpha }}=w_{i,j}(\mathbf {s}_{i}), \end{aligned}$$

for all $i\in I$ and $j\in J$. $\square$

Proof of Property 3

In order to show scale-consistency, apply the affine transformation function $\phi (s_{i,j})=a+bs_{i,j},$ where $a\in \mathbb {R}$ and $b\in \mathbb {R} _{+},$ to each grade $s_{i,j}$ in the WAF $\overline{s}_{i}^{w}(\mathbf {s} _{i}),$ given in expression (2). Since by Property 2 weights are scaling and translation invariant it is enough to show that:

$$\begin{aligned} \overline{s}_{i}^{w}(\phi (\mathbf {s}_{i}))=\sum \limits _{j=1}^{n}w_{i,j}(a+bs_{i,j})=a+b\sum \limits _{k=1}^{n}w_{i,j}s_{i,j}=\phi (\overline{s}_{i}^{w}(\mathbf {s} _{i})), \end{aligned}$$

because $\sum \nolimits _{j=1}^{n}w_{i,j}=1$ for all $i\in I$. $\square$

Proof of Property 4

We start by considering the second statement of Property 4, i.e., the case in which there is only one grade different from $s_{i,j}$.¹⁴ We have two different scenarios. (1) In the first scenario, $n-1$ judges coincide but judge j proposes something different. Let $s_{i,k}=s_{i,m}$ for all $k\ne j\in J,$ then, we can write $\sum \nolimits _{l\ne j}^{n}\left| s_{i,l}-\overline{s}_{i}\right| ^{\alpha }=(n-1)\left| s_{i,m}-\overline{s}_{i}\right| ^{\alpha }$. After having replaced it in expression (1) we obtain:

$$\begin{aligned} w_{i,j}=\frac{(n-1)\left| s_{i,m}-\overline{s}_{i}\right| ^{\alpha } }{(n-1)\left( \left| s_{i,j}-\overline{s}_{i}\right| ^{\alpha }+(n-1)\left| s_{i,m}-\overline{s}_{i}\right| ^{\alpha }\right) }=\frac{1}{ (n-1)^{\alpha }+(n-1)}\equiv w^{\min }, \end{aligned}$$

(4)

where in the last equality we made use of the fact that in this case $|s_{i,j}-\overline{s}_{i}|=(n-1)|s_{i,m}-\overline{s}_{i}|$ for $n>1$. Consequently, the weight given to judge j is constant at some minimal value $w^{\min }$. (2) In the second scenario, $n-2$ judges coincide with judge j but some other judge k proposes something different. Consequently, judge j grade receives a larger weight than $w^{\min }$ because she is in the group of judges that is likely to be correct, i.e., we have $w_{i,j}=(1-w_{i,k})/(n-1)=(1-w^{\min })w^{\max }$.¹⁵ In both cases $w_{i,j}$ is constant with $s_{i,j}$.

Now, we consider the case in which there are at least two grades different from $s_{i,j},$ i.e., the first statement of Property 4. In this case $w_{i,j}$ is given by expression (1). In order to reduce the size of the expression of the derivative $\partial w_{i,j}/\partial s_{i,j},$ let $z_{i,j}=sgn[s_{i,j}-\overline{s}_{i}]\in \{-1,0,1\}$ denote the sign function where $s_{i,j}>\overline{s}_{i}$ implies $z_{i,j}=1,$ $s_{i,j}=\overline{s}_{i}$ implies $z_{i,j}=0$ while $s_{i,j}<\overline{s}_{i}$ implies $z_{i,j}=-1.$ Then, differentiate $w_{i,j}$ with respect to $s_{i,j}$ to obtain:

$$\begin{aligned}&\frac{\partial w_{i,j}}{\partial s_{i,j}}\nonumber \\&\quad {=}\frac{-\alpha \frac{1}{n} \left| s_{i,j}{-}\overline{s}_{i}\right| ^{\alpha }\sum \nolimits _{l\ne j}^{n}z_{i,l}\left| s_{i,l}{-}\overline{s} _{i}\right| ^{\alpha -1}{-}\alpha \left( 1{-}\frac{1}{n}\right) z_{i,j}\left| s_{i,j}{-}\overline{s}_{i}\right| ^{\alpha -1}\sum \nolimits _{l\ne j}^{n}\left| s_{i,l}{-}\overline{s}_{i}\right| ^{\alpha }}{(n{-}1)\left( \sum \nolimits _{l=1}^{n}\left| s_{i,l}{-}\overline{s }_{i}\right| ^{\alpha }\right) ^{2}},\nonumber \\ \end{aligned}$$

(5)

for all $i\in I$ and $j\in J,$ where we made use of the fact that $\sum \nolimits _{l=1}^{n}\left| s_{i,l}-\overline{s}_{i}\right| ^{\alpha }=\left| s_{i,j}-\overline{s}_{i}\right| ^{\alpha }+\sum \nolimits _{l\ne j}^{n}\left| s_{i,l}-\overline{s}_{i}\right| ^{\alpha }$ and that the derivative of $\sum \nolimits _{l=1}^{n}\left| s_{i,l}-\overline{s}_{i}\right| ^{\alpha }$ with respect to $s_{i,j}$ is given by $\alpha z_{i,j}(1-1/n)\left| s_{i,j}-\overline{s} _{i}\right| ^{\alpha -1}-\alpha (1/n)\sum \nolimits _{l\ne j}^{n}z_{i,l}\left| s_{i,l}-\overline{s}_{i}\right| ^{\alpha -1}$. We want to show that $\partial w_{i,j}/\partial s_{i,j}>0$ if $s_{i,j}< \overline{s}_{i}$ ($z_{i,j}<0$) and $\partial w_{i,j}/\partial s_{i,j}<0$ otherwise. Since the denominator is strictly positive the sign of the derivative is given by the numerator. We have four cases to consider that depend on whether $z_{i,j}$ and $\sum \nolimits _{l\ne j}^{n}z_{i,l}\left| s_{i,l}-\overline{s}_{i}\right| ^{\alpha -1}$ are negative or positive. If $z_{i,j}>0$ and $\sum \nolimits _{l\ne j}^{n}z_{i,l}\left| s_{i,l}-\overline{s}_{i}\right| ^{\alpha -1}>0,$ it is immediate that $\partial w_{i,j}/\partial s_{i,j}<0,$ while if $z_{i,j}<0$ and $\sum \nolimits _{l\ne j}^{n}z_{i,l}\left| s_{i,l}- \overline{s}_{i}\right| ^{\alpha -1}<0,$ it is immediate that $\partial w_{i,j}/\partial s_{i,j}>0$. In the other two cases, if $z_{i,j}>0$ and $\sum \nolimits _{l\ne j}^{n}z_{i,l}\left| s_{i,l}-\overline{s} _{i}\right| ^{\alpha -1}<0,$ in order to show that $\partial w_{i,j}/\partial s_{i,j}<0$ we must consider the scenario that makes it more difficult to satisfy, i.e., when $z_{i,l}=-1$ for all $l\ne j$. Similarly, if $z_{i,j}<0$ and $\sum \nolimits _{l\ne j}^{n}z_{i,l}\left| s_{i,l}- \overline{s}_{i}\right| ^{\alpha -1}>0,$ in order to show that $\partial w_{i,j}/\partial s_{i,j}>0$ we must consider the scenario that makes it more difficult to satisfy, i.e., when $z_{i,l}=1$ for all $l\ne j$. In both cases we are left to show that the same inequality is true:

$$\begin{aligned} \left( n-1\right) \sum \limits _{l\ne j}^{n}\left| s_{i,l}-\overline{s} _{i}\right| ^{\alpha }>\left| s_{i,j}-\overline{s}_{i}\right| \sum \limits _{l\ne j}^{n}\left| s_{i,l}-\overline{s}_{i}\right| ^{\alpha -1}, \end{aligned}$$

for all $i\in I$ and $j\in J$. Note that for the case $s_{i,j}=\overline{s} _{i}$ the inequality holds trivially. Since the WAF is scale-consistent (Property 3) we can normalize $\overline{s}_{i}=0$ without loss of generality and for simplicity consider the case of two different grades other than $s_{i,j},$ i.e., $s_{i,j}<\overline{s}_{i}=0<s_{i,k},s_{i,l}$ (the other case $s_{i,j}>\overline{s}_{i}=0>s_{i,k},s_{i,l}$ follows the same argument). Then, the above inequality becomes:

$$\begin{aligned} 2\left( \left| s_{i,k}\right| ^{\alpha }+\left| s_{i,l}\right| ^{\alpha }\right) >\left( \left| s_{i,k}+s_{i,l}\right| \right) \left( \left| s_{i,k}\right| ^{\alpha -1}+\left| s_{i,l}\right| ^{\alpha -1}\right) , \end{aligned}$$

where we have used the fact that $\overline{s}_{i}=0$ implies that $-s_{i,j}=s_{i,k}+s_{i,l}$. Since all quantities are non-negative we can remove the absolute value function from the previous inequality to obtain $(s_{i,k}{}^{\alpha -1}-s_{i,l}{}^{\alpha -1})(s_{i,k}{}-s_{i,l}{})>0$ which is strictly positive for all $s_{i,k}{}\ne s_{i,l}$ with $k\ne l\in J$. $\square$

Proof of Property 5

We start by considering the limit $\alpha \rightarrow 0.$ Suppose that there are $t=0,1,\ldots ,n-1$ grades such that $s_{i,k}=\overline{s}_{i},$ denoted with the subindex $k\in K\subset J$ (the case $t=n$ is trivially true that $\overline{s}_{i}^{w}=\overline{s}_{i}$), and $n-t$ grades such that $s_{i,j}\ne \overline{s}_{i},$ denoted with the subindex $j\in J\backslash K$. Then, if $\alpha \rightarrow 0$ we have $\left| s_{i,j}- \overline{s}_{i}\right| ^{\alpha }\rightarrow 1$ for all $j\in J\backslash K,$ and $\left| s_{i,k}-\overline{s}_{i}\right| ^{\alpha }\rightarrow 0$ for all $k\in K$. Consequently, $\sum \nolimits _{l\ne j}^{n}\left| s_{i,l}-\overline{s}_{i}\right| ^{\alpha }\rightarrow n-1-t,$ $\sum \nolimits _{l\ne k}^{n}\left| s_{i,l}-\overline{s} _{i}\right| ^{\alpha }\rightarrow n-t$ and $\sum \nolimits _{l=1}^{n} \left| s_{i,l}-\overline{s}_{i}\right| ^{\alpha }\rightarrow n-t,$ which implies that $w_{i,j}\rightarrow (n-1-t)/((n-t)(n-1))$ for all $j\in J\backslash K$ and $w_{i,k}\rightarrow 1/(n-1)$ for all $k\in K$. Therefore, we can write $\overline{s}_{i}^{w}\rightarrow \sum \nolimits _{j\notin K}(n-1-t)s_{i,j}/((n-t)(n-1))+ts_{i,k}/(n-1),$ where $s_{i,k}=\overline{s} _{i}=\sum \nolimits _{j=1}^{n}s_{i,j}/n,$ which implies that $s_{i,k}= \overline{s}_{i}=\sum \nolimits _{j\notin K}s_{i,j}/(n-t)$. Replacing the latter two equalities into $\overline{s}_{i}^{w},$ after some algebra, we obtain that $\overline{s}_{i}^{w}\rightarrow \overline{s}_{i}$.

Now, consider the limit $\alpha \rightarrow \infty$. In this case we can rewrite $w_{i,j}$ as $w_{i,j}=1/((n-1)(1+\left| s_{i,j}-\overline{s} _{i}\right| ^{\alpha }/\sum \nolimits _{l\ne j}^{n}\left| s_{i,l}- \overline{s}_{i}\right| ^{\alpha })),$ for all $j\in J$. Suppose that there are $r=1,\ldots ,\lfloor (n-1)/2\rfloor$ extreme grades (not necessarily equal but at the same distance from the mean), where $\lfloor x\rfloor$ denotes floor function—the largest integer less than or equal to x (note: we cannot have more than $\lfloor (n-1)/2\rfloor$ extreme grades, otherwise, they will not be the extreme grades). There are two different situations to consider. (1) If $\left| s_{i,j}-\overline{s} _{i}\right|$ is one of the largest grade differences with respect to the mean (i.e., $s_{i,j}$ is an extreme grade) and $\alpha \rightarrow \infty$, then $\left| s_{i,j}-\overline{s}_{i}\right| ^{\alpha }/\sum \nolimits _{l\ne j}^{n}\left| s_{i,l}-\overline{s}_{i}\right| ^{\alpha }\rightarrow 1/(r-1)$ and

$$\begin{aligned} w_{i,j}\rightarrow 1/((n-1)(1+1/(r-1)))=(r-1)/((n-1)r). \end{aligned}$$

Note that in the particular case $r=1$ we have $w_{i,j}\rightarrow 0$. (2) If $\left| s_{i,j}-\overline{s}_{i}\right|$ is not one of the largest grade differences with respect to the mean (i.e., $s_{i,j}$ is not an extreme grade) and $\alpha \rightarrow \infty$, then $\left| s_{i,j}- \overline{s}_{i}\right| ^{\alpha }/\sum \nolimits _{l\ne j}^{n}\left| s_{i,l}-\overline{s}_{i}\right| ^{\alpha }\rightarrow 0$ and $w_{i,j}\rightarrow 1/(n-1)$ for all $j\ne k\in J$. Altogether, we have $n-r$ non-extreme grades, each weighted by $1/(n-1)$ and r extreme grades, each weighted by $(r-1)/((n-1)r)$. Let the r extreme grades be indexed as $j=n-r+1,\ldots ,n,$ then $\overline{s}_{i}^{w}\rightarrow (\sum \nolimits _{j=1}^{n-r}s_{i,j}+(r-1)\sum \nolimits _{j=n-r+1}^{n}s_{i,j}/r)/(n-1)$. After adding and subtracting $\sum \nolimits _{j=n-r+1}^{n}s_{i,j}$ in the numerator we obtain that $\overline{s}_{i}^{w}\rightarrow (\sum \nolimits _{j=1}^{n}s_{i,j}-\sum \nolimits _{j=n-r+1}^{n}s_{i,j}/r)/(n-1),$ where $\sum \nolimits _{j=n-r+1}^{n}s_{i,j}/r$ is the average extreme grade $\overline{s}_{i,e}$. $\square$

Proof of Property 6

We start by showing the first statement of Property 6, i.e., $\overline{s}_{i}^{w}$ is monotonic in $s_{i,j}$ for small $\alpha$. In order to do it we proceed as follows. Since $\alpha \in [0,\infty )$ we will show that $\overline{s}_{i}^{w}$ is monotonic increasing in $s_{i,j}$ for $\alpha$ small, i.e., in the zero neighborhood. Then, through a numerical example, we show that for sufficiently large $\alpha$ the monotonic relation between $\overline{s}_{i}^{w}$ and $s_{i,j}$ is not guaranteed. Note that the WAF defined in (2) can be written as (see Footnote 8):

$$\begin{aligned} \overline{s}_{i}^{w}=\left( \sum \limits _{l=1}^{n}\left| s_{i,l}- \overline{s}_{i}\right| ^{\alpha }\sum \limits _{k\ne l}^{n}s_{i,k}\right) /\left( (n-1)\sum \limits _{l=1}^{n}\left| s_{i,l}- \overline{s}_{i}\right| ^{\alpha }\right) =N/D, \end{aligned}$$

for all $i\in I$ and $j\in J,$ where N and D denote the expressions in the numerator and denominator, respectively. Recall that the derivative of $\overline{s}_{i}^{w}$ with respect to $s_{i,j}$ is given by $\partial \overline{s}_{i}^{w}/\partial s_{i,j}=(N^{\prime }D-ND^{\prime })/D^{2}$. In the case that $s_{i,j}\ne \overline{s}_{i}$ for all $j\in J$ (the case that $s_{i,k}=\overline{s}_{i}$ for some $k\in J$ follows the same argument) the derivative of the expression in the denominator with respect to $s_{i,j}$ is equal to the numerator of expression (5) multiplied by $n-1,$ that is:

$$\begin{aligned} D^{\prime }=(n-1)\alpha z_{i,j}(1-1/n)\left| s_{i,j}-\overline{s} _{i}\right| ^{\alpha -1}-(n-1)\alpha (1/n)\sum \limits _{l\ne j}^{n}z_{i,l}\left| s_{i,l}-\overline{s}_{i}\right| ^{\alpha -1}, \end{aligned}$$

where we use the same notation as in the Proof of Property 4. While the expression in the numerator can be written as $N=\left| s_{i,j}- \overline{s}_{i}\right| ^{\alpha }\sum \nolimits _{k\ne j}^{n}s_{i,k}+\sum \nolimits _{l\ne j}^{n}\left| s_{i,l}-\overline{s} _{i}\right| ^{\alpha }\sum \nolimits _{k\ne l}^{n}s_{i,k}$. The derivative of this expression with respect to $s_{i,j}$ is given by:

$$\begin{aligned} N^{\prime }= & {} \alpha z_{i,j}(1-1/n)\left| s_{i,j}-\overline{s} _{i}\right| ^{\alpha -1}\sum \limits _{k\ne j}^{n}s_{i,k} \\&-\alpha (1/n)\sum \limits _{l\ne j}^{n}z_{i,l}\left| s_{i,l}- \overline{s}_{i}\right| ^{\alpha -1}\sum \limits _{k\ne l}^{n}s_{i,k}+\sum \limits _{l\ne j}^{n}\left| s_{i,l}-\overline{s} _{i}\right| ^{\alpha }. \end{aligned}$$

In the limit $\alpha \rightarrow 0,$ we have $N\rightarrow (n-1)\sum \nolimits _{l=1}^{n}s_{i,l},$ $D\rightarrow (n-1)n,$ $N^{\prime }\rightarrow n-1$ and $D^{\prime }\rightarrow 0$. Therefore, $\partial \overline{s}_{i}^{w}/\partial s_{i,j}\rightarrow 1/n$ for $\alpha \rightarrow 0$ which is strictly positive for any profile of grades. For large values of $\alpha$ we show through a numerical example the failure of monotonicity. Column (6) of Table 1 shows that for $\alpha =3$ when the grade of the first judge raises from $s_{i,1}=8$ to $s_{i,1}=9$ the WAF falls from $\overline{s}_{i}^{w}=7.314$ to $\overline{s}_{i}^{w}=7.303,$ showing that monotonicity is not guaranteed for large $\alpha$.

Now, we consider the second statement of Property 6. The proof of monotonicity for $s_{i,j}$ around the mean follows a similar strategy. We evaluate $\partial \overline{s}_{i}^{w}/\partial s_{i,j}$ in the neighborhood of $\overline{s}_{i},$ i.e., for $s_{i,j}\rightarrow \overline{s }_{i}$ which is equivalent to $s_{i,j}\rightarrow \sum \nolimits _{l\ne j}^{n}s_{i,l}/(n-1)$ (the other judges mean). In this case we have $\left| s_{i,j}-\overline{s}_{i}\right| ^{\alpha }\rightarrow 0,$ and consequently, $N\rightarrow \sum \nolimits _{l\ne j}^{n}\left| s_{i,l}- \overline{s}_{i}\right| ^{\alpha }\sum \nolimits _{k\ne l}^{n}s_{i,k}$, $D\rightarrow (n-1)\sum \nolimits _{l\ne j}^{n}\left| s_{i,l}-\overline{s} _{i}\right| ^{\alpha },$

$$\begin{aligned} N^{\prime }\rightarrow -\alpha (1/n)\sum \limits _{l\ne j}^{n}z_{i,l}\left| s_{i,l}-\overline{s}_{i}\right| ^{\alpha -1}\sum \limits _{k\ne l}^{n}s_{i,k}+\sum \limits _{l\ne j}^{n}\left| s_{i,l}-\overline{s}_{i}\right| ^{\alpha }, \end{aligned}$$

and $D^{\prime }\rightarrow -(n-1)\alpha (1/n)\sum \nolimits _{l\ne j}^{n}z_{i,l}\left| s_{i,l}-\overline{s}_{i}\right| ^{\alpha -1}$. At this stage the expression for $\partial \overline{s}_{i}^{w}/\partial s_{i,j}$ is particularly large. Since the WAF is scale-consistent (Property 3) we can normalize $\overline{s}_{i}=0$ without loss of generality, which implies that $s_{i,j}\rightarrow 0$ and we can write $\sum \nolimits _{k\ne l}^{n}s_{i,k}=-s_{i,l}$. Then, the expression for $\partial \overline{s}_{i}^{w}/\partial s_{i,j}$ converges to:

$$\begin{aligned} \frac{\left( {-}\alpha \frac{1}{n}\sum \nolimits _{l\ne j}^{n}z_{i,l}\left| s_{i,l}\right| ^{\alpha {-}1}({-}s_{i,l}){+}\sum \nolimits _{l\ne j}^{n}\left| s_{i,l}\right| ^{\alpha }\right) \sum \nolimits _{l\ne j}^{n}\left| s_{i,l}\right| ^{\alpha }{+}\alpha \frac{1}{n} \sum \nolimits _{l\ne j}^{n}\left| s_{i,l}\right| ^{\alpha }({-}s_{i,l})\sum \nolimits _{l\ne j}^{n}z_{i,l}\left| s_{i,l}\right| ^{\alpha {-}1}}{(n{-}1)\left( \sum \nolimits _{l\ne j}^{n}\left| s_{i,l}\right| ^{\alpha }\right) ^{2}}, \end{aligned}$$

(6)

for all $i\in I$ and $j\in J$. Note that $s_{i,l}>0$ is equivalent to $z_{i,l}>0$ (positive and above the mean) and implies that $z_{i,l}\left| s_{i,l}\right| ^{\alpha -1}(-s_{i,l})=-\left| s_{i,l}\right| ^{\alpha },$ $\left| s_{i,l}\right| ^{\alpha }(-s_{i,l})=-\left| s_{i,l}\right| ^{\alpha +1}$ and $z_{i,l}\left| s_{i,l}\right| ^{\alpha -1}=\left| s_{i,l}\right| ^{\alpha -1},$ while $s_{i,l}<0$ is equivalent to $z_{i,l}<0$ (negative and below the mean) which implies that $z_{i,l}\left| s_{i,l}\right| ^{\alpha -1}(-s_{i,l})=-\left| s_{i,l}\right| ^{\alpha }$, $\left| s_{i,l}\right| ^{\alpha }(-s_{i,l})=\left| s_{i,l}\right| ^{\alpha +1}$ and $z_{i,l}\left| s_{i,l}\right| ^{\alpha -1}=-\left| s_{i,l}\right| ^{\alpha -1}$. Therefore, we have $\sum \nolimits _{l\ne j}^{n}z_{i,l}\left| s_{i,l}\right| ^{\alpha -1}(-s_{i,l})=\sum \nolimits _{l\ne j}^{n}\left| s_{i,l}\right| ^{\alpha }$, $\sum \nolimits _{l\ne j}^{n}\left| s_{i,l}\right| ^{\alpha }(-s_{i,l})=-\sum \nolimits _{l\ne j\wedge +}^{n}\left| s_{i,l}\right| ^{\alpha +1}+\sum \nolimits _{l\ne j\wedge -}^{n}\left| s_{i,l}\right| ^{\alpha +1}$ and $\sum \nolimits _{l\ne j}^{n}z_{i,l}\left| s_{i,l}\right| ^{\alpha -1}=\sum \nolimits _{l\ne j\wedge +}^{n}\left| s_{i,l}\right| ^{\alpha -1}-\sum \nolimits _{l\ne j\wedge -}^{n}\left| s_{i,l}\right| ^{\alpha -1},$ where the indices of summation $l\ne j\wedge +$ and $l\ne j\wedge -$ denote the summation over the positive and the negative terms different from j,  respectively. Since the denominator of expression (6) is strictly positive the numerator determines the sign of $\partial \overline{s}_{i}^{w}/\partial s_{i,j}$. After having replaced these equalities, the numerator of expression (6) becomes:

$$\begin{aligned}&\left( \alpha \frac{1}{n}\sum \limits _{l\ne j}^{n}\left| s_{i,l}\right| ^{\alpha }+\sum \limits _{l\ne j}^{n}\left| s_{i,l}\right| ^{\alpha }\right) \sum \limits _{l\ne j}^{n}\left| s_{i,l}\right| ^{\alpha } \\&\quad +\alpha \frac{1}{n}\left( -\sum \limits _{l\ne j\wedge +}^{n}\left| s_{i,l}\right| ^{\alpha +1}+\sum \limits _{l\ne j\wedge -}^{n}\left| s_{i,l}\right| ^{\alpha +1}\right) \\&\quad \times \,\left( \sum \limits _{l\ne j\wedge +}^{n}\left| s_{i,l}\right| ^{\alpha -1}-\sum \limits _{l\ne j\wedge -}^{n}\left| s_{i,l}\right| ^{\alpha -1}\right) , \end{aligned}$$

for all $i\in I$ and $j\in J$. The component in the first line is strictly positive. Therefore, the only way for this expression to be negative is when the term in the second line is sufficiently negative. We show that such is impossible. Consider the worst case scenario in which there is only one large positive term and many small negative terms such that $\sum \nolimits _{l\ne j\wedge +}^{n}\left| s_{i,l}\right| ^{\alpha +1}=\left| s_{i,l}\right| ^{\alpha +1}>0$ and $\sum \nolimits _{l\ne j\wedge -}^{n}\left| s_{i,l}\right| ^{\alpha +1}\approx 0,$ respectively. The objective of this assumption is to obtain the largest negative term in the second line. In this case we obtain that:

$$\begin{aligned} \frac{\partial \overline{s}_{i}^{w}}{\partial s_{i,j}}\rightarrow & {} \frac{ \left( \alpha \frac{1}{n}\left| s_{i,l}\right| ^{\alpha }+\left| s_{i,l}\right| ^{\alpha }\right) \left| s_{i,l}\right| ^{\alpha }-\alpha \frac{1}{n}\left| s_{i,l}\right| ^{\alpha +1}\left| s_{i,l}\right| ^{\alpha -1}}{(n-1)(\left| s_{i,l}\right| ^{\alpha })^{2}} \\= & {} \frac{\left( \alpha \frac{1}{n}\left| s_{i,l}\right| ^{2\alpha }+\left| s_{i,l}\right| ^{2\alpha }\right) -\alpha \frac{1}{n}\left| s_{i,l}\right| ^{2\alpha }}{(n-1)(\left| s_{i,l}\right| ^{\alpha })^{2}}=\frac{1}{(n-1)}, \end{aligned}$$

for all $i\in I$ and $j\in J,$ which is strictly positive for any profile of grades. The symmetric worst case scenario with a single negative term and many small positive terms follows the same argument. For values of $s_{i,j}$ distant from the other judges mean we show through a numerical example the failure of monotonicity. Column (6) of Table 1 shows that while the grade of the first judge moves from $s_{i,1}=8$ to $s_{i,1}=9,$ i.e., it gets more distant from the other judges mean $\sum \nolimits _{l=2}^{n}s_{i,l}/4=(6+7+7+8)/4=7,$ the WAF falls from $\overline{s }_{i}^{w}=7.314$ to $\overline{s}_{i}^{w}=7.303,$ showing that monotonicity is not guaranteed for $s_{i,1}$ sufficiently away from the other judges mean.

Finally, following Property 4, if there exist only one grade different than $s_{i,j}$ then we can construct a situation in which occurs a downward discontinuity in the weight of judge j after an increase in her grade by an infinitesimal amount. In order to complete the proof we must verify this case. In order to construct such case assume that initially all judges (including judge j) are awarding the mean grade $s_{i,j}=\overline{s }_{i}=s_{i,m}$. Consequently, all grades are equally weighted by 1 / n and $\overline{s}_{i}^{w}=\overline{s}_{i}=s_{i,m}$. Now, suppose that judge j increase her grade from $s_{i,m}$ to $s_{i,j}>s_{i,m}$. In this case she departs from the other $n-1$ judges and the weight $w_{i,j}$ given to her grade falls discontinuously from 1 / n to $w^{\min }=1/((n-1)+(n-1)^{\alpha })$ [see scenario (i) in the proof of Property 4] while the weight of the other $n-1$ judges jumps from 1 / n to $(1-w^{\min })/(n-1)$. Therefore, in the discontinuous case, the difference between the new WAF $\overline{s}_{i}^{w}$ and the initial WAF $\overline{s}_{i}^{w}=s_{i,m}$ is given by:

$$\begin{aligned} \overline{s}_{i}^{w}-s_{i,m}= & {} \frac{1}{(n-1)+(n-1)^{\alpha }} s_{i,j}+\left( 1-\frac{1}{(n-1)+(n-1)^{\alpha }}\right) s_{i,m}-s_{i,m} \\= & {} \frac{s_{i,j}-s_{i,m}}{(n-1)+(n-1)^{\alpha }}, \end{aligned}$$

which is strictly positive for all $s_{i,j}>s_{i,m}$. The proof of the case $s_{i,j}<\overline{s}_{i}$ follows from symmetry (Property 1). $\square$

Proof of Property 7

If the distribution of the grades has positive (negatively, respectively) $\alpha +1$ -th absolute central moment, defined as $\sum \nolimits _{k=1}^{n}\left| s_{i,k}-\overline{s}_{i}\right| ^{\alpha }(s_{i,j}-\overline{s}_{i})/n$ (where if $\alpha >0$ is even and integer we have the usual $\alpha +1$-th central moment), then $\overline{s} _{i}^{w}<\overline{s}_{i}$ ($\overline{s}_{i}^{w}>\overline{s}_{i},$ respectively). In order to show the relation between the $\alpha +1$-th absolute central moment, the skewness and the WAF manipulate the inequality $\overline{s}_{i}^{w}<\overline{s}_{i}$ to obtain:

$$\begin{aligned} \overline{s}_{i}>\overline{s}_{i}^{w}&=\sum \limits _{j=1}^{n}\frac{ \sum \nolimits _{l\ne j}^{n}\left| s_{i,l}-\overline{s}_{i}\right| ^{\alpha }}{(n-1)\sum \nolimits _{l=1}^{n}\left| s_{i,l}-\overline{s} _{i}\right| ^{\alpha }}s_{i,j} \\&\Leftrightarrow 0>\sum \limits _{j=1}^{n}\frac{\sum \nolimits _{l\ne j}^{n}\left| s_{i,l}-\overline{s}_{i}\right| ^{\alpha }+\left| s_{i,j}-\overline{s}_{i}\right| ^{\alpha }-\left| s_{i,j}-\overline{s }_{i}\right| ^{\alpha }}{(n-1)\sum \nolimits _{l=1}^{n}\left| s_{i,l}- \overline{s}_{i}\right| ^{\alpha }}(s_{i,j}-\overline{s}_{i}) \\&\Leftrightarrow \sum \limits _{j=1}^{n}\left| s_{i,j}-\overline{s} _{i}\right| ^{\alpha }(s_{i,j}-\overline{s}_{i})/\left( (n-1)\sum \limits _{l=1}^{n}\left| s_{i,l}-\overline{s}_{i}\right| ^{\alpha }\right) >0, \end{aligned}$$

for all $i\in I$. The sign of the expression in the numerator (i.e., the $\alpha +1$-th absolute central moment) determines the skewness, which is positive if $\overline{s}_{i}^{w}<\overline{s}_{i}$. The case $\overline{s} _{i}<\overline{s}_{i}^{w}$ follows the same argument. $\square$