Ordinal response variation of the polytomous Rasch model

Abstract

Polytomous Rasch model (PRM) is a general probabilistic measurement model widely used in psychometrics, social science and educational measurement. It describes the probability of certain ordinal response of an object under test as a function of its ability, given, so called thresholds, characterizing the specific test item. The model was also adapted to business and industry applications. In contrast to the behavior of the median PRM outcome value, monotonically increasing as the ability increases, the ordinal variation behavior, as shown in the article, can be very diverse and it is rather determined by the mutual position of the threshold values of the model. The article studies ordinal variation of the response vs. ability for different thresholds locations arrangements and different amounts of ordered response categories. It is shown under what circumstances this function becomes multimodal. If several objects are involved in the test, attention is paid to the possibility of the total variation decomposition into intra and inter components. Considering the intra object variation helps to avoid overestimation of the real variation between the tested objects as it is demonstrated by illustrative example.

Appendices

Appendix 1: Singular case $\tau_{i} = 0,\,\,i = 1,...,m - 1$

In this appendix, we analyze the case where all thresholds are equal to zero.

Due to (10), the probabilities are

$$ p_{i} (a) = \frac{{e^{{ia - \sum\nolimits_{k = 0}^{i} {\tau_{i} } }} }}{{\sum\nolimits_{j = 0}^{m - 1} {e^{{ja - \sum\nolimits_{k = 0}^{i} {\tau_{i} } }} } }},{\mkern 1mu} {\mkern 1mu} i = 0,1,...,m - 1. $$

For $\tau_{i} \to 0,\,\,\,i = 1,...,m - 1$:

$$ p_{i} (a) \to \frac{{e^{ia} }}{{\sum\limits_{j = 0}^{m - 1} {e^{ja} } }} = \frac{{e^{ia} (1 - e^{a} )}}{{1 - e^{ma} }},\,\,\,\,i = 0,1,...,m - 1, $$

i.e., the probabilities form a geometric progression of common ratio $e^{a}$.

In particular, for $a = 0$ by L’Hôpital rule:

$$ \mathop {\lim }\limits_{a \to 0} \frac{{e^{ia} (1 - e^{a} )}}{{1 - e^{ma} }} = \mathop {\lim }\limits_{a \to 0} \frac{{e^{ia} - e^{(i + 1)a} }}{{1 - e^{ma} }} = \mathop {\lim }\limits_{a \to 0} \frac{{ie^{ia} - (i + 1)e^{(i + 1)a} }}{{ - me^{ma} }} = \frac{1}{m}. $$

Due to (4), the ordinal variation is

$$ \begin{aligned} {\text{VAR}}(a) & = \frac{4}{m - 1}\left[ {p_{0} (p_{1} + p_{2} + \cdots + p_{m - 1} ) + (p_{0} + p_{1} )(p_{2} + \cdots + p_{m - 1} )} \right. \hfill \\ & + \left. {(p_{0} + p_{1} + p_{2} )(p_{3} + ... + p_{m - 1} ) + \cdots + (p_{0} + p_{2} + \cdots + p_{m - 2} )p_{m - 1} } \right]. \hfill \\ \end{aligned} $$

For $\tau_{i} \to 0,\,\,\,$

$$ \begin{gathered} {\text{VAR}}(a) \to \frac{4}{m - 1}\left[ {\sum\limits_{i = 0}^{0} {\frac{{e^{ia} (1 - e^{a} )}}{{1 - e^{ma} }}} \cdot \sum\limits_{i = 1}^{m - 1} {\frac{{e^{ia} (1 - e^{a} )}}{{1 - e^{ma} }}} + \sum\limits_{i = 0}^{1} {\frac{{e^{ia} (1 - e^{a} )}}{{1 - e^{ma} }}} \cdot \sum\limits_{i = 2}^{m - 1} {\frac{{e^{ia} (1 - e^{a} )}}{{1 - e^{ma} }}} } \right. \hfill \\ \left. { + \sum\limits_{i = 0}^{2} {\frac{{e^{ia} (1 - e^{a} )}}{{1 - e^{ma} }}} \cdot \sum\limits_{i = 3}^{m - 1} {\frac{{e^{ia} (1 - e^{a} )}}{{1 - e^{ma} }}} + ... + \sum\limits_{i = 0}^{m - 2} {\frac{{e^{ia} (1 - e^{a} )}}{{1 - e^{ma} }}} \cdot \sum\limits_{i = m - 1}^{m - 1} {\frac{{e^{ia} (1 - e^{a} )}}{{1 - e^{ma} }}} } \right] \hfill \\ = \frac{4}{ m - 1}\sum\limits_{j = 0}^{m - 2} {\left[ {\sum\limits_{i = 0}^{j} {\frac{{e^{ia} (1 - e^{a} )}}{{1 - e^{ma} }}} \cdot \sum\limits_{i = j + 1}^{m - 1} {\frac{{e^{ia} (1 - e^{a} )}}{{1 - e^{ma} }}} } \right]} = \frac{{4(1 - e^{a} )^{2} }}{{\left( {m - 1} \right)\left( {1 - e^{ma} } \right)^{2} }}\sum\limits_{j = 0}^{m - 2} {\left[ {\sum\limits_{i = 0}^{j} {e^{ia} } \cdot \sum\limits_{i = j + 1}^{m - 1} {e^{ia} } } \right]} \hfill \\ = \frac{{4(1 - e^{a} )^{2} }}{{\left( {m - 1} \right)\left( {1 - e^{ma} } \right)^{2} }}\sum\limits_{j = 0}^{m - 2} {\left[ {\frac{{1 - e^{(j + 1)a} }}{{1 - e^{a} }} \cdot \frac{{e^{(j + 1)a} \left( {1 - e^{(m - j - 1)a} } \right)}}{{1 - e^{a} }}} \right]} \hfill \\ = \frac{4}{{\left( {m - 1} \right)\left( {1 - e^{ma} } \right)^{2} }}\sum\limits_{j = 0}^{m - 2} {\left[ {\left( {1 - e^{(j + 1)a} } \right)e^{(j + 1)a} \left( {1 - e^{(m - j - 1)a} } \right)} \right]} \hfill \\ = \frac{4}{{\left( {m - 1} \right)\left( {1 - e^{ma} } \right)^{2} }}\sum\limits_{j = 0}^{m - 2} {\left[ {e^{(j + 1)a} - e^{2(j + 1)a} - e^{ma} + e^{(m + j + 1)a} } \right]} . \hfill \\ \end{gathered} $$

By using the formula of geometric sequence,

$$ \begin{gathered} \sum\limits_{j = 0}^{m - 2} {e^{(j + 1)a} } = \frac{{e^{a} (1 - e^{(m - 1)a} )}}{{1 - e^{a} }};\;\sum\limits_{j = 0}^{m - 2} {e^{2(j + 1)a} } = \frac{{e^{2a} (1 - e^{2(m - 1)a} )}}{{1 - e^{2a} }}; \hfill \\ \sum\limits_{j = 0}^{m - 2} {e^{ma} = (m - 1)} e^{ma} ;\;\sum\limits_{j = 0}^{m - 2} {e^{(m + j + 1)a} } = \frac{{e^{(m + 1)a} (1 - e^{(m - 1)a} )}}{{1 - e^{a} }}. \hfill \\ \end{gathered} $$

Consequently,

$$ \begin{gathered} \sum\limits_{j = 0}^{m - 2} {\left[ {e^{(j + 1)a} - e^{2(j + 1)a} - e^{ma} + e^{(m + j + 1)a} } \right]} \hfill \\ = \frac{{e^{a} (1 - e^{(m - 1)a} )}}{{1 - e^{a} }} - \frac{{e^{2a} (1 - e^{2(m - 1)a} )}}{{1 - e^{2a} }} - (m - 1)e^{ma} + \frac{{e^{(m + 1)a} (1 - e^{(m - 1)a} )}}{{1 - e^{a} }} \hfill \\ = \frac{{e^{a} - e^{2a} - e^{(2m + 1)a} + e^{(2m + 2)a} - me^{ma} + me^{(m + 1)a} + me^{(m + 2)a} - me^{(m + 3)a} }}{{\left( {1 - e^{a} } \right)\left( {1 - e^{2a} } \right)}}. \hfill \\ \end{gathered} $$

Thus,

$$ {\text{VAR}}(a) \to 4\frac{{e^{a} - e^{2a} - e^{(2m + 1)a} + e^{(2m + 2)a} - me^{ma} + me^{(m + 1)a} + me^{(m + 2)a} - me^{(m + 3)a} }}{{\left( {m - 1} \right)\left( {1 - e^{ma} } \right)^{2} \left( {1 - e^{a} } \right)\left( {1 - e^{2a} } \right)}}. $$

For $a \to 0$, by by L’Hôpital rule:

$$ \mathop {\lim }\limits_{a \to 0} 4\frac{{e^{a} - e^{2a} - e^{(2m + 1)a} + e^{(2m + 2)a} - me^{ma} + me^{(m + 1)a} + me^{(m + 2)a} - me^{(m + 3)a} }}{{\left( {m - 1} \right)\left( {1 - e^{ma} } \right)^{2} \left( {1 - e^{a} } \right)\left( {1 - e^{2a} } \right)}} = \frac{2(m + 1)}{{3m}}. $$

For $a \ne 0$,

$$ \mathop {\lim }\limits_{m \to \infty } 4\frac{{e^{a} - e^{2a} - e^{(2m + 1)a} + e^{(2m + 2)a} - me^{ma} + me^{(m + 1)a} + me^{(m + 2)a} - me^{(m + 3)a} }}{{\left( {m - 1} \right)\left( {1 - e^{ma} } \right)^{2} \left( {1 - e^{a} } \right)\left( {1 - e^{2a} } \right)}} = 0. $$

In the Fig. 8, ${\text{VAR}}(a)$ is shown as a function of $a$ for $m = 200$.

Fig. 8

Ordinal variation for $m = 200$

Appendix 2: Complete analysis of the case $m = 3$

In this appendix, the case of the ternary response is analyzed.

For $m = 3$ and symmetric thresholds $\tau_{1} = - \,\tau$ and $\tau_{2} = + \,\tau$,

$$ p_{0} (a) = \frac{1}{{1 + e^{\tau + a} + e^{2a} }}, $$

$$ p_{1} (a) = \frac{{e^{\tau + a} }}{{1 + e^{\tau + a} + e^{2a} }}, $$

and, consequently

$$ p_{2} (a) = 1 - \left( {p_{0} (a) + p_{1} (a)} \right) = \frac{{e^{2a} }}{{1 + e^{\tau + a} + e^{2a} }}. $$

Negative values of $\tau$ ("disordered thresholds") are allowed.

Thus,

$$ \begin{gathered} F_{0} = p_{0} = \frac{1}{{1 + e^{\tau + a} + e^{2a} }},\;1 - F_{0} = \frac{{e^{\tau + a} + e^{2a} }}{{1 + e^{\tau + a} + e^{2a} }}, \hfill \\ F_{1} = p_{0} + p_{1} = \frac{{1 + e^{\tau + a} }}{{1 + e^{\tau + a} + e^{2a} }},\;1 - F_{1} = \frac{{e^{2a} }}{{1 + e^{\tau + a} + e^{2a} }}. \hfill \\ \end{gathered} $$

Therefore, the ordinal variation is

$$ {\text{VAR}}\left( {a,\tau } \right) = 2\left[ {F_{0} \left( {1 - F_{0} } \right) + F_{1} \left( {1 - F_{1} } \right)} \right] = \frac{{2\left( {e^{\tau + a} + 2e^{2a} + e^{\tau + 3a} } \right)}}{{\left( {1 + e^{\tau + a} + e^{2a} } \right)^{2} }}. $$

(16)

Proposition A.1

Let $\tau^{*} = \ln \;(\sqrt 5 + 1) \approx 1.174$. Then for $\tau \le \tau^{*}$ the function ${\text{VAR}}\left( {\tau ,a} \right)$ has a single local maximum at $a = a_{1} = 0$, whereas for $\tau > \tau^{*}$ it has one local minimum at $a = a_{1} = 0$ and two local maxima at $a = a_{2,3}$,

$a_{2} < 0,\,a_{3} > 0,\,\,a_{2} = - \,a_{3}$.

Proof.

Denote $z \triangleq e^{a} > 0,$$b \triangleq e^{\tau } ,$ and introduce the functions

$$ g_{1} \left( z \right) = bz + 2z^{2} + bz^{3} , $$

$$ g_{2} \left( z \right) = 1 + bz + z^{2} . $$

Then,

$$ {\text{VAR = VAR}}(z) = \frac{{2g_{1} \left( z \right)}}{{g_{2}^{2} \left( z \right)}}. $$

(17)

Let us search for real positive critical points of (13) by setting to zero the first derivative:

$$ \frac{{d{\text{VAR}}(z)}}{dz} = \frac{{2\left( {g^{\prime}_{1} g_{2}^{2} - 2g_{2} g^{\prime}_{2} g_{1} } \right)}}{{g_{2}^{4} }} = \frac{{2\left( {g^{\prime}_{1} g_{2} - 2g_{1} g^{\prime}_{2} } \right)}}{{g_{2}^{3} }}. $$

Consider the equation

$$ g^{\prime}_{1} (z)g_{2} (z) - 2g_{1} (z)g^{\prime}(z) = 0, $$

(18)

which, by simple algebra is rewritten as

$$ - \left( {z^{2} - 1} \right)\left[ {bz^{2} - \left( {b^{2} - 4} \right)z + b} \right] = 0. $$

For any $\tau > 0$, this equation admits the root $z_{1} = 1$. Subject to the condition

$$ (b^{2} - 4)^{2} - 4b^{2} \ge 0, $$

(19)

it admits two additional real roots

$$ z_{2,3} = \frac{{b^{2} - 4 \pm \sqrt {(b^{2} - 4)^{2} - 4b^{2} } }}{2b}. $$

By some algebra, these roots are

$$ z_{2,3} = \frac{1}{2}e^{\tau } - 2e^{ - \tau } \pm \frac{1}{2}\sqrt {e^{2\tau } + 16e^{ - 2\tau } - 12} . $$

(20)

The condition (19) becomes

$$ e^{2\tau } + 16e^{ - 2\tau } - 12 \ge 0, $$

or

$$ e^{2\tau } \le 6 - \sqrt {20} , e^{2\tau } \ge 6 + \sqrt {20} . $$

Thus, the Eq. (18) admits three real roots w.r.t. $z$ if

$$ \tau \le \hat{\tau }, \tau \ge \tau^{*} , $$

(21)

where

$$ \hat{\tau } = \frac{1}{2}{\text{ln}}\left( {6 - \sqrt {20} } \right) = 0.2119, \tau^{*} = \frac{1}{2}{\text{ln}}\left( {6 + \sqrt {20} } \right) = 1.1744. $$

Note that $\sqrt {\left( {6 \pm \sqrt {20} } \right)} = \sqrt 5 \pm 1.$ This is directly derived by squaring: for example,

$$ (\sqrt 5 + 1)^{2} = 5 + 2\sqrt 5 + 1 = 6 + \sqrt {20} . $$

This means that the critical values of $\tau$ can be written as

$$ \hat{\tau } = {\text{ln}}\left( {\sqrt 5 - 1} \right), \tau^{*} = {\text{ln}}\left( {\sqrt 5 + 1} \right). $$

(22)

Let us show that for $0 \le \tau \le \hat{\tau }$,

$$ z_{2,3} < 0. $$

(23)

Since $z_{2} > z_{3}$, it is sufficient to prove the inequality (23) for $z_{2}$ with the sign "$+$" before the square root. Due to (20),

$$ z_{2} < g\left( \tau \right) = \frac{1}{2}e^{\tau } - 2e^{ - \tau } . $$

Note that

$$ g^{\prime}\left( \tau \right) = \frac{1}{2}e^{\tau } + 2e^{ - \tau } > 0, $$

i.e., the function $g\left( \tau \right)$ increases. By direct calculation,

$$ g\left( {\hat{\tau }} \right) = \frac{\sqrt 5 - 1}{2} - \frac{2}{\sqrt 5 - 1} = \frac{{(\sqrt 5 - 1)^{2} - 4}}{{2\left( {\sqrt 5 - 1} \right)}} = \frac{5 - 2\sqrt 5 + 1 - 4}{{2\left( {\sqrt 5 - 1} \right)}} = - 1. $$

Therefore, $g\left( \tau \right) < 0$ for $\tau \le \hat{\tau }$, and the inequality (23) is proved.

Similarly, we show that for $\tau \ge \tau^{*}$,

$$ z_{2,3} > 0. $$

(24)

Since $z_{2} > z_{3}$, it is sufficient to prove the inequality (24) for $z_{3}$ with the sign "$-$" before the square root. Note that

$$ e^{2\tau } + 16e^{ - 2\tau } - 12 < e^{2\tau } + 16e^{ - 2\tau } - 8 = (e^{\tau } - 4e^{ - \tau } )^{2} . $$

Denote

$$ h\left( \tau \right) \triangleq e^{\tau } - 4e^{ - \tau } . $$

Note that $h^{\prime}\left( \tau \right) = e^{\tau } + 4e^{ - \tau } > 0$, and by direct calculations $h\left( {\tau^{*} } \right) = 2$. Therefore, $h\left( \tau \right) > 0$ for $\tau \ge \tau^{*}$, and,

$$ \sqrt {e^{2\tau } + 16e^{ - 2\tau } - 12} < e^{\tau } - 4e^{ - \tau } . $$

Then,

$$ z_{3} = \frac{1}{2}e^{\tau } - 2e^{ - \tau } - \frac{1}{2}\sqrt {e^{2\tau } + 16e^{ - 2\tau } - 12} > \frac{1}{2}e^{\tau } - 2e^{ - \tau } - \frac{1}{2}\left( {e^{\tau } - 4e^{ - \tau } } \right) = 0, $$

which justifies the inequality (24). Figure 9 illustrates the inequalities (23) and (24).

Fig. 9

Roots $z_{2,3}$ for $\tau \le \hat{\tau }$ and for $\tau \ge \tau^{*}$

Thus, for $\tau \le \tau^{*}$, the Eq. (18) admits a single real positive root $z_{1} = 1$, whereas for $\tau > \tau^{*}$, it admits three real positive roots. Therefore, the critical points of ${\text{VAR}}\left( {a,\tau } \right)$ are $a_{1} = {\text{ln}}z_{1} = 0$ for any $\tau > 0$, and

$$ a_{2,3} = {\text{ln}}z_{3,2} = {\text{ln}}\left[ {\frac{1}{2}e^{\tau } - 2e^{ - \tau } \mp \frac{1}{2}\sqrt {e^{2\tau } + 16e^{ - 2\tau } - 12} } \right],\,\,\,\tau > \tau^{*} . $$

(25)

It can be directly shown that the critical points $a_{2}$ and $a_{3}$ are symmetric with respect to $a = 0$. Indeed,

$$ \begin{gathered} a_{2} + a_{3} \hfill \\ = {\text{ln}}\left[ {\frac{1}{2}e^{\tau } - 2e^{ - \tau } - \frac{1}{2}\sqrt {e^{2\tau } + 16e^{ - 2\tau } - 12} } \right] + {\text{ln}}\left[ {\frac{1}{2}e^{\tau } - 2e^{ - \tau } + \frac{1}{2}\sqrt {e^{2\tau } + 16e^{ - 2\tau } - 12} } \right] \hfill \\ = {\text{ln}}\left[ {\left( {\frac{1}{2}e^{\tau } - 2e^{ - \tau } - \frac{1}{2}\sqrt {e^{2\tau } + 16e^{ - 2\tau } - 12} } \right)\left( {\frac{1}{2}e^{\tau } - 2e^{ - \tau } + \frac{1}{2}\sqrt {e^{2\tau } + 16e^{ - 2\tau } - 12} } \right)} \right] \hfill \\ = {\text{ln}}\left[ {\left( {\frac{1}{2}e^{\tau } - 2e^{ - \tau } } \right)^{2} - \frac{1}{4}\left( {e^{2\tau } + 16e^{ - 2\tau } - 12} \right)} \right] \hfill \\ = {\text{ln}}\left[ {\frac{1}{4}e^{2\tau } - 2 + 4e^{ - 2\tau } - \frac{1}{4}e^{2\tau } - 4e^{ - 2\tau } + 3} \right] = {\text{ln}}1 = 0. \hfill \\ \end{gathered} $$

Let us show that the critical point $a_{1} = 0$ is local maximum for $\tau \in \left[ {0,\tau^{*} } \right]$ and local minimum for $\tau > \tau^{*}$. To this end, substitute $a = 0$ into the second derivative of ${\text{VAR}}(a,\tau )$ w.r.t. $a$:

$$ G_{1} \left( \tau \right) \triangleq \left. {\frac{{\partial^{2} VAR\left( {a,\tau } \right)}}{{\partial a^{2} }}} \right|_{a = 0} = \frac{{4\left( {e^{2\tau } - 2e^{\tau } - 4} \right)}}{{\left( {e^{\tau } + 2} \right)^{3} }}. $$

By solving the quadratic equation $\left( {e^{\tau } } \right)^{2} - 2e^{\tau } - 4 = 0$, we readily obtain $e^{\tau } = \sqrt 5 + 1$, i.e., the function $G_{1} \left( \tau \right)$ has a single real zero $\tau = {\text{ln}}\left( {\sqrt 5 + 1} \right) = \tau^{*}$. Moreover, by solving the equation

$$ G^{\prime}_{1} \left( \tau \right) = - \frac{{4e^{\tau } \left( {e^{2\tau } - 8e^{\tau } - 8} \right)}}{{\left( {e^{\tau } + 2} \right)^{4} }} = 0, $$

we obtain that the function $G_{1} \left( \tau \right)$ has the single critical point $\overline{\tau } = {\text{ln}}\left( {2\sqrt 6 + 4} \right) = 2.1859 > \tau^{*}$. Since $G^{\prime\prime}_{1} \left( {\overline{\tau }} \right) < 0$, $\tau = \overline{\tau }$ is the local maximum of $G_{1} \left( \tau \right)$. Thus, $G_{1} \left( \tau \right) < 0$ for $\tau < \tau^{*}$, $G_{1} \left( {\tau^{*} } \right) = 0$, and $G_{1} \left( \tau \right) > 0$ for $\tau > \tau^{*}$ (the function $G_{1} \left( \tau \right)$ is depicted in Fig. 10). This proves that $a = 0$ is the local maximum for $\tau \le \tau^{*}$ and local minimum for $\tau > \tau^{*}$.

Fig. 10

The function $G_{1} \left( \tau \right)$

It can be shown that

$$ G_{2} \left( \tau \right) \triangleq \left. {\frac{{\partial^{2} {\text{VAR}}\left( {a,\tau } \right)}}{{\partial a^{2} }}} \right|_{{a = a_{2} }} = \left. {\frac{{\partial^{2} {\text{VAR}}\left( {a,\tau } \right)}}{{\partial a^{2} }}} \right|_{{a = a_{3} }} . $$

Numerically, we can show that $G_{2} \left( \tau \right) < 0$ for $\tau > \tau^{*}$ (see Fig. 11).

Fig. 11

The function $G_{2} \left( \tau \right)$

This means that the critical points $a_{2,3}$ are local maximums of ${\text{VAR}}(a,\tau )$ for $\tau > \tau^{*}$.

This completes the proof of the proposition.

In Fig. 12, the local extrema $a_{1} ,a_{2}$ and $a_{3}$ of ${\text{VAR}}(a,\tau )$ are shown as functions of $\tau$.

Fig. 12

Critical points of ${\text{VAR}}(a,\tau )$ for ternary response

Proposition A.2.

The following limiting properties are valid:

$$ \mathop {{\text{lim}}}\limits_{\tau \to \infty } \left( {a_{2} (\tau ) + \tau } \right) = \mathop {{\text{lim}}}\limits_{\tau \to \infty } \left( {a_{3} (\tau ) - \tau } \right) = 0, $$

(26)

$$ \mathop {{\text{lim}}}\limits_{\tau \to \infty } {\text{VAR(0,}}\tau {)} = 0, $$

(27)

$$ \mathop {{\text{lim}}}\limits_{\tau \to 0} {\text{VAR(0,}}\tau {)} = \frac{8}{9}, $$

(28)

$$ \mathop {{\text{lim}}}\limits_{\tau \to \infty } {\text{VAR(}}a_{2} {(}\tau {),}\tau {)} = \mathop {{\text{lim}}}\limits_{\tau \to \infty } {\text{VAR(}}a_{3} {(}\tau {),}\tau {)} = \frac{1}{2}. $$

(29)

Proof

Straightforwardly,

$$ \begin{gathered} \mathop {{\text{lim}}}\limits_{\tau \to \infty } \left[ {{\text{ln}}\left( {\frac{1}{2}e^{\tau } - 2e^{ - \tau } - \frac{1}{2}\sqrt {e^{2\tau } + 16e^{ - 2\tau } - 12} } \right) + \tau } \right] \hfill \\ = \mathop {{\text{lim}}}\limits_{\tau \to \infty } \left[ {{\text{ln}}\left( {\frac{1}{2}e^{\tau } - 2e^{ - \tau } - \frac{1}{2}\sqrt {e^{2\tau } + 16e^{ - 2\tau } - 12} } \right) + {\text{ln}}e^{\tau } } \right] \hfill \\ = \mathop {{\text{lim}}}\limits_{\tau \to \infty } {\text{ln}}\left( {\frac{1}{2}e^{\tau } - 2e^{ - \tau } - \frac{1}{2}\sqrt {e^{2\tau } + 16e^{ - 2\tau } - 12} } \right)e^{\tau } \hfill \\ = \mathop {{\text{lim}}}\limits_{\tau \to \infty } {\text{ln}}\left[ {\frac{1}{2}e^{2\tau } - 2 - \frac{1}{2}\sqrt {e^{4\tau } + 16 - 12e^{2\tau } } } \right]. \hfill \\ \end{gathered} $$

Note that

$$ \begin{gathered} \mathop {{\text{lim}}}\limits_{\tau \to \infty } \left( {e^{2\tau } - \sqrt {e^{4\tau } + 16 - 12e^{2\tau } } } \right) = \mathop {{\text{lim}}}\limits_{\tau \to \infty } \frac{{\left( {e^{2\tau } - \sqrt {e^{4\tau } + 16 - 12e^{2\tau } } } \right)\left( {e^{2\tau } + \sqrt {e^{4\tau } + 16 - 12e^{2\tau } } } \right)}}{{e^{2\tau } + \sqrt {e^{4\tau } + 16 - 12e^{2\tau } } }} \hfill \\ = \mathop {{\text{lim}}}\limits_{\tau \to \infty } \frac{{e^{4\tau } - e^{4\tau } - 16 + 12e^{2\tau } }}{{e^{2\tau } + \sqrt {e^{4\tau } + 16 - 12e^{2\tau } } }} = \mathop {{\text{lim}}}\limits_{\tau \to \infty } \frac{{12 - 16e^{ - 2\tau } }}{{1 + \sqrt {1 + 16e^{ - 4\tau } - 12e^{ - 2\tau } } }} = \frac{12}{2} = 6, \hfill \\ \end{gathered} $$

yielding ${\text{lim}}_{\tau \to \infty } {\text{ln}}\left[ {\frac{1}{2}e^{2\tau } - 2 - \frac{1}{2}\sqrt {e^{4\tau } + 16 - 12e^{2\tau } } } \right] = {\text{lim}}_{\tau \to \infty } {\text{ln}}\left( {\frac{6}{2} - 2} \right) = {0}{\text{.}}$.

The second limiting expression in (26) is proved in the same manner.

Due to (16),

$$ {\text{VAR}}(0,\tau ) = \frac{{4e^{\tau } + 4}}{{(2 + e^{\tau } )^{2} }}, $$

which leads to (27) and (28). Note that (28) is a particular case of (7) for $m = 3$: the distance between the thresholds is $T = 2\tau$, and $T \to 0$ for $\tau \to 0$.

The limiting Eq. (29) are readily obtained by MATLAB R2015a symbolic calculations (see Fig. 13).

Fig. 13

MATLAB symbolic calculations for $a = a_{3}$

Appendix 3: Ordinal variation for equidistant thresholds

In this appendix, the ordinal variation is calculated explicitly in the case of equidistant thresholds.

Consider the set of symmetric equidistant thresholds $\tau_{1} ,\tau_{2} ,...,\tau_{m - 1}$ satisfying

$$ \tau_{i + 1} - \tau_{i} = T,\,\,\,\,i = 1,...,m - 2, $$

(30)

$$ \tau_{1} + \tau_{2} + \cdots + \tau_{m - 1} = 0. $$

(31)

We obtain from (30)–(31) that

$$ \begin{gathered} \tau_{1} + \tau_{2} + ... + \tau_{m - 1} = \tau_{1} + \tau_{1} + T + \tau_{1} + 2T + ... + \tau_{1} + (m - 2)T = \hfill \\ (m - 1)\tau_{1} + (m - 2)\frac{T + (m - 2)T}{2} = (m - 1)\tau_{1} + \frac{(m - 1)(m - 2)T}{2} = 0, \hfill \\ \end{gathered} $$

yielding $\tau_{1} = - \frac{m - 2}{2}T,\tau_{2} = - \frac{m - 4}{2}T,...,\tau_{m - 1} = - \frac{m - 2(m - 1)}{2}T$, or, generally

$$ \tau_{i} = - \frac{m - 2i}{2}T,\,\,i = 1,...,m - 1 $$

(32)

For example:

For $m = 4$,$\tau_{1} = - T,\tau_{2} = 0,\tau_{3} = T$;

For $m = 5$,$\tau_{1} = - \frac{3T}{2},\tau_{2} = - \frac{T}{2},\tau_{3} = \frac{T}{2},\tau_{4} = \frac{3T}{2}$;

For $m = 6$, $\tau_{1} = - 2T,\tau_{2} = - T,\tau_{3} = 0,\tau_{4} = T,\tau_{5} = 2T$, etc.

In order to calculate the ordinal variation, let us write down the expressions for the sums of the thresholds:

$$ \begin{aligned} S_{i} & = \tau_{1} + \tau_{2} + \cdots + \tau_{i} = \sum\limits_{k = 1}^{i} {\tau_{i} } = \sum\limits_{k = 1}^{i} {\left( { - \frac{m - 2k}{2}T} \right)} \hfill \\ & = - \frac{i}{2}T \cdot \frac{m - 2 + m - 2i}{2} \hfill \\ & = - \frac{i(m - i - 1)T}{2},\,\,i = 1, \ldots ,m - 1. \hfill \\ \end{aligned} $$

By adding a “fictitious” threshold $\tau_{0} = 0$,

$$ S_{i} = - \frac{i(m - i - 1)T}{2},\,\,i = 0,1,...,m - 1. $$

(33)

Thus,

$$ S_{i} = S_{m - i + 1} ,\,\,i = 0,1, \ldots ,M - 1, $$

(34)

where $M = \frac{m}{2}$, if $m$ is even, and $M = \frac{m - 1}{2}$ otherwise. Note that the sums (33) are located symmetrically on the parabola $S = - \frac{x(m - x - 1)T}{2},\,x \in [0,m - 1]$(see an example in Fig. 14).

Fig. 14

Sums $S_{i}$ for $m = 9,T = 0.5$

Due to (33), the probabilities are

$$ p_{i} (a,T) = \frac{{e^{{ia - \sum\nolimits_{k = 0}^{i} {\tau_{i} } }} }}{{\sum\nolimits_{i = 0}^{m - 1} {e^{{ia - \sum\nolimits_{k = 0}^{i} {\tau_{i} } }} } }} = \frac{{e^{{ia}{ - S_{i} }} }}{{\sum\limits_{i = 0}^{m - 1} {e^{{ia}{ - S_{i} }} } }} = \frac{{e^{{ia + \frac{i(m - i - 1)T}{2}}} }}{{\sum\nolimits_{i = 0}^{m - 1} {e^{{ia + \frac{i(m - i - 1)T}{2}}} } }} = \frac{{e^{{i\left( {a - \frac{(i + 1 - m)T}{2}} \right)}} }}{{\sum\nolimits_{i = 0}^{m - 1} {e^{{i\left( {a - \frac{(i + 1 - m)T}{2}} \right)}} } }},{\mkern 1mu} {\mkern 1mu} i = 0,1, \ldots ,m - 1. $$

The ordinal variation is

$$ \begin{gathered} {\text{VAR}}(a,T) \hfill \\ \quad{ = }\frac{4}{m - 1}\left[ {p_{0} (a,T)\left( {p_{1} (a,T) + p_{2} (a,T) + \cdots + p_{m - 1} (a,T)} \right)} \right. \hfill \\ \qquad+ \left( {p_{0} (a,T) + p_{1} (a,T)} \right)\left( {p_{2} (a,T) + \cdots + p_{m - 1} (a,T)} \right) + \cdots \hfill \\ \qquad\left. { + \left( {p_{0} (a,T) + p_{1} (a,T) + \cdots + p_{m - 2} (a,T)} \right)p_{m - 1} (a,T)} \right]. \hfill \\ \end{gathered} $$

In particular, for $m = 4$,

$$ {\text{VAR}}(a,T) \, { = } \, \frac{4}{3} \cdot \frac{{ \, e^{a + T} { + 2}e^{2a + T} + \left( {3 + e^{2T} } \right)e^{3a} { + 2}e^{4a + T} { + }e^{5a + T} }}{{\left( {1 + e^{a + T} + e^{2a + T} + e^{3a} } \right)^{2} }}. $$

Appendix 4: Maximum likelihood for $m = 3$

In this appendix, maximum likelihood estimates of abilities are calculated for three quality categories.

Assume that a classifier’s responses are distributed as follows: $n_{0}$ times—"0", $n_{1}$ times—"1" and $n_{2}$ times—"2", and $n_{0} + n_{1} + n_{2} = N$. Then, the likelihood function for $m = 3$ is

$$ \ell \left( a \right) = p_{0}^{{n_{0} }} p_{1}^{{n_{1} }} p_{2}^{{n_{2} }} = \left[ {\frac{1}{{1 + e^{\tau + a} + e^{2a} }}} \right]^{{n_{0} }} \left[ {\frac{{e^{\tau + a} }}{{1 + e^{\tau + a} + e^{2a} }}} \right]^{{n_{1} }} \left[ {\frac{{e^{2a} }}{{1 + e^{\tau + a} + e^{2a} }}} \right]^{{n_{2} }} . $$

(35)

Logarithm of likelihood function is

$$ \begin{gathered} L\left( {a;\tau ,N,n_{1} ,n_{2} } \right) = \ln \ell (a) \hfill \\ \quad= n_{0} {\text{ln}}\left[ {\frac{1}{{1 + e^{\tau + a} + e^{2a} }}} \right] + n_{1} {\text{ln}}\left[ {\frac{{e^{\tau + a} }}{{1 + e^{\tau + a} + e^{2a} }}} \right] + n_{2} {\text{ln}}\left[ {\frac{{e^{2a} }}{{1 + e^{\tau + a} + e^{2a} }}} \right] \hfill \\ \quad= n_{0} \left[ {{\text{ln}}1 - {\text{ln}}\left( {1 + e^{\tau + a} + e^{2a} } \right)} \right] + n_{1} \left[ {{\text{ln}}e^{\tau + a} - {\text{ln}}\left( {1 + e^{\tau + a} + e^{2a} } \right)} \right] \hfill \\ \qquad + n_{2} \left[ {{\text{ln}}e^{2a} - {\text{ln}}\left( {1 + e^{\tau + a} + e^{2a} } \right)} \right] \hfill \\ \quad= n_{1} \left( {\tau + a} \right) + 2n_{2} a - N{\kern 1pt} {\text{ln}}\left( {1 + e^{\tau + a} + e^{2a} } \right). \hfill \\ \end{gathered} $$

(36)

In Fig. 15, the graphs of the function (36) are shown for $N = 10$, $\tau = 0.1$ and two pairs of $\left( {n_{1} ,n_{2} } \right)$. It is seen that the function has a unique maximum.

Fig. 15

Function $L\left( {a\,;\tau ,N,n_{1} ,n_{2} } \right)$

The necessary condition for the maximum of (36) w.r.t. $a$ is

$$ \frac{\partial L}{{\partial a}} = n_{1} + 2n_{2} - N\frac{{e^{\tau + a} + 2e^{2a} }}{{1 + e^{\tau + a} + e^{2a} }} = 0, $$

(37)

yielding the equation

$$ f\left( {a\,;\tau } \right) \triangleq \frac{{e^{\tau + a} + 2e^{2a} }}{{1 + e^{\tau + a} + e^{2a} }} = n, $$

(38)

where

$$ n \triangleq \frac{{n_{1} + 2n_{2} }}{N}. $$

(39)

See, e.g., the graphs of $f\left( {a\,;\tau } \right)$ for $\tau = 0.1$ and $\tau = 1$ in Fig. 16.

Fig. 16

Function $f\left( {a\,;\tau } \right)$

Proposition A.3

There exists a unique maximum of the log-likelihood function (36) w.r.t. a.

Proof.

By differentiating $f\left( {a\,;\tau } \right)$ w.r.t.$a$

$$ \frac{\partial f}{{\partial a}} = \frac{{e^{\tau + a} + e^{\tau + 3a} + 4e^{2a} }}{{\left( {1 + e^{\tau + a} + e^{2a} } \right)^{2} }} > 0, $$

(40)

meaning that the function $f\left( {a\,;\tau } \right)$ increases monotonically for any $\tau \in R$. Moreover,

$$ \mathop {\lim }\limits_{a \to - \infty } f\left( {a\,;\tau } \right) = 0,\,\mathop {\lim }\limits_{a \to \infty } f\left( {a\,;\tau } \right) = 2. $$

(41)

Thus, for all $a,\tau \in R$, $f\left( {a;\tau } \right) \in \left( {0,2} \right).$

Due to (39),

$n = \frac{{n_{1} + n_{2} }}{N} + \frac{{n_{2} }}{N} \in (0,2)$.

Therefore, the Eq. (38) has a unique real root. By (36),

$$ \frac{{\partial^{2} L}}{{\partial a^{2} }} = - N\frac{{\partial {\kern 1pt} f}}{\partial a} < 0, $$

(42)

yielding the strict concavity of the function $L\left( {a\,;\tau ,N,n_{1} ,n_{2} } \right)$ w.r.t. a. Thus, the unique zero of the Eq. (38) is the maximum of (36) w.r.t. a.

By denoting $e^{a} = z$ ($a = {\text{ln}}z$), the Eq. (38) becomes

$$ \left( {2 - n} \right)z^{2} + \left( {1 - n} \right)e^{\tau } z - n = 0, $$

(43)

yielding the roots

$$ z_{1,2} = \frac{{\left( {n - 1} \right)e^{\tau } \pm \sqrt {(n - 1)^{2} e^{2\tau } + 4n\left( {2 - n} \right)} }}{{2\left( {2 - n} \right)}}. $$

(44)

Note that $\left( {n - 1} \right)e^{2\tau } + 4n\left( {2 - n} \right) > 0$ for $n \in \left( {0,2} \right)$, and the roots (44) are real, one of which is negative (with “−”) and the other positive (with “+”). Since $a = {\text{ln}}z$, only a positive root with “$+$” provides the root of the Eq. (38) w.r.t. a.

$$ a^{0} = {\text{ln}}\left[ {\frac{{\left( {n - 1} \right)e^{\tau } + \sqrt {(n - 1)^{2} e^{2\tau } + 4n\left( {2 - n} \right)} }}{{2\left( {2 - n} \right)}}} \right]. $$

(45)

For example, for $\tau = 0.1$, $N = 10$, $n_{\,1} = 2$, $n_{2} = 3$: $a^{0} = - 0.3153$ (the first curve in Fig. 15); for $\tau = 0.1$, $N = 10$, $n_{\,1} = 3$, $n_{2} = 5$:$a^{0} = 0.4824$ (the second curve in Fig. 15).