Ordering Individuals with Sum Scores: The Introduction of the Nonparametric Rasch Model

Abstract

When a simple sum or number-correct score is used to evaluate the ability of individual testees, then, from an accountability perspective, the inferences based on the sum score should be the same as the inferences based on the complete response pattern. This requirement is fulfilled if the sum score is a sufficient statistic for the parameter of a unidimensional model. However, the models for which this holds true are known to be restrictive. It is shown that the less restrictive nonparametric models could result in an ordering of persons that is different from an ordering based on the sum score. To arrive at a fair evaluation of ability with a simple number-correct score, ordinal sufficiency is defined as a minimum condition for scoring. The monotone homogeneity model, together with the property of ordinal sufficiency of the sum score, is introduced as the nonparametric Rasch model. A basic outline for testable hypotheses about ordinal sufficiency, as well as illustrations with real data, is provided.

Appendix

Let n be the number of items in $\mathbf{z}$, and define

$$\begin{aligned} X_+&= \sum _i X_i\\ Y_+&= \sum _h Y_h\\ Z_+&= \sum _g Z_g \end{aligned}$$

with realizations $x_+$, $y_+$ and $z_+$, respectively.

Following from the partitioning of $\mathbf{x}_1$ and $\mathbf{x}_2$,

$$\begin{aligned} x_{+2}&= y_+ + z_+,\\ x_{+1}&= y_+ + (n-z_+). \end{aligned}$$

In cases where

$$\begin{aligned} x_{+2}>x_{+1}, \end{aligned}$$

we obtain

$$\begin{aligned} z_+&>(n-z_+), \\ z_+&> \frac{n}{2}. \end{aligned}$$

Now we consider the posterior distribution

$$\begin{aligned} f(\theta |\mathbf{X}=\mathbf{x})=\frac{\prod _iP(x_i|\theta )^{x_i}[1-P(x_i|\theta )]^{1-x_i} f(\theta )}{P(\mathbf{X}=\mathbf{x})}. \end{aligned}$$

The likelihood ratio can be written as

$$\begin{aligned} \frac{f(\theta |\mathbf{X}=\mathbf{x}_2)}{f(\theta |\mathbf{X}=\mathbf{x}_1)}&= \frac{\frac{\prod _iP(x_{i2}|\theta )^{x_{i2}}[1-P(x_{i2}|\theta )]^{1-x_{i2}} f(\theta )}{P(\mathbf{X}=\mathbf{x}_2)}}{\frac{\prod _iP(x_{i1}\theta )^{x_{i1}}[1-P(x_{i1}\theta )]^{1-x_{i1}} f(\theta )}{P(\mathbf{X}=\mathbf{x}_1)}} \\&=\frac{\prod _iP(x_{i2}|\theta )^{x_{i2}}[1-P(x_{i2}|\theta )]^{1-x_{i2}} }{P(\mathbf{X}=\mathbf{x}_1)}\frac{P(\mathbf{X}=\mathbf{x}_2)}{\prod _iP(x_{i1}|\theta )^{x_{i1}}[1-P(x_{i1}|\theta )]^{1-x_{i1}}} \\&=\prod _i\frac{P(x_{i2}|\theta )^{x_{i2}}[1-P(x_{i2}\theta )]^{1-x_{i2}} }{P(x_{i1}|\theta )^{x_{i1}}[1-P(x_{i1}|\theta )]^{1-x_{i1}} }\frac{P(\mathbf{X}=\mathbf{x}_1)}{P(\mathbf{X}=\mathbf{x}_2)} \\&=\prod _h\frac{P(y_h|\theta )^{y_h}[1-P(y_h|\theta )]^{1-y_h}}{P(y_h|\theta )^{y_h}[1-P(y_h|\theta )]^{1-y_h}} \prod _g\frac{P(z_g|\theta )^{z_g}[1-P(z_g|\theta )]^{1-z_g} }{P(z_g|\theta )^{1-z_g}[1-P(z_g|\theta )]^{z_g}} \frac{P(\mathbf{X}=\mathbf{x}_1)}{P(\mathbf{X}=\mathbf{x}_2)} \\&=\prod _g\left( \frac{P(z_g|\theta )}{1-P(z_g|\theta )}\right) ^{z_g}\left( \frac{1-P(z_g|\theta )}{P(z_g|\theta )}\right) ^{1-z_g}\frac{P(\mathbf{X}=\mathbf{x}_1)}{P(\mathbf{X}=\mathbf{x}_2)}. \end{aligned}$$

The natural logarithm of likelihood ratio is

$$\begin{aligned} \log \left( \frac{f(\theta |\mathbf{X}=\mathbf{x}_2)}{f(\theta |\mathbf{X}=\mathbf{x}_1)}\right)= & {} \log \left( \prod _g\left( \frac{P(z_g|\theta )}{1-P(z_g|\theta )}\right) ^{z_g}\left( \frac{1-P(z_g|\theta )}{P(z_g|\theta )}\right) ^{1-z_g}\frac{P(\mathbf{X}=\mathbf{x}_1)}{P(\mathbf{X}=\mathbf{x}_2)}\right) \nonumber \\= & {} \log \left( \frac{P(\mathbf{X}=\mathbf{x}_1)}{P(\mathbf{X}=\mathbf{x}_2)}\right) + \sum _g z_g \log \left( \frac{P(z_g|\theta )}{1-P(z_g|\theta )}\right) \nonumber \\&+ \sum _g (1-z_g) \log \left( \frac{1-P(z_g|\theta )}{P(z_g|\theta )}\right) \nonumber \\= & {} \log \left( \frac{P(\mathbf{X}=\mathbf{x}_1)}{P(\mathbf{X}=\mathbf{x}_2)}\right) + \sum _g z_g \log \left( \frac{P(z_g|\theta )}{1-P_g(z_g|\theta )}\right) \nonumber \\&+ \sum _g (z_g-1) \log \left( \frac{P(z_g|\theta )}{1-P(z_g|\theta )}\right) \nonumber \\= & {} \log \left( \frac{P(\mathbf{X}=\mathbf{x}_1)}{P(\mathbf{X}=\mathbf{x}_2)}\right) + \sum _g (2z_g-1) \log \left( \frac{P(z_g|\theta )}{1-P(z_g|\theta )} \right) . \end{aligned}$$

(11)

It is generally known that if

$$\begin{aligned} \frac{f(\theta _2|\mathbf{X}=\mathbf{x}_2)}{f(\theta _2|\mathbf{X}=\mathbf{x}_1)}>\frac{f(\theta _1|\mathbf{X}=\mathbf{x}_2)}{f(\theta _1|\mathbf{X}=\mathbf{x}_1)}, \end{aligned}$$

then

$$\begin{aligned} \log \left( \frac{f(\theta _2|\mathbf{X}=\mathbf{x}_2)}{f(\theta _2|\mathbf{X}=\mathbf{x}_1)}\right) >\log \left( \frac{f(\theta _1|\mathbf{X}=\mathbf{x}_2)}{f(\theta _1|\mathbf{X}=\mathbf{x}_1)}\right) . \end{aligned}$$

Now, following from (11), the likelihood ratio can be written as

$$\begin{aligned}&\log \left( \frac{P(\mathbf{X}=\mathbf{x}_1)}{P(\mathbf{X}=\mathbf{x}_2)}\right) + \sum _g (2z_g-1) \log \left( \frac{P(z_g|\theta _2)}{1-P(z_g|\theta _2)}\right) > \\&\log \left( \frac{P(\mathbf{X}=\mathbf{x}_1)}{P(\mathbf{X}=\mathbf{x}_2)}\right) + \sum _g (2z_g-1) \log \left( \frac{P(z_g|\theta _1)}{1-P(z_g|\theta _1)}\right) \\&\Downarrow \\&\sum _g (2z_g-1) \log \left( \frac{P(z_g|\theta _2)}{1-P(z_g|\theta _2)}\right) > \sum _g (2z_g-1) \log \left( \frac{P(z_g|\theta _1)}{1-P(z_g|\theta _1)}\right) \\&\Downarrow \\&\sum _g (2z_g-1) \log \left( \frac{P(z_g|\theta _2)}{1-P(z_g|\theta _2)}\right) - \sum _g (2z_g-1) \log \left( \frac{P(z_g|\theta _1)}{1-P(z_g|\theta _1)}\right) >0\\&\Downarrow \\&\sum _g (2z_g-1) \left[ \log \left( \frac{P(z_g|\theta _2)}{1-P(z_g|\theta _2)}\right) - \log \left( \frac{P(z_g|\theta _1)}{1-P(z_g|\theta _1)}\right) \right] >0\\&\Downarrow \\&\sum _g (2z_g-1) \log \left[ \frac{\left( \frac{P(z_g|\theta _2)}{1-P(z_g|\theta _2)}\right) }{ \left( \frac{P(z_g|\theta _1)}{1-P(z_g|\theta _1)}\right) }\right] >0\\&\Downarrow \\&\sum _g z_g \log \left[ \frac{\left( \frac{P(z_g|\theta _2)}{1-P(z_g|\theta _2)}\right) }{ \left( \frac{P(z_g|\theta _1)}{1-P(z_g|\theta _1)}\right) }\right] >\sum _g (1-z_g) \log \left[ \frac{\left( \frac{P(z_g|\theta _2)}{1-P(z_g|\theta _2)}\right) }{ \left( \frac{P(z_g|\theta _1)}{1-P(z_g|\theta _1)}\right) }\right] ,\\&\;\theta _2>\theta _1,\;z_+>\frac{n}{2}. \end{aligned}$$