Predicting Rights-Only Score Distributions from Data Collected Under Formula Score Instructions

Abstract

Under a formula score instruction (FSI), test takers omit items. If students are encouraged to answer every item (under a rights-only scoring instruction, ROI), the score distribution will be different. In this paper, we formulate a simple statistical model to predict the score ROI distribution using the FSI data. Estimation error is also provided. In addition, a preliminary investigation of the probability of guessing correctly on omitted items and its sensitivity is presented in the paper. Based on the data used in this paper, the probability of guessing correctly may be close or slightly greater than the chance score.

Appendices

Appendix

Verification of the assumption in the central limit theorem (CLT) for Proposition 2

In Proposition 2, we used Lyapunov’s theorem (Durrett, 1995).

Theorem 2

Let $\alpha _n=\{\mathrm{var (S_n)}\}^{1/2}$, where $S_n=\sum _{i=1}^nX_i$. If there is a $\delta >0$ so that

$$\begin{aligned} \lim _{n\rightarrow \infty } \alpha _n^{-(2+\delta )}\sum _{i=1}^n E(|X_i-EX_i|^{2+\delta })=0, \end{aligned}$$

(5)

then $(S_n-ES_n)/\alpha _n$ converges to a standard normal distribution.

Here we show that the requirement holds in Proposition 2. In Proposition 2, for the group $\{ R=r\}$, $X_{r,i}$ is a Bernoulli random variable with a success rate of $p_{r,i}$ for an examinee i. We omit the subscript r for simplicity. Since there are limited options for each item, the success rate $1>c_1\ge p_i\ge c_2>0$ holds for some constants $c_1$ and $c_2$. For any $\delta >0$, $E(|X_i-EX_i|^{2+\delta })= q_i^{2+\delta }p_i+p_i^{2+\delta }q_i$, where $q_i=1-p_i$. Hence,

$$\begin{aligned} \lim _{n\rightarrow \infty } \alpha _n^{-(2+\delta )}\sum _{i=1}^n E\big (|X_i-EX_i|^{2+\delta }\big )= & {} \lim _{n\rightarrow \infty } \alpha _n^{-(2+\delta )}\sum _{i=1}^n p_iq_i\bigg (q_i^{1+\delta }+p_i^{1+\delta }\bigg )\\= & {} \lim _{n\rightarrow \infty } \left( \sum _{i=1}^n p_iq_i\right) ^{-(1+\delta /2)}\sum _{i=1}^n p_iq_i\bigg (q_i^{1+\delta }+p_i^{1+\delta }\bigg )\\< & {} \lim _{n\rightarrow \infty } \left( \sum _{i=1}^n p_iq_i\right) ^{-(1+\delta /2)}\sum _{i=1}^n p_iq_i(p_i+q_i) \\= & {} \lim _{n\rightarrow \infty } \left( \sum _{i=1}^n p_iq_i\right) ^{-(1+\delta /2)}\sum _{i=1}^n p_iq_i \\= & {} \lim _{n\rightarrow \infty }\left( \sum _{i=1}^n p_iq_i\right) ^{-\delta /2} \le \lim _{n\rightarrow \infty }(nc)^{-\delta /2}\rightarrow 0, \end{aligned}$$

where in the last inequality we use the fact that $p_i\ge c_2$ and $q_i\ge 1-c_1>0$. Therefore, (5) in Lyapunov’s CLT holds.