Higher-Order Asymptotics and Its Application to Testing the Equality of the Examinee Ability Over Two Sets of Items

Abstract

In educational and psychological measurement, researchers and/or practitioners are often interested in examining whether the ability of an examinee is the same over two sets of items. Such problems can arise in measurement of change, detection of cheating on unproctored tests, erasure analysis, detection of item preknowledge, etc. Traditional frequentist approaches that are used in such problems include the Wald test, the likelihood ratio test, and the score test (e.g., Fischer, Appl Psychol Meas 27:3–26, 2003; Finkelman, Weiss, & Kim-Kang, Appl Psychol Meas 34:238–254, 2010; Glas & Dagohoy, Psychometrika 72:159–180, 2007; Guo & Drasgow, Int J Sel Assess 18:351–364, 2010; Klauer & Rettig, Br J Math Stat Psychol 43:193–206, 1990; Sinharay, J Educ Behav Stat 42:46–68, 2017). This paper shows that approaches based on higher-order asymptotics (e.g., Barndorff-Nielsen & Cox, Inference and asymptotics. Springer, London, 1994; Ghosh, Higher order asymptotics. Institute of Mathematical Statistics, Hayward, 1994) can also be used to test for the equality of the examinee ability over two sets of items. The modified signed likelihood ratio test (e.g., Barndorff-Nielsen, Biometrika 73:307–322, 1986) and the Lugannani–Rice approximation (Lugannani & Rice, Adv Appl Prob 12:475–490, 1980), both of which are based on higher-order asymptotics, are shown to provide some improvement over the traditional frequentist approaches in three simulations. Two real data examples are also provided.

Appendix: The MSLRT and the LRA for the GPCM

Donoghue (1994) provided the result that for log likelihood given by Eq. 21,

$$\begin{aligned} \frac{\partial ^2 \ell (\theta ) }{\partial \theta ^2}= \sum _{i=1}^{n} a^2_i \left( \left[ \sum _{k=0}^{m_i} kP_{ik}(\theta ) \right] ^2-\sum _{k=0}^{m_i} k^2P_{ik}(\theta ) \right) \cdot \end{aligned}$$

(25)

Noting that

$$\begin{aligned} \mathop {\mathrm {Var}}\nolimits (X_i|\theta )=\sum _{k=0}^{m_i} k^2P_{ik}(\theta ) - \left[ \sum _{k=0}^{m_i} kP_{ik}(\theta ) \right] ^2, \end{aligned}$$

Equation 25 can be rewritten as

$$\begin{aligned} \frac{\partial ^2 \ell (\theta ) }{\partial \theta ^2}= - \sum _{i=1}^{n} a^2_i \mathop {\mathrm {Var}}\nolimits (X_i|\theta ) \cdot \end{aligned}$$

(26)

For two sets of items, using notations introduced below Eq. 22, the joint log likelihood of $\theta _1$ and $\theta _2$ is given by

$$\begin{aligned} \ell (\theta _1,\theta _2)= & {} \sum _{{i}=1}^{n_1} \sum _{k=0}^{m_{i}} d_k(X_{i}) \log P_{{i}k}(\theta _1)+\sum _{j=1}^{n_2} \sum _{k=0}^{m_{j}} d_k(Y_j) \log \tilde{P}_{{j}k}(\theta _2) \nonumber \\= & {} \sum _{{i}=1}^{n_1} \sum _{k=0}^{m_{i}} d_k(X_{i})\left\{ \sum _{h=0}^{k} a_{i}(\theta _1-b_{{i}h}) - \log (\Gamma _{i}(\theta _1)) \right\} \nonumber \\&\quad +\, \sum _{{j}=1}^{n_2} \sum _{k=0}^{m_{j}} d_k(Y_j)\left\{ \sum _{h=0}^{k} \tilde{a}_{j}(\theta _2-\tilde{b}_{{j}h}) - \log (\tilde{\Gamma }_{{j}}(\theta _2)) \right\} \nonumber \\= & {} \theta _1 \sum _{{i}=1}^{n_1} a_{i} \sum _{k=0}^{m_{i}} (k+1) d_k(X_{i}) - \sum _{{i}=1}^{n_1} a_{i} \sum _{k=0}^{m_{i}} d_k(X_{i})\sum _{h=0}^{k} b_{{i}h} - \sum _{{i}=1}^{n_1} \log (\Gamma _{{i}}(\theta _1))\nonumber \\&\quad +\, \theta _2 \sum _{{j}=1}^{n_2} \tilde{a}_{j} \sum _{k=0}^{m_{j}} (k+1) d_k(Y_j) - \sum _{{j}=1}^{n_2} \tilde{a}_{j} \sum _{k=0}^{m_{j}} d_k(Y_j)\sum _{h=0}^{k} \tilde{b}_{{j}h} - \sum _{{j}=1}^{n_2} \log (\tilde{\Gamma }_{{j}}(\theta _2))\nonumber \\ \end{aligned}$$

(27)

The last equality holds because $\sum _{k=0}^{m_{i}} d_k(X_{i})=\sum _{k=0}^{m_{j}} d_k(Y_j)=1$ under the assumption that no data are missing, which means, for example, that $\sum _{k=0}^{m_{i}} d_k(X_{i})\log (\Gamma _{i}(\theta _1))=\log (\Gamma _{i}(\theta _1))$.

Let us apply the transformations $\psi =\theta _2-\theta _1$ and $\lambda =\theta _1$, which means that $\theta _1=\lambda$ and $\theta _2=\psi +\lambda$. Let us also denote

$$\begin{aligned} S_1=\sum _{{i}=1}^{n_1} a_{i} \sum _{k=0}^{m_{i}} (k+1) d_k(X_{i}), \text{ and } S_2= \sum _{{j}=1}^{n_2} \tilde{a}_{j} \sum _{k=0}^{m_{j}} (k+1) d_k(Y_j) \cdot \end{aligned}$$

Note that both $S_1$ and $S_2$ are functions of the data ($X_{i}$’s and $Y_j$’s) and not of the parameters ($\theta _1$ and $\theta _2$). The above log likelihood, $\ell (\theta _1,\theta _2)$, or, $\ell (\psi ,\lambda )$, then is given by

$$\begin{aligned} \ell (\psi ,\lambda )= & {} \sum _{{i}=1}^{n_1} \sum _{k=0}^{m_{i}} d_k(X_{i}) \log P_{{i}k}(\lambda )+\sum _{j=1}^{n_2} \sum _{k=0}^{m_{j}} d_k(Y_j) \log \tilde{P}_{{j}k}(\psi +\lambda ) \end{aligned}$$

(28)

$$\begin{aligned}= & {} \lambda S_1 - \sum _{{i}=1}^{n_1} a_{i} \sum _{k=0}^{m_{i}} d_k(X_{i})\sum _{h=0}^{k} b_{{i}h} - \sum _{{i}=1}^{n_1} \log (\Gamma _{{i}}(\lambda )) \nonumber \\&+\, (\psi +\lambda ) S_2 - \sum _{{j}=1}^{n_2} \tilde{a}_{j} \sum _{k=0}^{m_{j}} d_k(Y_j)\sum _{h=0}^{k} \tilde{b}_{{j}h} - \sum _{{j}=1}^{n_2} \log (\tilde{\Gamma }_{{j}}(\psi +\lambda )) \nonumber \\= & {} S_2 \psi + (S_1+S_2) \lambda - \sum _{{i}=1}^{n_1} \log (\Gamma _{{i}}(\lambda )) - \sum _{{j}=1}^{n_2} \log (\tilde{\Gamma }_{{j}}(\psi +\lambda )) \nonumber \\&-\, \sum _{{i}=1}^{n_1} a_{i} \sum _{k=0}^{m_{i}} d_k(X_{i})\sum _{h=0}^{k} b_{{i}h} - \sum _{{j}=1}^{n_2} \tilde{a}_{j} \sum _{k=0}^{m_{j}} d_k(Y_j)\sum _{h=0}^{k} \tilde{b}_{{j}h} \cdot \end{aligned}$$

(29)

The above log likelihood belongs to the exponential family of distributions with canonical parameters $\psi$ and $\lambda$ and joint sufficient statistics $S_1$ and $(S_1+S_2)$.

Then, given the discussion on the applicability of the MSLRT and LRA to the exponential family of distributions, the MSLRT and LRA can be applied to test $H_0:\psi =0$, or, $H_0:\theta _1=\theta _2$, in applications of the GPCM. The SLR statistic is given by

$$\begin{aligned} r(\psi _0)= & {} \text{ sign }(\hat{\psi }-\psi _0) \left[ 2[\ell (\hat{\psi },\hat{\lambda })- \ell (\psi _0,\hat{\lambda }_{\psi _0})] \right] ^{1/2} \nonumber \\= & {} \text{ sign }(\hat{\theta }_2-\hat{\theta }_1) \sqrt{2} \left[ \sum _{{i}=1}^{n_1} \sum _{k=0}^{m_{i}} d_k(X_{i}) \log P_{{i}k}(\hat{\theta }_1)+\sum _{j=1}^{n_2} \sum _{k=0}^{m_{j}} d_k(Y_j) \log \tilde{P}_{{j}k}(\hat{\theta }_2) \right. \nonumber \\&- \left. \sum _{{i}=1}^{n_1} \sum _{k=0}^{m_{i}} d_k(X_{i}) \log P_{{i}k}(\hat{\theta }_0)-\sum _{j=1}^{n_2} \sum _{k=0}^{m_{j}} d_k(Y_j) \log \tilde{P}_{{j}k}(\hat{\theta }_0) \right] ^{1/2} \cdot \end{aligned}$$

(30)

Then, using the result provided in Eq. 26 (or, by differentiating the joint log likelihood provided in Eq. 29 twice),

$$\begin{aligned} j_{\psi \psi }({\psi },{\lambda })= & {} \sum _{j=1}^{n_2} \tilde{a}^2_{j} \mathop {\mathrm {Var}}\nolimits (Y_j|\psi +\lambda ), \nonumber \\ j_{\lambda \lambda }({\psi },{\lambda })= & {} \sum _{i=1}^{n_1} a^2_{i} \mathop {\mathrm {Var}}\nolimits (X_{i}|\lambda ) + \sum _{j=1}^{n_2} \tilde{a}^2_{j} \mathop {\mathrm {Var}}\nolimits (Y_j|\psi +\lambda ), \nonumber \\ j_{\psi \lambda }({\psi },{\lambda })= & {} j_{\lambda \psi }({\psi },{\lambda })=j_{\psi \psi }({\psi },{\lambda })\cdot \end{aligned}$$

(31)

Then,

$$\begin{aligned} |j({\psi },{\lambda })|= & {} j_{\psi \psi }({\psi },{\lambda })j_{\lambda \lambda }({\psi },{\lambda })-[j_{\psi \psi }({\psi },{\lambda })]^2= j_{\psi \psi }({\psi },{\lambda }) [j_{\lambda \lambda }({\psi },{\lambda })-j_{\psi \psi }({\psi },{\lambda }]\\= & {} \sum _{j=1}^{n_2} \tilde{a}^2_{j} \mathop {\mathrm {Var}}\nolimits (Y_j|\psi +\lambda ) \sum _{i=1}^{n_1} a^2_{i} \mathop {\mathrm {Var}}\nolimits (X_{i}|\lambda ), \end{aligned}$$

which implies that

$$\begin{aligned} |j({\hat{\psi }},{\hat{\lambda }})|= & {} \sum _{i=1}^{n_1}a^2_{i} \mathop {\mathrm {Var}}\nolimits (X_{i}|\hat{\theta }_1) \sum _{j=1}^{n_2} \tilde{a}^2_{j} \mathop {\mathrm {Var}}\nolimits (Y_j|\hat{\theta }_2) \cdot \end{aligned}$$

(32)

One can obtain $j_{\lambda \lambda }(\psi _0,\hat{\lambda }_{\psi _0})$ from Eq. 31 as

$$\begin{aligned} j_{\lambda \lambda }(\psi _0,\hat{\lambda }_{\psi _0})= & {} \sum _{i=1}^{n_1} a^2_{i} \mathop {\mathrm {Var}}\nolimits (X_{i}|\hat{\theta }_0) + \sum _{j=1}^{n_2} \tilde{a}^2_{j} \mathop {\mathrm {Var}}\nolimits (Y_j|\hat{\theta }_0) \end{aligned}$$

(33)

Then, $q(\psi _0)$ is given by

$$\begin{aligned} q(\psi _0)= & {} (\hat{\theta }_2-\hat{\theta }_1) \sqrt{\frac{|j(\hat{\psi },\hat{\lambda })|}{j_{\lambda \lambda }(\psi _0,\hat{\lambda }_{\psi _0})}} \nonumber \\= & {} (\hat{\theta }_2-\hat{\theta }_1) \sqrt{\frac{\sum _{i=1}^{n_1}a^2_{i} \mathop {\mathrm {Var}}\nolimits (X_{i}|\hat{\theta }_1) \sum _{j=1}^{n_2} \tilde{a}^2_{j} \mathop {\mathrm {Var}}\nolimits (Y_j|\hat{\theta }_2)}{\sum _{i=1}^{n_1} a^2_{i} \mathop {\mathrm {Var}}\nolimits (X_{i}|\hat{\theta }_0) + \sum _{j=1}^{n_2} \tilde{a}^2_{j} \mathop {\mathrm {Var}}\nolimits (Y_j|\hat{\theta }_0) }} \cdot \end{aligned}$$

(34)

Once $q(\psi _0)$ is computed using Eq. 34 and $r(\psi _0)$ is computed using Eq. 30, one can compute the MSLR statistic $r^*(\psi _0)$ using Eq. 12 and can compute the LRA using Eq. 14.