On Wald tests for differential item functioning detection

Abstract

Wald-type tests are a common procedure for DIF detection among the IRT-based methods. However, the empirical type I error rate of these tests departs from the significance level. In this paper, two reasons that explain this discrepancy will be discussed and a new procedure will be proposed. The first reason is related to the equating coefficients used to convert the item parameters to a common scale, as they are treated as known constants whereas they are estimated. The second reason is related to the parameterization used to estimate the item parameters, which is different from the usual IRT parameterization. Since the item parameters in the usual IRT parameterization are obtained in a second step, the corresponding covariance matrix is approximated using the delta method. The proposal of this article is to account for the estimation of the equating coefficients treating them as random variables and to use the untransformed (i.e. not reparameterized) item parameters in the computation of the test statistic. A simulation study is presented to compare the performance of this new proposal with the currently used procedure. Results show that the new proposal gives type I error rates closer to the significance level.

Appendices

Appendix A: Equating of untransformed item parameters

Equation (12) is obtained from Eqs. (7) and (4) as follows:

$$\begin{aligned} \hat{\beta }_{2jk}^* = D \hat{a}_{jk}^* = \frac{D \hat{a}_{jk}}{\hat{A}_k} = \frac{\hat{\beta }_{2jk}}{\hat{A}_k}. \end{aligned}$$

(A1)

Equations (7), (8) and (5) lead to Eq. (13):

$$\begin{aligned} \hat{\beta }_{1jk}^*= & {} - D \hat{a}_{jk}^* \hat{b}_{jk}^* = - D \frac{\hat{a}_{jk}}{\hat{A}_k} \left( \hat{A}_k \, \hat{b}_{jk} + \hat{B}_k\right) = - D \hat{a}_{jk} \hat{b}_{jk} - D \hat{a}_{jk} \frac{\hat{B}_k}{\hat{A}_k}\nonumber \\= & {} \hat{\beta }_{1jk}-\hat{\beta }_{2jk}\frac{\hat{B}_k}{\hat{A}_k}. \end{aligned}$$

(A2)

Appendix B: Covariance matrix of item parameters

The covariance matrix $\varvec{\varOmega }_j$ entering in Eq. (15) is a block matrix given by

$$\begin{aligned} \varvec{\varOmega }_j = \begin{pmatrix} \mathsf {COV}\left( \varvec{\beta }_{j1}\right) &{}\quad \mathsf {COV}\left( \varvec{\beta }_{j1},\varvec{\beta }_{j2}^*\right) &{}\quad \mathsf {COV}\left( \varvec{\beta }_{j1},\varvec{\beta }_{j3}^*\right) &{}\quad \dots &{}\quad \mathsf {COV}\left( \varvec{\beta }_{j1},\varvec{\beta }_{jK}^*\right) \\ \mathsf {COV}\left( \varvec{\beta }_{j2}^*,\varvec{\beta }_{j1}\right) &{}\quad \mathsf {COV}\left( \varvec{\beta }_{j2}^*\right) &{}\quad \mathsf {COV}\left( \varvec{\beta }_{j2}^*,\varvec{\beta }_{j3}^*\right) &{}\quad \dots &{}\quad \mathsf {COV}\left( \varvec{\beta }_{j2}^*,\varvec{\beta }_{jK}^*\right) \\ \mathsf {COV}\left( \varvec{\beta }_{j3}^*,\varvec{\beta }_{j1}\right) &{}\quad \mathsf {COV}\left( \varvec{\beta }_{j3}^*,\varvec{\beta }_{j2}^*\right) &{}\quad \mathsf {COV}\left( \varvec{\beta }_{j3}^*\right) &{}\quad \dots &{}\quad \mathsf {COV}\left( \varvec{\beta }_{j3}^*,\varvec{\beta }_{jK}^*\right) \\ \vdots &{}\quad \vdots &{}\quad \vdots &{}\quad \ddots &{}\quad \vdots \\ \mathsf {COV}\left( \varvec{\beta }_{jK}^*,\varvec{\beta }_{j1}\right) &{}\quad \mathsf {COV}\left( \varvec{\beta }_{jK}^*,\varvec{\beta }_{j2}^*\right) &{}\quad \mathsf {COV}\left( \varvec{\beta }_{jK}^*,\varvec{\beta }_{j3}^*\right) &{}\quad \dots &{}\quad \mathsf {COV}\left( \varvec{\beta }_{jK}^*\right) \end{pmatrix}. \end{aligned}$$

Let $\varvec{\beta }_{(k)}=(\varvec{\beta }_{1k}^\top , \dots ,\varvec{\beta }_{Jk}^\top )^\top$ denote the item parameters estimates in group k, and $\varvec{\varOmega }_{(k)} = \mathsf {COV}( \varvec{\beta }_{(k)})$ denote the covariance matrix of the item parameter estimates in group k, which is estimated along with the estimation of the item parameters. Using the delta method, it is possible to compute the covariance matrix $\varvec{\varOmega } = \mathsf {COV}(\varvec{\beta }_{(1)}^\top ,{\varvec{\beta }_{(2)}^*}^\top , \dots ,{\varvec{\beta }_{(K)}^*}^\top )^\top$, from which to extract $\varvec{\varOmega }_j$:

$$\begin{aligned} \varvec{\varOmega }&= \frac{\partial \left( \varvec{\beta }_{(1)}^\top , {\varvec{\beta }_{(2)}^*}^\top ,\dots ,{\varvec{\beta }_{(K)}^*}^\top \right) ^\top }{\partial \left( \varvec{\beta }_{(1)}^\top ,\varvec{\beta }_{(2)}^\top , \dots ,\varvec{\beta }_{(K)}^\top \right) }\mathsf {COV}\left( \left( \varvec{\beta }_{(1)}^\top , \varvec{\beta }_{(2)}^\top ,\dots ,\varvec{\beta }_{(K)}^\top \right) ^\top \right) \frac{\partial \left( \varvec{\beta }_{(1)}^\top , {\varvec{\beta }_{(2)}^*}^\top ,\dots ,{\varvec{\beta }_{(K)}^*}^\top \right) }{\partial \left( \varvec{\beta }_{(1)}^\top ,\varvec{\beta }_{(2)}^\top ,\dots , \varvec{\beta }_{(K)}^\top \right) ^\top } \\&= \begin{pmatrix} \frac{\partial \varvec{\beta }_{(1)}}{\partial \varvec{\beta }_{(1)}^\top } &{}\quad \frac{\partial \varvec{\beta }_{(1)}}{\partial \varvec{\beta }_{(2)}^\top } &{}\quad \cdots &{}\quad \frac{\partial \varvec{\beta }_{(1)}}{\partial \varvec{\beta }_{(K)}^\top } \\ \frac{\partial \varvec{\beta }_{(2)}^*}{\partial \varvec{\beta }_{(1)}^\top } &{}\quad \frac{\partial \varvec{\beta }_{(2)}^*}{\partial \varvec{\beta }_{(2)}^\top } &{}\quad \cdots &{}\quad \frac{\partial \varvec{\beta }_{(2)}^*}{\partial \varvec{\beta }_{(K)}^\top } \\ \vdots &{}\quad \vdots &{}\quad \ddots &{}\quad \vdots \\ \frac{\partial \varvec{\beta }_{(K)}^*}{\partial \varvec{\beta }_{(1)}^\top } &{}\quad \frac{\partial \varvec{\beta }_{(K)}^*}{\partial \varvec{\beta }_{(2)}^\top } &{}\quad \cdots &{}\quad \frac{\partial \varvec{\beta }_{(K)}^*}{\partial \varvec{\beta }_{(K)}^\top } \\ \end{pmatrix} \begin{pmatrix} \varvec{\varOmega }_{(1)} &{}\quad 0 &{}\quad \cdots &{}\quad 0 \\ 0 &{}\quad \varvec{\varOmega }_{(2)} &{}\quad \cdots &{}\quad 0 \\ \vdots &{}\quad \vdots &{}\quad \ddots &{}\quad \vdots \\ 0 &{}\quad 0 &{}\quad \cdots &{}\quad \varvec{\varOmega }_{(K)} \end{pmatrix} \begin{pmatrix} \frac{\partial \varvec{\beta }_{(1)}^\top }{\partial \varvec{\beta }_{(1)}} &{}\quad \frac{\partial {\varvec{\beta }_{(2)}^*}^\top }{\partial \varvec{\beta }_{(1)}} &{}\quad \cdots &{}\quad \frac{\partial {\varvec{\beta }_{(K)}^*}^\top }{\partial \varvec{\beta }_{(1)}} \\ \frac{\partial \varvec{\beta }_{(1)}^\top }{\partial \varvec{\beta }_{(2)}} &{}\quad \frac{\partial {\varvec{\beta }_{(2)}^*}^\top }{\partial \varvec{\beta }_{(2)}} &{}\quad \cdots &{}\quad \frac{\partial {\varvec{\beta }_{(K)}^*}^\top }{\partial \varvec{\beta }_{(2)}} \\ \vdots &{}\quad \vdots &{}\quad \ddots &{}\quad \vdots \\ \frac{\partial \varvec{\beta }_{(1)}^\top }{\partial \varvec{\beta }_{(K)}} &{}\quad \frac{\partial {\varvec{\beta }_{(2)}^*}^\top }{\partial \varvec{\beta }_{(K)}} &{}\quad \cdots &{}\quad \frac{\partial {\varvec{\beta }_{(K)}^*}^\top }{\partial \varvec{\beta }_{(K)}} \\ \end{pmatrix}\\&= \begin{pmatrix} \varvec{\varOmega }_{(1)} &{}\quad \varvec{\varOmega }_{(1)} \frac{\partial {\varvec{\beta }_{(2)}^*}^\top }{\partial \varvec{\beta }_{(1)}} &{}\quad \cdots &{}\quad \varvec{\varOmega }_{(1)} \frac{\partial {\varvec{\beta }_{(K)}^*}^\top }{\partial \varvec{\beta }_{(1)}} \\ \frac{\partial \varvec{\beta }_{(2)}^*}{\partial \varvec{\beta }_{(1)}^\top }\varvec{\varOmega }_{(1)} &{}\quad \frac{\partial \varvec{\beta }_{(2)}^*}{\partial \varvec{\beta }_{(1)}^\top }\varvec{\varOmega }_{(1)}\frac{\partial {\varvec{\beta }_{(2)}^*}^\top }{\partial \varvec{\beta }_{(1)}}+ \frac{\partial \varvec{\beta }_{(2)}^*}{\partial \varvec{\beta }_{(2)}^\top } \varvec{\varOmega }_{(2)} \frac{\partial {\varvec{\beta }_{(2)}^*}^\top }{\partial \varvec{\beta }_{(2)}} &{}\quad \cdots &{}\quad \frac{\partial \varvec{\beta }_{(2)}^*}{\partial \varvec{\beta }_{(1)}^\top }\varvec{\varOmega }_{(1)}\frac{\partial {\varvec{\beta }_{(K)}^*}^\top }{\partial \varvec{\beta }_{(1)}} \\ \vdots &{}\quad \vdots &{}\quad \ddots &{}\quad \vdots \\ \frac{\partial \varvec{\beta }_{(K)}^*}{\partial \varvec{\beta }_{(1)}^\top }\varvec{\varOmega }_{(1)} &{}\quad \frac{\partial \varvec{\beta }_{(K)}^*}{\partial \varvec{\beta }_{(1)}^\top }\varvec{\varOmega }_{(1)}\frac{\partial {\varvec{\beta }_{(2)}^*}^\top }{\partial \varvec{\beta }_{(1)}} &{}\quad \cdots &{}\quad \frac{\partial \varvec{\beta }_{(K)}^*}{\partial \varvec{\beta }_{(1)}^\top }\varvec{\varOmega }_{(1)}\frac{\partial {\varvec{\beta }_{(K)}^*}^\top }{\partial \varvec{\beta }_{(1)}}+ \frac{\partial \varvec{\beta }_{(K)}^*}{\partial \varvec{\beta }_{(K)}^\top } \varvec{\varOmega }_{(K)} \frac{\partial {\varvec{\beta }_{(K)}^*}^\top }{\partial \varvec{\beta }_{(K)}} \end{pmatrix}, \end{aligned}$$

since $\frac{\partial \varvec{\beta }_{(1)}}{\partial \varvec{\beta }_{(1)}^\top }$ is the identity matrix, $\frac{\partial \varvec{\beta }_{(1)}}{\partial \varvec{\beta }_{(k)}^\top }=0$ for all $k \ne 1$ and $\frac{\partial \varvec{\beta }_{(k)}^*}{\partial \varvec{\beta }_{(h)}^\top }=0$ for all $h \ne k$ with $h\ne 1$. The blocks on the main diagonal of $\varvec{\varOmega }$ are then

$$\begin{aligned} \mathsf {COV}\left( \varvec{\beta }_{(k)}^*\right) = \frac{\partial \varvec{\beta }_{(k)}^*}{\partial \varvec{\beta }_{(1)}^\top }\varvec{\varOmega }_{(1)}\frac{\partial {\varvec{\beta }_{(k)}^*}^\top }{\partial \varvec{\beta }_{(1)}}+ \frac{\partial \varvec{\beta }_{(k)}^*}{\partial \varvec{\beta }_{(k)}^\top } \varvec{\varOmega }_{(k)} \frac{\partial {\varvec{\beta }_{(k)}^*}^\top }{\partial \varvec{\beta }_{(k)}} , \end{aligned}$$

while the matrices outside the main diagonal are given by

$$\begin{aligned} \mathsf {COV}\left( \varvec{\beta }_{(1)},\varvec{\beta }_{(k)}^*\right) = \varvec{\varOmega }_{(1)}\frac{\partial {\varvec{\beta }_{(k)}^*}^\top }{\partial \varvec{\beta }_{(1)}}, \end{aligned}$$

and

$$\begin{aligned} \mathsf {COV}\left( \varvec{\beta }_{(h)}^*,\varvec{\beta }_{(k)}^*\right) = \frac{\partial \varvec{\beta }_{(h)}^*}{\partial \varvec{\beta }_{(1)}^\top }\varvec{\varOmega }_{(1)} \frac{\partial {\varvec{\beta }_{(k)}^*}^\top }{\partial \varvec{\beta }_{(1)}}. \end{aligned}$$

The chain rule can be exploited to find the derivatives

$$\begin{aligned} \frac{\partial \varvec{\beta }_{(k)}^*}{\partial \left( \varvec{\beta }_{(1)}^\top ,\varvec{\beta }_{(k)}^\top \right) } = \frac{\partial \varvec{\beta }_{(k)}^*}{\partial \left( \varvec{\beta }_{(k)}^\top , \hat{A}_k, \hat{B}_k\right) } \frac{\partial \left( \varvec{\beta }_{(k)}^\top , \hat{A}_k, \hat{B}_k\right) ^\top }{\partial \left( \varvec{\beta }_{(1)}^\top ,\varvec{\beta }_{(k)}^\top \right) }, \end{aligned}$$

(B1)

where

$$\begin{aligned} \frac{\partial \left( \hat{A}_k, \hat{B}_k\right) ^\top }{\partial \left( \varvec{\beta }_{(1)}^\top ,\varvec{\beta }_{(k)}^\top \right) } = \frac{\partial \left( \hat{A}_k, \hat{B}_k\right) ^\top }{\partial \left( \mathbf{v}_{(1)}^\top , \mathbf{v}_{(k)}^\top \right) } \frac{\partial \left( \mathbf{v}_{(1)}^\top , \mathbf{v}_{(k)}^\top \right) ^\top }{\partial \left( \varvec{\beta }_{(1)}^\top ,\varvec{\beta }_{(k)}^\top \right) }, \end{aligned}$$

(B2)

where $\mathbf{v}_{(k)}=(\mathbf{v}_{1k}^\top ,\dots , \mathbf{v}_{Jk}^\top )^\top$. The non-zero derivatives entering in (B1) and (B2) are given in the following (derivatives of a variable with respect to itself are not shown):

$$\begin{aligned} \frac{\partial \hat{\beta }_{1jk}^*}{\partial \hat{\beta }_{1jk}}= & {} 1, \quad \frac{\partial \hat{\beta }_{1jk}^*}{\partial \hat{\beta }_{2jk}} = -\frac{\hat{B}_k}{\hat{A}_k}, \quad \frac{\partial \hat{\beta }_{1jk}^*}{\partial \hat{A}_k} = \hat{\beta }_{2jk}\frac{\hat{B}_k}{\hat{A}_k^2}, \\ \frac{\partial \hat{\beta }_{1jk}^*}{\partial \hat{B}_k}= & {} - \frac{\hat{\beta }_{2jk}}{\hat{A}_k}, \quad \frac{\partial \hat{\beta }_{2jk}^*}{\partial \hat{\beta }_{2jk}} = \frac{1}{\hat{A}_k}, \quad \frac{\partial \hat{\beta }_{2jk}^*}{\partial \hat{A}_k} = -\frac{\hat{\beta }_{2jk}}{\hat{A}_k^2} \\ \frac{\partial \hat{a}_{jk}}{\partial \hat{\beta }_{2jk}}= & {} \frac{1}{D}, \quad \frac{\partial \hat{b}_{jk}}{\partial \hat{\beta }_{1jk}} = -\frac{1}{\hat{\beta }_{2j1}}, \quad \frac{\partial \hat{b}_{jk}}{\partial \hat{\beta }_{2jk}} = \frac{\hat{\beta }_{1jk}}{\hat{\beta }_{2jk}^2}. \end{aligned}$$

The derivatives $\frac{\partial ( \hat{A}_k, \hat{B}_k)^\top }{\partial (\mathbf{v}_{(1)}^\top ,\mathbf{v}_{(k)}^\top )}$ are given in Ogasawara (2000, 2001).