On Wald tests for differential item functioning detection

Battauz, Michela

doi:10.1007/s10260-018-00442-w

Original Paper
Published: 01 October 2018

On Wald tests for differential item functioning detection

Michela Battauz ORCID: orcid.org/0000-0002-3098-689X¹

Statistical Methods & Applications volume 28, pages 103–118 (2019)Cite this article

421 Accesses
7 Citations
Metrics details

Abstract

Wald-type tests are a common procedure for DIF detection among the IRT-based methods. However, the empirical type I error rate of these tests departs from the significance level. In this paper, two reasons that explain this discrepancy will be discussed and a new procedure will be proposed. The first reason is related to the equating coefficients used to convert the item parameters to a common scale, as they are treated as known constants whereas they are estimated. The second reason is related to the parameterization used to estimate the item parameters, which is different from the usual IRT parameterization. Since the item parameters in the usual IRT parameterization are obtained in a second step, the corresponding covariance matrix is approximated using the delta method. The proposal of this article is to account for the estimation of the equating coefficients treating them as random variables and to use the untransformed (i.e. not reparameterized) item parameters in the computation of the test statistic. A simulation study is presented to compare the performance of this new proposal with the currently used procedure. Results show that the new proposal gives type I error rates closer to the significance level.

This is a preview of subscription content, access via your institution.

Access options

Buy single article

Instant access to the full article PDF.

USD 39.95

Price includes VAT (USA)
Tax calculation will be finalised during checkout.

References

Bartholomew DJ, Knott M, Moustaki I (2011) Latent variable models and factor analysis: a unified approach. Wiley, West Sussex
Book MATH Google Scholar
Battauz M (2015) equateIRT: an R package for IRT test equating. J. Stat. Softw. 68(7):1–22. https://doi.org/10.18637/jss.v068.i07
Article Google Scholar
Bock RD, Aitkin M (1981) Marginal maximum likelihood estimation of item parameters: application of an EM algorithm. Psychometrika 46(4):443–459. https://doi.org/10.1007/BF02293801
MathSciNet Article Google Scholar
Candell GL, Drasgow F (1988) An iterative procedure for linking metrics and assessing item bias in item response theory. Appl. Psychol. Meas. 12(3):253–260. https://doi.org/10.1177/014662168801200304
Article Google Scholar
Casella G, Berger RL (2002) Statistical inference. Duxbury, Pacific Grove
MATH Google Scholar
Gregory AW, Veall MR (1985) Formulating Wald tests of nonlinear restrictions. Econometrica 53(6):1465–1468. https://doi.org/10.2307/1913221
Article Google Scholar
Kim SH, Cohen AS, Kim HO (1994) An investigation of Lord’s procedure for the detection of differential item functioning. Appl. Psychol. Meas. 18(3):217–228. https://doi.org/10.1177/014662169401800303
Article Google Scholar
Kim SH, Cohen AS, Park TH (1995) Detection of differential item functioning in multiple groups. J. Educ. Meas. 32(3):261–276. https://doi.org/10.1111/j.1745-3984.1995.tb00466.x
Article Google Scholar
Kolen M, Brennan R (2014) Test equating, scaling, and linking: methods and practices, 3rd edn. Springer, New York
Book MATH Google Scholar
Lord FM (1980) Applications of item response theory to practical testing problems. Erlbaum, Hillsdale, NJ
Google Scholar
Magis D, Béland S, Tuerlinckx F, De Boeck P (2010) A general framework and an R package for the detection of dichotomous differential item functioning. Behav. Res. Methods 42(3):847–862. https://doi.org/10.3758/BRM.42.3.847
Article Google Scholar
Mislevy RJ (1986) Bayes modal estimation in item response models. Psychometrika 51(2):177–195. https://doi.org/10.1007/BF02293979
MathSciNet Article MATH Google Scholar
Ogasawara H (2000) Asymptotic standard errors of IRT equating coefficients using moments. Econ. Rev. (Otaru Univ. Commer.) 51(1):1–23
Google Scholar
Ogasawara H (2001) Standard errors of item response theory equating/linking by response function methods. Appl. Psychol. Meas. 25(1):53–67. https://doi.org/10.1177/01466216010251004
MathSciNet Article Google Scholar
Patz RJ, Junker BW (1999) Applications and extensions of MCMC in IRT: multiple item types, missing data, and rated responses. J. Educ. Behav. Stat. 24(4):342–366. https://doi.org/10.3102/10769986024004342
Article Google Scholar
R Development Core Team (2017) R: a language and environment for statistical computing. R Foundation for Statistical Computing, Vienna. http://www.R-project.org/, ISBN 3-900051-07-0
Reise SP, Revicki DA (2014) Handbook of item response theory modeling: applications to typical performance assessment. Routledge, New York
Book Google Scholar
Rizopoulos D (2006) ltm: an R package for latent variable modeling and item response theory analyses. J. Stat. Softw. 17(5):1–25. https://doi.org/10.18637/jss.v017.i05
Article Google Scholar
van der Linden W (2016) Handbook of item response theory, volume one: models. Chapman & Hall, Boca Raton
Book MATH Google Scholar

Download references

Acknowledgements

Funding was provided by Universitá degli Studi di Udine (Grant No. PRID 2017). This work was supported by PRID 2017, University of Udine.

Author information

Authors and Affiliations

Department of Economics and Statistics, University of Udine, Via Tomadini 30/A, 33100, Udine, Italy
Michela Battauz

Authors

Michela Battauz
View author publications
You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Michela Battauz.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (pdf 65 KB)

Appendices

Appendix A: Equating of untransformed item parameters

Equation (12) is obtained from Eqs. (7) and (4) as follows:

$$\begin{aligned} \hat{\beta }_{2jk}^* = D \hat{a}_{jk}^* = \frac{D \hat{a}_{jk}}{\hat{A}_k} = \frac{\hat{\beta }_{2jk}}{\hat{A}_k}. \end{aligned}$$

(A1)

Equations (7), (8) and (5) lead to Eq. (13):

$$\begin{aligned} \hat{\beta }_{1jk}^*= & {} - D \hat{a}_{jk}^* \hat{b}_{jk}^* = - D \frac{\hat{a}_{jk}}{\hat{A}_k} \left( \hat{A}_k \, \hat{b}_{jk} + \hat{B}_k\right) = - D \hat{a}_{jk} \hat{b}_{jk} - D \hat{a}_{jk} \frac{\hat{B}_k}{\hat{A}_k}\nonumber \\= & {} \hat{\beta }_{1jk}-\hat{\beta }_{2jk}\frac{\hat{B}_k}{\hat{A}_k}. \end{aligned}$$

(A2)

Appendix B: Covariance matrix of item parameters

The covariance matrix $\varvec{\varOmega }_j$ entering in Eq. (15) is a block matrix given by

$$\begin{aligned} \varvec{\varOmega }_j = \begin{pmatrix} \mathsf {COV}\left( \varvec{\beta }_{j1}\right) &{}\quad \mathsf {COV}\left( \varvec{\beta }_{j1},\varvec{\beta }_{j2}^*\right) &{}\quad \mathsf {COV}\left( \varvec{\beta }_{j1},\varvec{\beta }_{j3}^*\right) &{}\quad \dots &{}\quad \mathsf {COV}\left( \varvec{\beta }_{j1},\varvec{\beta }_{jK}^*\right) \\ \mathsf {COV}\left( \varvec{\beta }_{j2}^*,\varvec{\beta }_{j1}\right) &{}\quad \mathsf {COV}\left( \varvec{\beta }_{j2}^*\right) &{}\quad \mathsf {COV}\left( \varvec{\beta }_{j2}^*,\varvec{\beta }_{j3}^*\right) &{}\quad \dots &{}\quad \mathsf {COV}\left( \varvec{\beta }_{j2}^*,\varvec{\beta }_{jK}^*\right) \\ \mathsf {COV}\left( \varvec{\beta }_{j3}^*,\varvec{\beta }_{j1}\right) &{}\quad \mathsf {COV}\left( \varvec{\beta }_{j3}^*,\varvec{\beta }_{j2}^*\right) &{}\quad \mathsf {COV}\left( \varvec{\beta }_{j3}^*\right) &{}\quad \dots &{}\quad \mathsf {COV}\left( \varvec{\beta }_{j3}^*,\varvec{\beta }_{jK}^*\right) \\ \vdots &{}\quad \vdots &{}\quad \vdots &{}\quad \ddots &{}\quad \vdots \\ \mathsf {COV}\left( \varvec{\beta }_{jK}^*,\varvec{\beta }_{j1}\right) &{}\quad \mathsf {COV}\left( \varvec{\beta }_{jK}^*,\varvec{\beta }_{j2}^*\right) &{}\quad \mathsf {COV}\left( \varvec{\beta }_{jK}^*,\varvec{\beta }_{j3}^*\right) &{}\quad \dots &{}\quad \mathsf {COV}\left( \varvec{\beta }_{jK}^*\right) \end{pmatrix}. \end{aligned}$$

Let $\varvec{\beta }_{(k)}=(\varvec{\beta }_{1k}^\top , \dots ,\varvec{\beta }_{Jk}^\top )^\top $ denote the item parameters estimates in group k, and $\varvec{\varOmega }_{(k)} = \mathsf {COV}( \varvec{\beta }_{(k)})$ denote the covariance matrix of the item parameter estimates in group k, which is estimated along with the estimation of the item parameters. Using the delta method, it is possible to compute the covariance matrix $\varvec{\varOmega } = \mathsf {COV}(\varvec{\beta }_{(1)}^\top ,{\varvec{\beta }_{(2)}^*}^\top , \dots ,{\varvec{\beta }_{(K)}^*}^\top )^\top $, from which to extract $\varvec{\varOmega }_j$:

$$\begin{aligned} \varvec{\varOmega }&= \frac{\partial \left( \varvec{\beta }_{(1)}^\top , {\varvec{\beta }_{(2)}^*}^\top ,\dots ,{\varvec{\beta }_{(K)}^*}^\top \right) ^\top }{\partial \left( \varvec{\beta }_{(1)}^\top ,\varvec{\beta }_{(2)}^\top , \dots ,\varvec{\beta }_{(K)}^\top \right) }\mathsf {COV}\left( \left( \varvec{\beta }_{(1)}^\top , \varvec{\beta }_{(2)}^\top ,\dots ,\varvec{\beta }_{(K)}^\top \right) ^\top \right) \frac{\partial \left( \varvec{\beta }_{(1)}^\top , {\varvec{\beta }_{(2)}^*}^\top ,\dots ,{\varvec{\beta }_{(K)}^*}^\top \right) }{\partial \left( \varvec{\beta }_{(1)}^\top ,\varvec{\beta }_{(2)}^\top ,\dots , \varvec{\beta }_{(K)}^\top \right) ^\top } \\&= \begin{pmatrix} \frac{\partial \varvec{\beta }_{(1)}}{\partial \varvec{\beta }_{(1)}^\top } &{}\quad \frac{\partial \varvec{\beta }_{(1)}}{\partial \varvec{\beta }_{(2)}^\top } &{}\quad \cdots &{}\quad \frac{\partial \varvec{\beta }_{(1)}}{\partial \varvec{\beta }_{(K)}^\top } \\ \frac{\partial \varvec{\beta }_{(2)}^*}{\partial \varvec{\beta }_{(1)}^\top } &{}\quad \frac{\partial \varvec{\beta }_{(2)}^*}{\partial \varvec{\beta }_{(2)}^\top } &{}\quad \cdots &{}\quad \frac{\partial \varvec{\beta }_{(2)}^*}{\partial \varvec{\beta }_{(K)}^\top } \\ \vdots &{}\quad \vdots &{}\quad \ddots &{}\quad \vdots \\ \frac{\partial \varvec{\beta }_{(K)}^*}{\partial \varvec{\beta }_{(1)}^\top } &{}\quad \frac{\partial \varvec{\beta }_{(K)}^*}{\partial \varvec{\beta }_{(2)}^\top } &{}\quad \cdots &{}\quad \frac{\partial \varvec{\beta }_{(K)}^*}{\partial \varvec{\beta }_{(K)}^\top } \\ \end{pmatrix} \begin{pmatrix} \varvec{\varOmega }_{(1)} &{}\quad 0 &{}\quad \cdots &{}\quad 0 \\ 0 &{}\quad \varvec{\varOmega }_{(2)} &{}\quad \cdots &{}\quad 0 \\ \vdots &{}\quad \vdots &{}\quad \ddots &{}\quad \vdots \\ 0 &{}\quad 0 &{}\quad \cdots &{}\quad \varvec{\varOmega }_{(K)} \end{pmatrix} \begin{pmatrix} \frac{\partial \varvec{\beta }_{(1)}^\top }{\partial \varvec{\beta }_{(1)}} &{}\quad \frac{\partial {\varvec{\beta }_{(2)}^*}^\top }{\partial \varvec{\beta }_{(1)}} &{}\quad \cdots &{}\quad \frac{\partial {\varvec{\beta }_{(K)}^*}^\top }{\partial \varvec{\beta }_{(1)}} \\ \frac{\partial \varvec{\beta }_{(1)}^\top }{\partial \varvec{\beta }_{(2)}} &{}\quad \frac{\partial {\varvec{\beta }_{(2)}^*}^\top }{\partial \varvec{\beta }_{(2)}} &{}\quad \cdots &{}\quad \frac{\partial {\varvec{\beta }_{(K)}^*}^\top }{\partial \varvec{\beta }_{(2)}} \\ \vdots &{}\quad \vdots &{}\quad \ddots &{}\quad \vdots \\ \frac{\partial \varvec{\beta }_{(1)}^\top }{\partial \varvec{\beta }_{(K)}} &{}\quad \frac{\partial {\varvec{\beta }_{(2)}^*}^\top }{\partial \varvec{\beta }_{(K)}} &{}\quad \cdots &{}\quad \frac{\partial {\varvec{\beta }_{(K)}^*}^\top }{\partial \varvec{\beta }_{(K)}} \\ \end{pmatrix}\\&= \begin{pmatrix} \varvec{\varOmega }_{(1)} &{}\quad \varvec{\varOmega }_{(1)} \frac{\partial {\varvec{\beta }_{(2)}^*}^\top }{\partial \varvec{\beta }_{(1)}} &{}\quad \cdots &{}\quad \varvec{\varOmega }_{(1)} \frac{\partial {\varvec{\beta }_{(K)}^*}^\top }{\partial \varvec{\beta }_{(1)}} \\ \frac{\partial \varvec{\beta }_{(2)}^*}{\partial \varvec{\beta }_{(1)}^\top }\varvec{\varOmega }_{(1)} &{}\quad \frac{\partial \varvec{\beta }_{(2)}^*}{\partial \varvec{\beta }_{(1)}^\top }\varvec{\varOmega }_{(1)}\frac{\partial {\varvec{\beta }_{(2)}^*}^\top }{\partial \varvec{\beta }_{(1)}}+ \frac{\partial \varvec{\beta }_{(2)}^*}{\partial \varvec{\beta }_{(2)}^\top } \varvec{\varOmega }_{(2)} \frac{\partial {\varvec{\beta }_{(2)}^*}^\top }{\partial \varvec{\beta }_{(2)}} &{}\quad \cdots &{}\quad \frac{\partial \varvec{\beta }_{(2)}^*}{\partial \varvec{\beta }_{(1)}^\top }\varvec{\varOmega }_{(1)}\frac{\partial {\varvec{\beta }_{(K)}^*}^\top }{\partial \varvec{\beta }_{(1)}} \\ \vdots &{}\quad \vdots &{}\quad \ddots &{}\quad \vdots \\ \frac{\partial \varvec{\beta }_{(K)}^*}{\partial \varvec{\beta }_{(1)}^\top }\varvec{\varOmega }_{(1)} &{}\quad \frac{\partial \varvec{\beta }_{(K)}^*}{\partial \varvec{\beta }_{(1)}^\top }\varvec{\varOmega }_{(1)}\frac{\partial {\varvec{\beta }_{(2)}^*}^\top }{\partial \varvec{\beta }_{(1)}} &{}\quad \cdots &{}\quad \frac{\partial \varvec{\beta }_{(K)}^*}{\partial \varvec{\beta }_{(1)}^\top }\varvec{\varOmega }_{(1)}\frac{\partial {\varvec{\beta }_{(K)}^*}^\top }{\partial \varvec{\beta }_{(1)}}+ \frac{\partial \varvec{\beta }_{(K)}^*}{\partial \varvec{\beta }_{(K)}^\top } \varvec{\varOmega }_{(K)} \frac{\partial {\varvec{\beta }_{(K)}^*}^\top }{\partial \varvec{\beta }_{(K)}} \end{pmatrix}, \end{aligned}$$

since $\frac{\partial \varvec{\beta }_{(1)}}{\partial \varvec{\beta }_{(1)}^\top }$ is the identity matrix, $\frac{\partial \varvec{\beta }_{(1)}}{\partial \varvec{\beta }_{(k)}^\top }=0$ for all $k \ne 1$ and $\frac{\partial \varvec{\beta }_{(k)}^*}{\partial \varvec{\beta }_{(h)}^\top }=0$ for all $h \ne k$ with $h\ne 1$. The blocks on the main diagonal of $\varvec{\varOmega }$ are then

$$\begin{aligned} \mathsf {COV}\left( \varvec{\beta }_{(k)}^*\right) = \frac{\partial \varvec{\beta }_{(k)}^*}{\partial \varvec{\beta }_{(1)}^\top }\varvec{\varOmega }_{(1)}\frac{\partial {\varvec{\beta }_{(k)}^*}^\top }{\partial \varvec{\beta }_{(1)}}+ \frac{\partial \varvec{\beta }_{(k)}^*}{\partial \varvec{\beta }_{(k)}^\top } \varvec{\varOmega }_{(k)} \frac{\partial {\varvec{\beta }_{(k)}^*}^\top }{\partial \varvec{\beta }_{(k)}} , \end{aligned}$$

while the matrices outside the main diagonal are given by

$$\begin{aligned} \mathsf {COV}\left( \varvec{\beta }_{(1)},\varvec{\beta }_{(k)}^*\right) = \varvec{\varOmega }_{(1)}\frac{\partial {\varvec{\beta }_{(k)}^*}^\top }{\partial \varvec{\beta }_{(1)}}, \end{aligned}$$

and

$$\begin{aligned} \mathsf {COV}\left( \varvec{\beta }_{(h)}^*,\varvec{\beta }_{(k)}^*\right) = \frac{\partial \varvec{\beta }_{(h)}^*}{\partial \varvec{\beta }_{(1)}^\top }\varvec{\varOmega }_{(1)} \frac{\partial {\varvec{\beta }_{(k)}^*}^\top }{\partial \varvec{\beta }_{(1)}}. \end{aligned}$$

The chain rule can be exploited to find the derivatives

$$\begin{aligned} \frac{\partial \varvec{\beta }_{(k)}^*}{\partial \left( \varvec{\beta }_{(1)}^\top ,\varvec{\beta }_{(k)}^\top \right) } = \frac{\partial \varvec{\beta }_{(k)}^*}{\partial \left( \varvec{\beta }_{(k)}^\top , \hat{A}_k, \hat{B}_k\right) } \frac{\partial \left( \varvec{\beta }_{(k)}^\top , \hat{A}_k, \hat{B}_k\right) ^\top }{\partial \left( \varvec{\beta }_{(1)}^\top ,\varvec{\beta }_{(k)}^\top \right) }, \end{aligned}$$

(B1)

where

$$\begin{aligned} \frac{\partial \left( \hat{A}_k, \hat{B}_k\right) ^\top }{\partial \left( \varvec{\beta }_{(1)}^\top ,\varvec{\beta }_{(k)}^\top \right) } = \frac{\partial \left( \hat{A}_k, \hat{B}_k\right) ^\top }{\partial \left( \mathbf{v}_{(1)}^\top , \mathbf{v}_{(k)}^\top \right) } \frac{\partial \left( \mathbf{v}_{(1)}^\top , \mathbf{v}_{(k)}^\top \right) ^\top }{\partial \left( \varvec{\beta }_{(1)}^\top ,\varvec{\beta }_{(k)}^\top \right) }, \end{aligned}$$

(B2)

where $\mathbf{v}_{(k)}=(\mathbf{v}_{1k}^\top ,\dots , \mathbf{v}_{Jk}^\top )^\top $. The non-zero derivatives entering in (B1) and (B2) are given in the following (derivatives of a variable with respect to itself are not shown):

$$\begin{aligned} \frac{\partial \hat{\beta }_{1jk}^*}{\partial \hat{\beta }_{1jk}}= & {} 1, \quad \frac{\partial \hat{\beta }_{1jk}^*}{\partial \hat{\beta }_{2jk}} = -\frac{\hat{B}_k}{\hat{A}_k}, \quad \frac{\partial \hat{\beta }_{1jk}^*}{\partial \hat{A}_k} = \hat{\beta }_{2jk}\frac{\hat{B}_k}{\hat{A}_k^2}, \\ \frac{\partial \hat{\beta }_{1jk}^*}{\partial \hat{B}_k}= & {} - \frac{\hat{\beta }_{2jk}}{\hat{A}_k}, \quad \frac{\partial \hat{\beta }_{2jk}^*}{\partial \hat{\beta }_{2jk}} = \frac{1}{\hat{A}_k}, \quad \frac{\partial \hat{\beta }_{2jk}^*}{\partial \hat{A}_k} = -\frac{\hat{\beta }_{2jk}}{\hat{A}_k^2} \\ \frac{\partial \hat{a}_{jk}}{\partial \hat{\beta }_{2jk}}= & {} \frac{1}{D}, \quad \frac{\partial \hat{b}_{jk}}{\partial \hat{\beta }_{1jk}} = -\frac{1}{\hat{\beta }_{2j1}}, \quad \frac{\partial \hat{b}_{jk}}{\partial \hat{\beta }_{2jk}} = \frac{\hat{\beta }_{1jk}}{\hat{\beta }_{2jk}^2}. \end{aligned}$$

The derivatives $\frac{\partial ( \hat{A}_k, \hat{B}_k)^\top }{\partial (\mathbf{v}_{(1)}^\top ,\mathbf{v}_{(k)}^\top )}$ are given in Ogasawara (2000, 2001).

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Battauz, M. On Wald tests for differential item functioning detection. Stat Methods Appl 28, 103–118 (2019). https://doi.org/10.1007/s10260-018-00442-w

Download citation

Accepted: 22 September 2018
Published: 01 October 2018
Issue Date: 11 March 2019
DOI: https://doi.org/10.1007/s10260-018-00442-w

Keywords

Differential item functioning
False positive rate
Item response theory
Lord test
Type I error rate
Wald test

Mathematics Subject Classification

62 Statistics