Examining Differential Item Functioning from a Multidimensional IRT Perspective

Abstract

Differential item functioning (DIF) is a standard analysis for every testing company. Research has demonstrated that DIF can result when test items measure different ability composites, and the groups being examined for DIF exhibit distinct underlying ability distributions on those composite abilities. In this article, we examine DIF from a two-dimensional multidimensional item response theory (MIRT) perspective. We begin by delving into the compensatory MIRT model, illustrating and how items and the composites they measure can be graphically represented. Additionally, we discuss how estimated item parameters can vary based on the underlying latent ability distributions of the examinees. Analytical research highlighting the consequences of ignoring dimensionally and applying unidimensional IRT models, where the two-dimensional latent space is mapped onto a unidimensional, is reviewed. Next, we investigate three different approaches to understanding DIF from a MIRT standpoint: 1. Analytically Uniform and Nonuniform DIF: When two groups of interest have different two-dimensional ability distributions, a unidimensional model is estimated. 2. Accounting for complete latent ability space: We emphasize the importance of considering the entire latent ability space when using DIF conditional approaches, which leads to the mitigation of DIF effects. 3. Scenario-Based DIF: Even when underlying two-dimensional distributions are identical for two groups, differing problem-solving approaches can still lead to DIF. Modern software programs facilitate routine DIF procedures for comparing response data from two identified groups of interest. The real challenge is to identify why DIF could occur with flagged items. Thus, as a closing challenge, we present four items (Appendix A) from a standardized test and invite readers to identify which group was favored by a DIF analysis.

Appendices

Appendix A

Groups indicated as being favored in the Mantel–Haenszel analysis.

• Item 1: Male examinees

• Item 2: Black examinees

• Item 3: No DIF

• Item 4: Male examinees. This is the only item which has a possible explanation: that males, for the most part, know more about cars than females.

Appendix B

Example illustrating formulation of how the unidimensional 2PL model gets mapped into a two-dimensional latent ability space:

$$\begin{aligned} \varepsilon _{\upsilon _{2}}[P(u=1\left| \upsilon _{1},\upsilon _{2}) \right| \upsilon _{1}]=\int _{-\infty }^{+\infty } {P\left( u=1{\vert \upsilon }_{1},\upsilon _{2}){G(\upsilon }_{2}\vert \upsilon _{1} \right) \textrm{d}\upsilon _{2}.} \end{aligned}$$

Assume you want to find the unidimensional 2PL $\hat{a}$ and $\hat{b}$ value for a two-item test where the two-dimensional compensatory parameters are given as $A = [\{1.5,0\}, \{0,1.5\}]$ and $D = \{.5,.5\}$ and the underlying model is given as

$$\begin{aligned} P\left( {u_{ij}=1}\vert {\theta _{1j},\theta _{2j},a_{1i},a_{2i},d_{i}}\right) =\frac{1.0}{1.0+e^{-1.7(a_{1i}\theta _{1j}+a_{2i}\theta _{2j}+d_{i})}}. \end{aligned}$$

It is also given that the underlying two-dimensional distribution is a bivariate normal with a mean vector of {0,0}and the covariance matrix, $\Omega$, as [{1,.4}, {.4,1}]. Note these are chosen only for illustration purposes. Item 1 measures only $\theta _{1}$ and item 2 measures only $\theta _{2}$.

Following the work of Wang (1986) and Camilli (1992), we first determine the Cholesky decomposition, L, of $\Omega$. L[$\{1.,0.4\},\{0.,0.91651\}]$. To compute the reference composite, we first need to calculate the L’A’AL matrix which equals, $\{2.25,0.9\},\{0.9,2.25\}$. The eigenvalues of this matrix are $3.15,1.35\}$ and the eigenvectors associated with the eigenvalues, w$_{\textrm{ij}}$, are $\{0.7071,0.7071\},\{-0.7071,0.7071\}$.

The reference composite is then calculated as the arccosine of the first element of the eigenvector associated with the largest eigenvalue. The arccosine of.7071 corresponds to 45$^\circ$ which corresponds to the reference composite direction from the positive $\theta _{\textrm{1}}$-axis. This is the composite that would represent the unidimensional $\theta$-scale if the data were fit to the 2PL model.

It should also be noted that the first and second factor scores, $\upsilon _{1}$ and $\upsilon _{2}$ are then defined as:

$$\begin{aligned} \left( {\begin{array}{*{20}c} \upsilon _{1}\\ \upsilon _{2}\\ \end{array} } \right) =\left[ {\begin{array}{*{20}c} w_{11\left( \theta _{1}-\mu _{\theta _{1}} \right) }+w_{12\left( \theta _{1}-\mu _{\theta _{1}} \right) }\\ w_{21\left( \theta _{1}-\mu _{\theta _{1}} \right) }+w_{22\left( \theta _{2}-\mu _{\theta _{2}} \right) }\\ \end{array} } \right] =\left[ {\begin{array}{*{20}c}.7071\theta _{1}+.7071\theta _{2}\\ -.7071\theta _{1}+.7071\theta _{2}\\ \end{array} } \right] \end{aligned}$$

In Fig. 20, the left panel is a contour plot of Item 1 with the reference composite ($\upsilon _{\textrm{1}})$ direction indicated with a solid red arrow and the perpendicular $\upsilon _{\textrm{2}}$ direction indicated with a dotted red arrow. We then substitute $\upsilon _{\textrm{1}}$ and $\upsilon _{\textrm{2 }}$ in for $\theta _{\textrm{1}}$ and $\theta _{\textrm{2 in}}$ the compensatory model to get

$$\begin{aligned} p\left( {u_{ij}=1}\vert {\upsilon _{1},\upsilon _{2}}\right) =\frac{1.0}{1.0+e^{-1.7(1.5 \upsilon _{1}+.0\upsilon _{2}+.5)}}. \end{aligned}$$

Fig. 20

The contour graph of the original item response surface with direction of first ($\nu _{1})$ and second principal component ($\nu _{2})$ (left) and contour surface rotated 45$^\circ$ (right).

To determine G ($\upsilon$2|$\upsilon$1), we must first rotate the bivariate normal distribution 45o and then determine the conditional distribution. Assuming $\Sigma$ is the original covariance ${\Sigma }=\left[ {\begin{array}{*{20}c} \sigma _{1}^{2} &{} \rho \sigma _{1}\sigma _{2}\\ \rho \sigma _{1}\sigma _{2} &{} \sigma _{2}^{2}\\ \end{array} } \right]$and R$_{{\uptheta }}$ is the rotation matrix,

$$\begin{aligned} R_{\theta }=\left[ {\begin{array}{*{20}c} \left( \cos {45}^\circ \right) { - (}\sin {45}^\circ \mathrm {) }\\ \left( \sin {45}^\circ \right) {+ ( }\cos {45}^\circ \mathrm {) }\\ \end{array} } \right] =\left[ {\begin{array}{*{20}c} \frac{\sqrt{2} }{2} &{} \frac{-\sqrt{2} }{2}\\ \frac{\sqrt{2} }{2} &{} \frac{\sqrt{2} }{2}\\ \end{array} } \right] \end{aligned}$$

then the rotated mean vector, $\mu$’, and rotated covariance matrix, $\Sigma$’, are given by $\mu ^{'}=R_{\theta }\mu =\frac{\sqrt{2} }{2} \left[ {\begin{array}{*{20}c} \mu _{1}-\mu _{2}\\ \mu _{1}+\mu _{2}\\ \end{array} } \right]$ and ${\Sigma }^{'}=R_{\theta }{\Sigma }R_{\theta }^{T}= \frac{1}{2}\left[ {\begin{array}{*{20}c} \sigma _{1}^{2}+\sigma _{2}^{2}-2\rho \sigma _{1}\sigma _{2} &{} \sigma _{1}^{2}-\sigma _{2}^{2}\\ \sigma _{1}^{2}-\sigma _{2}^{2} &{} \sigma _{1}^{2}+\sigma _{2}^{2}+2{\rho \sigma }_{1}\sigma _{2}\\ \end{array} } \right] ,$ where $\sigma _{1}^{2}$, $\sigma _{2}^{2}$ and $\rho$ are the original variances and correlation of the original random variables.

The rotated mean vector is [0,0] and the rotated covariance matrix, ${\Sigma }^{'}$ is [{.6,0}, {0,1.4}]. The formula for the conditional distribution of G($\upsilon$1|$\upsilon$2) equals (Fig. 21)

$$\begin{aligned} G\left( {{\upupsilon 1}}\vert {{\upupsilon 2}}\right) =\sim N\left( \mu _{Y}+\rho \frac{\sigma _{Y}}{\sigma _{X}}\left( x-\mu _{X} \right) ,\sigma _{Y}^{2}\left( 1-\rho ^{2} \right) \right) =\sim N\left( 0,1.4 \right) . \end{aligned}$$

Fig. 21

A contour plot of the original bivariate normal distribution (left) and the contour plot of the rotated distribution (right).

In Fig. 22 on the left are conditional normal distributions, $G\left( {{\upupsilon 1}}\vert {{\upupsilon 2}}\right) ,$ for $\upsilon$1 $=$ -2, -1,0,1,2. On the right are the conditional ICCs, $p\left( {u_{ij}=1}\vert {\upsilon _{1},\upsilon _{2}}\right)$, for $\upsilon$1 $=$ -2, -1,0,1,2.

Fig. 22

Conditional normal distributions, $G\left( {{\upupsilon 1}}\vert {{\upupsilon 2}}\right) ,$ for $\upsilon$1 $=$ -2, -1,0,1,2 (left) and conditional ICCs, $p\left( {u_{ij}=1}\vert {\upsilon _{1},\upsilon _{2}}\right)$, for $\upsilon$1 $=$ -2, -1, 0, 1, 2 (right).

Using the formula

$$\begin{aligned} \left( {u_{ij}=1}\vert {\upsilon _{1},\upsilon _{2}}\right) =\int _{-6}^{+6} {P(u=1{\vert \upsilon }_{1},\upsilon _{2}){G(\upsilon }_{2}} \vert \upsilon _{1})\textrm{d}\upsilon _{2} \end{aligned}$$

4where $\textrm{d}\upsilon _{2}=$.001 we can estimate the unidimensional ICC for values of $\upsilon$1 values of -2, -1,0,1,2. These values are.13,.34,.64,.86, and.95, respectively. Using Camilli’s derivational formulas,

$$\begin{aligned} \hat{a}_{j}=\frac{\varvec{a}_{j}^{'}\varvec{W}_{1}}{\sqrt{2.89+\varvec{a}_{j}^{'}{\varvec{W}}_{2}\varvec{W}_{2}^{'}\varvec{a}_{j}} } \end{aligned}$$

and

$$\begin{aligned} \hat{b}_{j}\mathrm {=}\frac{d_{j}-a_{j}^{'}\varvec{\mu }}{\varvec{a}_{j}\varvec{W}_{1}}. \end{aligned}$$

we obtain the 2PL item parameter estimates: $\hat{a}= .73$ and $\hat{b}=-.47.$ Fig. 23 shows the estimated ICC using the $\hat{a}$ and $\hat{b}$ and the five color-coded estimated ($\upsilon _{1},p)$ values. Figure 24 illustrates three different perspectives of all the elements of Camilli’s formulation, including the M2PL response surface and corresponding contour plot, the RC (v1) which represents the estimated unidimensional scale, v2 (the orthogonal second principal component, the RC plane, the underlying conditional latent ability distribution, $G\left( {{\upupsilon 1}}\vert {{\upupsilon 2}}\right)$, and the estimated unidimensional ICC.

Fig. 23

Estimated unidimensional ICC with five estimated ($\upsilon _{1},p)$ values plotted.

Fig. 24

Three different perspectives of different elements that were used in the mapping of the two-dimensional compensatory model onto a unidimensional ICC.

Appendix C

Ackerman and Xie (2019) created a DIF Graphical Simulator. This simulator enables researchers to modify the underlying two-dimensional latent distributions for the Reference and Focal groups and the M2PL item parameters for a given suspect item. Using the Camilli (1992) analytical derivations, the 2PL unidimensional discrimination (a) and difficulty (b) parameters are estimated and the resulting ICC is illustrated. A mean–mean transformation is used to place the Focal group’s estimated parameters onto the scale of the Reference group. The transformed ICCs are then displayed, and the degree of misfit, defined as: $\sum \nolimits _{\theta =-3}^{\theta =3} {({P(\theta )}_{Ref}-{P(\theta )}_{Foc})}^{2}$, is calculated. The DIF Graphical Simulator is shown in Fig. 25

Fig. 25

The graphical display is shown by the DIF Graphical Simulator.