Adjusted Residuals for Evaluating Conditional Independence in IRT Models for Multistage Adaptive Testing

Abstract

The key assumption of conditional independence of item responses given latent ability in item response theory (IRT) models is addressed for multistage adaptive testing (MST) designs. Routing decisions in MST designs can cause patterns in the data that are not accounted for by the IRT model. This phenomenon relates to quasi-independence in log-linear models for incomplete contingency tables and impacts certain types of statistical inference based on assumptions on observed and missing data. We demonstrate that generalized residuals for item pair frequencies under IRT models as discussed by Haberman and Sinharay (J Am Stat Assoc 108:1435–1444, 2013. https://doi.org/10.1080/01621459.2013.835660) are inappropriate for MST data without adjustments. The adjustments are dependent on the MST design, and can quickly become nontrivial as the complexity of the routing increases. However, the adjusted residuals are found to have satisfactory Type I errors in a simulation and illustrated by an application to real MST data from the Programme for International Student Assessment (PISA). Implications and suggestions for statistical inference with MST designs are discussed.

Appendices

Appendix A: R Code

In this appendix, R code to compute the adjusted residuals is presented. First, we need the item-response functions for the 2PL:

Next, the Lord-Wingersky algorithm is needed (code works for dichotomous items only):

Then, we can create a function for the adjusted residuals for the basic MST design, which is shown below. To compute the adjusted residuals, several pieces of output from the MIRT program (Haberman, 2013) are needed. These are the estimated item parameters, the individual gradients, the individual posterior distributions of $\theta$, and the estimated asymptotic covariance matrix of item parameters. Since the MIRT program can write each piece of output to a separate csv-file, it is straightforward to obtain them and read them into R (see the manual on https://github.com/EducationalTestingService/MIRT).

Table 3 Mean (SD) simulation results with respect to parameter recovery (200 replications).

Fig. 7

Distributions of $Q_3$ statistic across all item pairs under different designs ($N=6000$, $J=45$).

Fig. 8

Heatmaps of average $Q_3$ statistic under complete and basic MST designs ($N=6000$, $J=45$).

Appendix B: Additional Simulation Results

Table 3 shows the mean results with respect to recovery of item parameters over 200 replications. The table displays the mean and standard deviation (SD) of average bias and RMSE of intercept and slope parameters for each simulation condition. The means were computed by, for each replication, averaging the bias or RMSE across the parameters, and then averaging across replications. The SDs were computed by taking the standard deviation across replications of these means for each replication. In addition, the mean and SD of bias, RMSE, and reliability are shown for EAP ability estimates. In the last column, the logarithm of the determinant of the item parameter information matrix is shown.

Apart from the results for the complete design, which serve as a reference, the results with respect to item parameter recovery are best for the random design. For $N=600$ and $J=9$, there are substantial differences in bias in slope between the designs with the basic MST producing the largest bias. However, the differences in bias between the designs for the other conditions are relatively small. This could be due to the dependencies being relatively larger for smaller J in the basic MST design. For the RMSE, the random design produces the smallest values, although the results for the balanced MST design are generally close. The observation that the random design and the balanced MST design produce similar item parameter recovery is supported by the similar values for the log determinant of the item parameter information matrix. For the basic MST, the results for the intercept seem somewhat surprising, but the intercept parameter should not be confused with the difficulty parameter. That is, the adaptivity targets the difficulty, not the intercept.

With respect to ability estimation, differences in bias between the designs are negligible. For the RMSE and reliability, apart from the complete design, the basic and balanced MST designs produce the best values. In these simulations, the balanced MST design seems to have the best of both worlds in terms item and ability parameter recovery.

Figure 7 shows the distribution of the $Q_3$ statistic across all item pairs under the different designs with $N=6000$ and $J=45$ using either the posterior or the WLE (Warm, 1989). For the posterior-based $Q_3$, there appears to be a negative bias, which is claimed to be approximately $-1/(J-1)$ (Yen, 1993) but appears to be slightly closer to zero here. For the WLE-based $Q_3$, this bias does not seem to appear for complete and random designs. The basic MST design does seem to result in a small negative bias.

Although the above $Q_3$ distributions are not that strange, there are interactions between how the $Q_3$ is computed and which design is used. These can be revealed using heatmaps for the average $Q_3$ across replications for item pairs, which are shown in Fig. 8. The top row shows the heatmap when sorted by item difficulty and the bottom row shows the heatmap when sorted by module in the basic MST. Clearly, the $Q_3$ computation and the design interact, although the range of these average $Q_3$ statistics is actually not that large (see the legend).

Table 4 shows summary statistics of the distribution of the $M_2$ statistic in the conditions with $N=6000$ and $J=45$ as computed by the R package mirt (Chalmers, 2012). Since the mirt package can only compute the statistic for complete data, we only show results based on items in the routing and easy modules for the complete design and the basic MST design (so that $J=30$).

Table 4 Summary statistics of $M_2$ statistic for items in routing and easy modules in complete and basic MST design ($N=6000$ and $J=30$).

Figures 9 and 10 show QQ plots of the generalized residuals for item pair frequencies under the different designs for the two conditions with $N=600$ and $N=6000$ with $J=9$. The QQ plots for the other two conditions with $J=45$ are not shown as the images are very large.

Fig. 9

QQ plots of residuals ($N=600$, $J=9$).

Fig. 10

QQ plots of residuals ($N=6000$, $J=9$).