Sequential Detection of Compromised Items Using Response Times in Computerized Adaptive Testing

Abstract

Item compromise persists in undermining the integrity of testing, even secure administrations of computerized adaptive testing (CAT) with sophisticated item exposure controls. In ongoing efforts to tackle this perennial security issue in CAT, a couple of recent studies investigated sequential procedures for detecting compromised items, in which a significant increase in the proportion of correct responses for each item in the pool is monitored in real time using moving averages. In addition to actual responses, response times are valuable information with tremendous potential to reveal items that may have been leaked. Specifically, examinees that have preknowledge of an item would likely respond more quickly to it than those who do not. Therefore, the current study proposes several augmented methods for the detection of compromised items, all involving simultaneous monitoring of changes in both the proportion correct and average response time for every item using various moving average strategies. Simulation results with an operational item pool indicate that, compared to the analysis of responses alone, utilizing response times can afford marked improvements in detection power with fewer false positives.

Appendix

1.1 Application of Lyapunov’s Central Limit Theorem

Assume that log RT is normally distributed as follows: $\log T_{ij} \sim \mathcal {N}(\mu _{ij},\sigma _j^2)$, where $\mu _{ij}=\beta _j-\tau _i$ and $\sigma _j^2=1/\alpha _j^2$. The mean log RT of the moving sample for item j is then given as $\hat{\mu }_j^{(m)}=\dfrac{1}{m}\sum \nolimits _{i=n-m+1}^n \log T_{ij}$. Also, define the following: $s_m^2=\sum \nolimits _{i=n-m+1}^n \sigma _j^2=m\sigma _j^2$. In this context, Lyapunov’s CLT states that

$$\begin{aligned} \dfrac{1}{s_m}\sum \limits _{i=n-m+1}^n(\log T_{ij}-\mu _{ij}) = \dfrac{\hat{\mu }_j^{(m)}-\sum \nolimits _{i=n-m+1}^n\mu _{ij}/m}{\sigma _j/\sqrt{m}} \; {\mathop {\longrightarrow }\limits ^{\text{ d }}} \; \mathcal {N}(0,1) \end{aligned}$$

(A.1)

if, for any $\delta >0$, the following condition is met:

$$\begin{aligned} \lim _{m\rightarrow \infty }\dfrac{1}{s_m^{2+\delta }}\sum \limits _{i=n-m+1}^nE\left( |\log T_{ij}-\mu _{ij}|^{2+\delta }\right) =0. \end{aligned}$$

(A.2)

Recognizing that the expectation term is a central absolute moment of $\log T_{ij}$,

$$\begin{aligned} E\left( |\log T_{ij}-\mu _{ij}|^{2+\delta }\right) = \sigma _j^{2+\delta }(1+\delta )!! \cdot {\left\{ \begin{array}{ll} \sqrt{2/\pi } &{} \mathrm {if} \; 2+\delta \; \mathrm {is \; odd} \\ \;\;\;\; 1 &{} \mathrm {if} \; 2+\delta \; \mathrm {is \; even} \end{array}\right. }. \end{aligned}$$

(A.3)

Therefore, using $\delta =2$ for simplicity,

$$\begin{aligned} \lim _{m\rightarrow \infty }\dfrac{1}{s_m^{4}}\sum \limits _{i=n-m+1}^nE\left( |\log T_{ij}-\mu _{ij}|^{4}\right) =&\lim _{m\rightarrow \infty }\dfrac{1}{m^2\sigma _j^4}\sum \limits _{i=n-m+1}^n3\sigma _j^{4} \\ =&\lim _{m\rightarrow \infty }\dfrac{m\left( 3\sigma _j^{4}\right) }{m^2\sigma _j^4} \\ =&\lim _{m\rightarrow \infty }\dfrac{3}{m} \\ =&\; 0, \end{aligned}$$

thereby meeting Lyapunov’s condition for the asymptotic normality of the test statistic.