Using Response Times and Response Accuracy to Measure Fluency Within Cognitive Diagnosis Models

Abstract

The recent “Every Student Succeed Act" encourages schools to use an innovative assessment to provide feedback about students’ mastery level of grade-level content standards. Mastery of a skill requires the ability to complete the task with not only accuracy but also fluency. This paper offers a new sight on using both response times and response accuracy to measure fluency with cognitive diagnosis model framework. Defining fluency as the highest level of a categorical latent attribute, a polytomous response accuracy model and two forms of response time models are proposed to infer fluency jointly. A Bayesian estimation approach is developed to calibrate the newly proposed models. These models were applied to analyze data collected from a spatial rotation test. Results demonstrate that compared with the traditional CDM that using response accuracy only, the proposed joint models were able to reveal more information regarding test takers’ spatial skills. A set of simulation studies were conducted to evaluate the accuracy of model estimation algorithm and illustrate the various degrees of model complexities.

Appendices

Appendix

Appendix I: Gibbs Algorithm for Parameter Updating

The proposed MCMC algorithm is used to sample from the posterior distribution of the model parameters. To do this, we first assigned initial values to all model parameters as follows:

• The initial population membership probabilities ${{\varvec{\pi }}}^{[0]}$ were randomly generated from $\text{ Dirichlet }(1,1,\ldots ,1)$.

• The initial ${\varvec{\alpha }}^{[0]}$ were randomly sampled from the discrete uniform distribution over all possible patterns.

• The initial $\phi _i^{[0]}$ was randomly generated from $\text{ Uniform }(0,1)$.

• For the DINA model item parameters, we randomly generated the $g_j^{[0]},s_{1j}^{[0]},s_{2j}^{[0]}$ for each item from $\text{ Uniform }(.1,.3)$.

• For the lognormal response time model parameters, we sampled $a_j^{[0]} \sim \text{ Uniform }(2,4),$ and $\gamma _j^{[0]}\sim N(3.45, .5^2).$

• For response time model (2), the initial variance ${\sigma ^2}^{[0]}_\tau$ was generated from Uniform (1,1.5). For each test taker i, $\tau ^{[0]}_{i}$ was generated from $N(0,{\sigma ^2}^{[0]}_\tau )$

• For response time model (3), the initial covariance matrix ${{\varvec{\Sigma }}}_{\tau _0\tau _1}^{[0]}$ was set to be the $2\times 2$ identity matrix. For each test taker i, $(\tau _{0i}^{[0]}, \tau _{1i}^{[0]})$ was randomly sampled from $MVN(\mathbf {0},{{\varvec{\Sigma }}}_{\tau _0\tau _1}^{[0]}).$

The following procedures were then used to update the parameters in the rth iteration of the MCMC chain.

• (1) For $i = 1,\ldots , N$, sample ${\varvec{\alpha }}_{i}^{[r+1]}$ from the multinomial distribution with probabilities $\tilde{\pi }_{ic}$ is given in Table 1.

• (2) For response time model (2), for $i = 1,\ldots , N$, $\tau _i^{[r+1]}$ is updated using a Gibbs step based on the conditional distribution specified in Table 1.

• (3) For response time model (3), for $i=1,\ldots ,N$, $(\tau _{0i}^{[r+1]},\tau _{1i}^{r+1})$ is updated based on the conditional distribution specified in Table 1.

• (4) Given a specified response time model, for $i = 1,\ldots , N$, obtain $\phi _i^{[r+1]}$ based on its conditional distribution in Table 1.

• (5) For response time model (2), using the conditional distribution in Table 1 and $\tau ^{r+1}_i$, update ${\sigma ^2}^{[r+1]}_{\tau }$ from the inverse gamma distribution.

• (6) For response time model (3), based on the conditional distribution in Table 1 and $({{\varvec{\tau }}}_0,{{\varvec{\tau }}}_{1})^{[r+1]}$, obtain ${{\varvec{\Sigma }}}^{[r+1]}_{\tau _0\tau _1}$ from the inverse Wishart distribution.

• (7) Based on ${\varvec{\alpha }}^{[r+1]},$ update ${{\varvec{\pi }}}^{[r+1]}$ according to the Dirichlet distribution in Table 1.

• (8) For $j = 1,\ldots , J$, sample $s_{2j}^{[r+1]}$ from the truncated beta distribution in Table 1, based on ${\varvec{\alpha }}^{[r+1]}, \mathbf{Y},$ and $s^{[r]}_{1j}$,$g_j^{[r]}$; and then sample $s^{[r+1]}_{j1}$ based on ${\varvec{\alpha }}^{[r+1]}, \mathbf{Y},$ $g_j^{[r]}$ and $s_{j2}^{[r+1]}$. Finally, update $g_j^{[r+1]}$ using the truncated beta distribution based on ${\varvec{\alpha }}^{[r+1]}, \mathbf{Y},$ $s_{j1}^{[r+1]}$ and $s_{j2}^{[r+1]}$.

• (9) Given the specified response time model, for $j = 1,\ldots , J$, sample $a^{[r+1]}$ from the inverse gamma distribution given in Table 1, based on $\varvec{L}, {{\varvec{\tau }}}^{[r+1]} (\text {or } ({{\varvec{\tau }}}_0,{{\varvec{\tau }}}_1)^{[r+1]}),{\varvec{\alpha }}^{[r+1]}, {{\varvec{\phi }}}^{[r]}$ and $\gamma _j^{[r]}$, and sample $\gamma _j^{[r+1]}$ from the normal distribution in Table 1, based on $\varvec{L}, {{\varvec{\tau }}}^{[r+1]}(\text {or } ({{\varvec{\tau }}}_0,{{\varvec{\tau }}}_1)^{[r+1]}),{\varvec{\alpha }}^{[r+1]}, {{\varvec{\phi }}}^{[r]}$ and $a_j^{[r+1]}.$

Appendix II: Supplementary Simulation Results

We report the results for the four models when $N=500, J=40$ and $N=1000, J=20$ in this appendix. Tables 9 and 10 document the latent attribute profile classification results and the recovery of $\phi _i$.

Table 9 Classification results of $\phi$ generation condition 1

Table 10 Classification results of $\phi$ generation condition 2

For recovery of latent speed parameter, the correlations between estimated and true $\tau _{i}$ for models 1 and 2 were always above 0.95, and the performance were comparable across all settings. For model 3 and 4, the correlations between estimated and true $\tau _{0i}$ were similar to models 1 and 2, and the correlations between estimated and true $\tau _{1i}$ were from 0.552 to 0.933. Model 4 performed better than model 3 across different settings.

For models 1 and 2, the RMSEs of estimated $\sigma _{\tau }$ are below 0.05, and the relative bias was always below 0.08. Both models performed better with a more idealized $\phi$ condition and were comparable under the two item conditions. For models 3 and 4, the range of the absolute bias of $\sigma _{00}$ was from 0.017 to 0.051 for model 3 and from 0.012 to 0.038 for model 4. The range of absolute bias of $\sigma _{01}$ (resp. $\sigma _{11}$) was from 0.013 to 0.062 (resp. 0.015–0.017) for model 3 and from 0.035 to 0.046 (resp. 0.012–0.016) for model 4. The estimation of $\sigma _{00}$ and $\sigma _{11}$ was better when N increased, whereas the performance of $\sigma _{01}$ improved when J increased. The estimation of response time model parameters $\gamma _j$ and $a_j$ were consistently good across all simulation conditions. The correlation of estimated and true parameter $a_j$ was greater than 98% for all situations, and the correlation of estimated and true parameter $\gamma _j$ was always greater than 95%. For both parameters, 99% of the relative absolute deviations were smaller than 0.2 for all situations.

The range of RMSEs of response model item parameter was from 0.021 to 0.070 for models 1 and 2 and from 0.021 to 0.042 for models 3 and 4. Given the same $\phi _i$ generation condition, the estimation of all item parameters was better with a more idealized true item parameter condition, that is $(g, s_1, s_2)=(0.1, 0.2, 0.1)$. Given the same true item parameter generation condition, the estimation of item parameters was better with the more idealized $\phi _i$ generation condition. All four models performed better with a larger item size J despite the smaller subject size N.

Appendix III: Model Fit of Four Simplified Fluency Models ($\phi _i=\phi$)

See Table 11.

Table 11 DIC of four simplified fluency models