Estimating the Cognitive Diagnosis $\varvec{Q}$ Matrix with Expert Knowledge: Application to the Fraction-Subtraction Dataset

Abstract

Cognitive diagnosis models (CDMs) are an important psychometric framework for classifying students in terms of attribute and/or skill mastery. The $\varvec{Q}$ matrix, which specifies the required attributes for each item, is central to implementing CDMs. The general unavailability of $\varvec{Q}$ for most content areas and datasets poses a barrier to widespread applications of CDMs, and recent research accordingly developed fully exploratory methods to estimate Q. However, current methods do not always offer clear interpretations of the uncovered skills and existing exploratory methods do not use expert knowledge to estimate Q. We consider Bayesian estimation of $\varvec{Q}$ using a prior based upon expert knowledge using a fully Bayesian formulation for a general diagnostic model. The developed method can be used to validate which of the underlying attributes are predicted by experts and to identify residual attributes that remain unexplained by expert knowledge. We report Monte Carlo evidence about the accuracy of selecting active expert-predictors and present an application using Tatsuoka’s fraction-subtraction dataset.

Appendices

Appendix A: Monte Carlo Estimates of Bias and Mean Absolute Deviation for the GDM $\varvec{\Theta }$ and $\varvec{\pi }$

See Tables A1, A2, A3, A4, A5, A6, A7, A8, A9 and A10 .

Table 6 Summary of bias for GDM item parameters, $\varvec{\Theta }$, using the MVN prior by latent class and item for $K=3$, $J=20$, sample size ($N=500$), and $\rho =0$.

Table 7 Summary of bias for GDM item parameters, $\varvec{\Theta }$, using the MVN prior by latent class and item for $K=3$, $J=20$, sample size ($N=1500$), and $\rho =0$.

Table 8 Summary of bias for GDM item parameters, $\varvec{\Theta }$, using the MVN prior by latent class and item for $K=3$, $J=20$, sample size ($N=500$), and $\rho =0.5$.

Table 9 Summary of bias for GDM item parameters, $\varvec{\Theta }$, using the MVN prior by latent class and item for $K=3$, $J=20$, sample size ($N=1500$), and $\rho =0.5$.

Table 10 Summary of mean absolute deviation for GDM item parameters, $\varvec{\Theta }$, using the MVN prior by latent class and item for $K=3$, $J=20$, sample size ($N=500$), and $\rho =0$.

Table 11 Summary of mean absolute deviation for GDM item parameters, $\varvec{\Theta }$, using the MVN prior by latent class and item for $K=3$, $J=20$, sample size ($N=1500$), and $\rho =0$.

Table 12 Summary of mean absolute deviation for GDM item parameters, $\varvec{\Theta }$, using the MVN prior by latent class and item for $K=3$, $J=20$, sample size ($N=500$), and $\rho =0.5$.

Table 13 Summary of mean absolute deviation for GDM item parameters, $\varvec{\Theta }$, using the MVN prior by latent class and item for $K=3$, $J=20$, sample size ($N=1500$), and $\rho =0.5$.

Table 14 Summary of bias for GDM structural parameters, $\varvec{\pi }$, using the MVN prior by latent class, sample size (N), and attribute tetrachoric correlation $\rho$ for $K=3$.

Table 15 Summary of mean absolute deviation for GDM structural parameters, $\varvec{\pi }$, using the MVN prior by latent class, sample size (N), and attribute tetrachoric correlation ($\rho$) for $K=3$.

Appendix B: DINA Monte Carlo Simulation Study

Overview

In this section, we review results from two simulation studies to assess expert-predictor selection accuracy when a DINA model is the data generating mechanism. The goal of the simulation studies is to assess how expert-predictor selection accuracy is affected by the specification of the number of attributes K, the size of attribute tetrachoric correlations, and degree of expert-predictor correlation. We focus our attention on model selection accuracy given that we found evidence in preliminary investigations that accurately selecting the expert-predictors for the more parsimonious DINA model typically coincided with accurately recovering $\varvec{Q}$ and the other model parameters.

First, the degree of mismatch between the true K and the estimated K was manipulated to understand the extent to which over-specifying K impacts expert-predictor selection accuracy. The true number of attributes was fixed to $K=3$ in both simulation studies and we varied the estimated K to be 3, 4, 5, or 6 (i.e., the amount by which K was over-specified was 0, 1, 2, or 3 attributes). Second, as for the GDM simulation study reported above, we manipulated attribute dependence by generating attributes using the multivariate normal probit model described by Chiu et al. (2009) with the population tetrachoric correlation taking values of $\rho = 0$ or 0.5. Third, the two simulation studies reported in this section differ in terms of the extent to which expert-predictor correlate. Study #1 sets $V=7$ and generates orthogonal attributes by sampling $x_{jv}\sim \text {Bernoulli}(0.5)$ and $\varvec{Q}$ is defined as the first three columns of $\varvec{X}$. In contrast, Study #2 sets $V=7$ and uses an expert-derived $\varvec{X}$ based upon Tatsuoka (1990) specified $\varvec{Q}$ for Tatsuoka’s FS dataset (see Table 3). Note attribute (VII) was dropped given minimal variability and, in order to match the results of the FS application, the true $\varvec{Q}$ in Study #2 was defined as attributes (I), (IV), and (V).

The simulation studies focus on conditions similar to the empirical application, so $N=500$ and $J=20$. Additionally, $\varvec{Y}$ is generated from a DINA model with slipping and guessing parameters equal to 0.2 for all j. A Gibbs sampler was implemented to approximate the posterior distribution of $\varvec{Q}$ and $\varvec{\Gamma }$, as well as the other DINA model parameters. A Bayesian formulation for the DINA model (e.g., see Chen et al., 2018; Culpepper, 2015) was implemented to estimate attributes, using an unstructured Dirichlet prior for $\varvec{\pi }$, and a uniform prior for slipping and guessing parameters. Note that 100 replications were executed for all conditions and a chain of length 40,000 was run with a burnin of 20,000. Model performance was measured by computing the matrix-wise accuracy of $\varvec{\Delta }$ separately for the active and inactive predictors.

Simulation Study #1 Results

The third and fourth columns of Table B1 report expert-predictor selection accuracy for the randomly generated $\varvec{X}$ and $\varvec{Q}$. The results suggest that the proportion of correctly identified inactive expert-predictors exceeds 0.94 for all estimated K and $\rho$. The active predictors are accurately selected for $\rho =0$, but the chance of selecting the active expert-predictors declined to 0.57 when $\rho =0.5$ and three unnecessary attributes are estimated.

Table 16 Summary of matrix-wise accuracy (i.e., $\widehat{\varvec{\Delta }}=\varvec{\Delta }$) for active and inactive expert-predictors across simulation study, estimated K, and $\rho$.

Simulation Study #2 Results

The accuracy of expert-predictor selection for an expert-derived $\varvec{X}$ and $\varvec{Q}$ is presented in the right two columns of Table B1. The results suggest that the proportion of correctly identified inactive expert-predictors exceeds 0.93 for all estimated K and $\rho$. The active predictors are accurately selected for $\rho =0$, but the chance of selecting the accurate predictors declines to 0.37 when $\rho =0.5$ and K is over-specified by three attributes.