A General Method of Empirical Q-matrix Validation

Abstract

In contrast to unidimensional item response models that postulate a single underlying proficiency, cognitive diagnosis models (CDMs) posit multiple, discrete skills or attributes, thus allowing CDMs to provide a finer-grained assessment of examinees’ test performance. A common component of CDMs for specifying the attributes required for each item is the Q-matrix. Although construction of Q-matrix is typically performed by domain experts, it nonetheless, to a large extent, remains a subjective process, and misspecifications in the Q-matrix, if left unchecked, can have important practical implications. To address this concern, this paper proposes a discrimination index that can be used with a wide class of CDM subsumed by the generalized deterministic input, noisy “and” gate model to empirically validate the Q-matrix specifications by identifying and replacing misspecified entries in the Q-matrix. The rationale for using the index as the basis for a proposed validation method is provided in the form of mathematical proofs to several relevant lemmas and a theorem. The feasibility of the proposed method was examined using simulated data generated under various conditions. The proposed method is illustrated using fraction subtraction data.

Appendix : Proofs of the Lemmas and the Theorem

Lemma 1

$\bar{p}(\varvec{\alpha }_{k:K''})=\bar{p}(\varvec{\alpha }_{1:K''})$ for all $k<K''$.

Proof

According to Definition 1,

$$\begin{aligned} \bar{p}(\varvec{\alpha }_{k:K''})= & {} \sum _{\alpha _{k}=0}^{1}\cdots \sum _{\alpha _{K''}=0}^{1} w(\varvec{\alpha }_{k:K''})p(\varvec{\alpha }_{k:K''})\nonumber \\= & {} \sum _{\alpha _{k}=0}^{1}\cdots \sum _{\alpha _{K''}=0}^{1} w(\varvec{\alpha }_{k:K''})\frac{\sum _{\alpha _{1}=0}^{1}\cdots \sum _{\alpha _{k-1}=0}^{1} w(\varvec{\alpha }_{1:K''})p(\varvec{\alpha }_{1:K''})}{w(\varvec{\alpha }_{k:K''})}\nonumber \\= & {} \sum _{\alpha _{k}=0}^{1}\cdots \sum _{\alpha _{K''}=0}^{1} \sum _{\alpha _{1}=0}^{1}\cdots \sum _{\alpha _{k-1}=0}^{1}w(\varvec{\alpha }_{1:K''})p(\varvec{\alpha }_{1:K''})\nonumber \\= & {} \sum _{\alpha _{1}=0}^{1}\cdots \sum _{\alpha _{K''}=0}^{1} w(\varvec{\alpha }_{1:K''})p(\varvec{\alpha }_{1:K''})\nonumber \\= & {} \bar{p}(\varvec{\alpha }_{1:K''}) \end{aligned}$$

(12)

Because (12) holds for all $k<K''$, $\bar{p}(\varvec{\alpha }_{K':K''})=\bar{p}(\varvec{\alpha }_{1:K'})=\bar{p}(\varvec{\alpha }_{1:K^{*}})=\bar{p}(\varvec{\alpha }_{1:K''})$. $\square$

Lemma 2

Suppose $K'+1\le K^{*}$.

$$\begin{aligned} \sum _{\alpha _{K'+1}=0}^1\cdots \sum _{\alpha _{K''}=0}^1 w(\varvec{\alpha }_{(K'+1):K''})p^{2}(\varvec{\alpha }_{(K'+1):K''}) \le \sum _{\alpha _{K'}=0}^1\cdots \sum _{\alpha _{K''}=0}^1 w(\varvec{\alpha }_{K':K''})p^{2}(\varvec{\alpha }_{K':K''}). \end{aligned}$$

(13)

Proof

The LHS of (11) equals

$$\begin{aligned}&\sum _{\alpha _{K'+1}=0}^{1}\cdots \sum _{\alpha _{K''}=0}^{1}\sum _{\alpha _{K'}=0}^{1}w(\varvec{\alpha }_{K':K''})\Bigg [ \frac{\sum _{\alpha _{K'}=0}^{1}w(\varvec{\alpha }_{K':K''})p(\varvec{\alpha }_{K':K''})}{\sum _{\alpha _{K'}=0}^{1}w(\varvec{\alpha }_{K':K''})}\Bigg ]^{2}\nonumber \\&\quad =\sum _{\alpha _{K'+1}=0}^{1}\cdots \sum _{\alpha _{K''}=0}^{1} \frac{\Bigg [\sum _{\alpha _{K'}=0}^{1}w(\varvec{\alpha }_{K':K''})p(\varvec{\alpha }_{K':K''})\Bigg ]^{2}}{\sum _{\alpha _{K'}=0}^{1}w(\varvec{\alpha }_{K':K''})}\nonumber \\&\quad =\sum _{\alpha _{K'+1}=0}^{1}\cdots \sum _{\alpha _{K''}=0}^{1} \left[ w(0,\varvec{\alpha }_{(K'+1):K''})+w(1,\varvec{\alpha }_{(K'+1):K''})\right] ^{-1}\nonumber \\&\qquad \qquad \quad \Big [w^{2}(0,\varvec{\alpha }_{(K'+1):K''})p^{2}(0,\varvec{\alpha }_{(K'+1):K''})+ w^{2}(1,\varvec{\alpha }_{(K'+1):K''})p^{2}(1,\varvec{\alpha }_{(K'+1):K''})\nonumber \\&\qquad \qquad \qquad 2w(0,\varvec{\alpha }_{(K'+1):K''})w(1,\varvec{\alpha }_{(K'+1):K''}) p(0,\varvec{\alpha }_{(K'+1):K''})p(1,\varvec{\alpha }_{(K'+1):K''}) \Big ]. \end{aligned}$$

(14)

The RHS of (13) is

$$\begin{aligned} \sum _{\alpha _{K'+1}=0}^{1}\cdots \sum _{\alpha _{K''}=0}^{1}\Big [ w(0,\varvec{\alpha }_{(K'+1):K''})p^{2}(0,\varvec{\alpha }_{(K'+1):K''})+ w(1,\varvec{\alpha }_{(K'+1):K''})p^{2}(1,\varvec{\alpha }_{(K'+1):K''}) \Big ]. \end{aligned}$$

(15)

Subtracting (14) from (15), and simplifying,

$$\begin{aligned}&\sum _{\alpha _{K'+1}=0}^{1}\cdots \sum _{\alpha _{K''}=0}^{1}\Bigg [ \frac{w(0,\varvec{\alpha }_{(K'+1):K''})w(1,\varvec{\alpha }_{(K'+1):K''})}{w(0,\varvec{\alpha }_{(K'+1):K''})+w(1,\varvec{\alpha }_{(K'+1):K''})}\Bigg ] \Big [p(0,\varvec{\alpha }_{(K'+1):K''})-p(1,\varvec{\alpha }_{(K'+1):K''})\Big ]^{2}\\&\quad \ge 0. \end{aligned}$$

Therefore, (11) holds. $\square$

Theorem 1

$\varsigma ^2_{K':K''}\le \varsigma ^2_{1:K^{*}}$.

Proof

Case 1: The provisional q-vector is strictly overspecified, that is, $K'=K^{*}<K''$ resulting in $\varvec{q}=\varvec{q}_{1:K''}$ and $\varsigma ^2=\varsigma ^2_{1:K''}$.

By Lemma 1, the theorem for this case can be proved by showing that

$$\begin{aligned} \sum _{\alpha _{1}=0}^{1}\cdots \sum _{\alpha _{K''}=0}^{1}w(\varvec{\alpha }_{1:K''})p^{2}(\varvec{\alpha }_{1:K''}) =\sum _{\alpha _{1}=0}^{1}\cdots \sum _{\alpha _{K^{*}}=0}^{1}w(\varvec{\alpha }_{1:{K^{*}}})p^{2}(\varvec{\alpha }_{1:{K^{*}}}). \end{aligned}$$

(16)

Based on Definition 1,

$$\begin{aligned} w(\varvec{\alpha }_{1:K^{*}})=\sum _{\alpha _{K^{*}+1}=0}^{1}\cdots \sum _{\alpha _{K''}=0}^{1} w(\varvec{\alpha }_{1:K''}), \end{aligned}$$

and

$$\begin{aligned} p(\varvec{\alpha }_{1:K^{*}})=\frac{\sum _{\alpha _{K^{*}+1}=0}^{1}\cdots \sum _{\alpha _{K''}=0}^{1} w(\varvec{\alpha }_{1:K''})p(\varvec{\alpha }_{1:K''}) }{\sum _{\alpha _{K^{*}+1}=0}^{1},\cdots ,\sum _{\alpha _{1:K''}=0}^{1}w(\varvec{\alpha }_{1:K''})}. \end{aligned}$$

The RHS of (16) can be expressed as

$$\begin{aligned} \sum _{\alpha _{1}=0}^{1}\cdots \sum _{\alpha _{K^{*}}=0}^{1} w(\varvec{\alpha }_{1:K^{*}})\Bigg [\frac{\sum _{\alpha _{K^{*}+1}=0}^{1}\cdots \sum _{\alpha _{K''}=0}^{1}w(\varvec{\alpha }_{1:K''}) p(\varvec{\alpha }_{1:K''})}{\sum _{\alpha _{K^{*}+1}=0}^{1}\cdots \sum _{\alpha _{K''}=0}^{1}w(\varvec{\alpha }_{1:K''})}\Bigg ]^{2}. \end{aligned}$$

(17)

By definition of a correct q-vector,

$$\begin{aligned} p(\varvec{\alpha }_{1:K''})=p(\varvec{\alpha }_{1:K^{*}}) \forall \alpha _{(K^{*}+1)},\ldots ,\alpha _{K''}. \end{aligned}$$

Thus, (17) is equal to

$$\begin{aligned}&\sum _{\alpha _{1}=0}^{1}\cdots \sum _{\alpha _{K^{*}}=0}^{1} w(\varvec{\alpha }_{1:K^{*}})\Bigg [\frac{p(\varvec{\alpha }_{1:K^{*}})\sum _{\alpha _{K^{*}+1}=0}^{1}\cdots \sum _{\alpha _{K''}=0}^{1}w(\varvec{\alpha }_{1:K''})}{\sum _{\alpha _{K^{*}+1}=0}^{1}\cdots \sum _{\alpha _{K''}=0}^{1}w(\varvec{\alpha }_{1:K''})}\Bigg ]^{2}\\&\quad =\sum _{\alpha _{1}=0}^{1}\cdots \sum _{\alpha _{K^{*}}=0}^{1} \sum _{\alpha _{K^{*}+1}=0}^{1}\cdots \sum _{\alpha _{K''}=0}^{1}w(\varvec{\alpha }_{1:K''}) p^{2}(\varvec{\alpha }_{1:K''})\\&\quad =\sum _{\alpha _{1}=0}^{1}\cdots \sum _{\alpha _{K''}=0}^{1}w(\varvec{\alpha }_{1:K''}) p^{2}(\varvec{\alpha }_{1:K''}) \end{aligned}$$

which is the LHS of (16).

Case 2: When the provisional q-vector is both under- and overspecified, that is, $K'<K^{*}<K''$, $\varvec{q}=\varvec{q}_{K':K''}$ and $\varsigma ^2=\varsigma ^2_{K':K''}$.

Case 2 will be proved by induction. By Lemma 1 and the result for Case 1, Case 2 of the theorem can be proved by showing that

$$\begin{aligned}&\sum _{\alpha _{K'}=0}^{1}\cdots \sum _{\alpha _{K''}=0}^{1}w(\varvec{\alpha }_{K':K''})p^{2}(\varvec{\alpha }_{K':K''}) \nonumber \\&\quad \le \sum _{\alpha _{1}=0}^{1}\cdots \sum _{\alpha _{K^{*}}=0}^{1}w(\varvec{\alpha }_{1:K^{*}})p^{2}(\varvec{\alpha }_{1:K^{*}})\nonumber \\&\quad =\sum _{\alpha _{1}=0}^{1}\cdots \sum _{\alpha _{K''}=0}^{1}w(\varvec{\alpha }_{1:K''})p^{2}(\varvec{\alpha }_{1:K''}) \end{aligned}$$

(18)

Step 1: Show that (18) is true for $K'=1$.

The LHS of (18) is

$$\begin{aligned} \sum _{\alpha _{K'}=0}^{1}\cdots \sum _{\alpha _{K''}=0}^{1}w(\varvec{\alpha }_{K':K''})p(\varvec{\alpha }_{K':K''}) =\sum _{\alpha _1=0}^{1}\cdots \sum _{\alpha _{K''}=0}^{1}w(\varvec{\alpha }_{1:K''})p(\varvec{\alpha }_{1:K''}), \end{aligned}$$

which is the RHS of (18).

Step 2: Assume that (18) is true for $K'=k$, that is,

$$\begin{aligned} \sum _{\alpha _{k}=0}^{1}\cdots \sum _{\alpha _{K''}=0}^{1}w(\varvec{\alpha }_{k:K''}) p^{2}(\varvec{\alpha }_{k:K''}) \le \sum _{\alpha _{1}=0}^{1}\cdots \sum _{\alpha _{K''}=0}^{1}w(\varvec{\alpha }_{1:K''})p^{2}(\varvec{\alpha }_{1:K''}). \end{aligned}$$

Step 3: Show that (18) is true for $K'=k+1$, as in,

$$\begin{aligned}&\sum _{\alpha _{k+1}=0}^{1}\cdots \sum _{\alpha _{K''}=0}^{1}w(\varvec{\alpha }_{k+1:K''}) p^{2}(\varvec{\alpha }_{k+1:K''})\nonumber \\&\quad \le \sum _{\alpha _{1}=0}^{1}\cdots \sum _{\alpha _{K^{*}}=0}^{1}w(\varvec{\alpha }_{1:K^{*}})p^{2}(\varvec{\alpha }_{1:K^{*}})\nonumber \\&\quad =\sum _{\alpha _{1}=0}^{1}\cdots \sum _{\alpha _{K''}=0}^{1}w(\varvec{\alpha }_{1:K''}) p^{2}(\varvec{\alpha }_{1:K''}) \end{aligned}$$

(19)

According to Lemma 2, the LHS of (19) is

$$\begin{aligned} \sum _{\alpha _{k+1}=0}^{1}\cdots \sum _{\alpha _{K''}=0}^{1} w(\varvec{\alpha }_{k+1:K''})p^{2}(\varvec{\alpha }_{k+1:K''})\le \sum _{\alpha _{k}=0}^{1}\cdots \sum _{\alpha _{K''}=0}^{1}w(\varvec{\alpha }_{k:K''}) p^{2}(\varvec{\alpha }_{k:K''}). \end{aligned}$$

(20)

According to the assumption in Step 2, (20) can further be expressed as

$$\begin{aligned} \sum _{\alpha _{k}=0}^{1}\cdots \sum _{\alpha _{K''}=0}^{1}w(\varvec{\alpha }_{k:K''}) p^{2}(\varvec{\alpha }_{k:K''})\le \sum _{\alpha _{1}=0}^{1}\cdots \sum _{\alpha _{K''}=0}^{1}w(\varvec{\alpha }_{1:K''})p^{2}(\varvec{\alpha }_{1:K''}), \end{aligned}$$

which is the RHS of (19). $\square$