Online Calibration Methods for the DINA Model with Independent Attributes in CD-CAT

Abstract

Item replenishing is essential for item bank maintenance in cognitive diagnostic computerized adaptive testing (CD-CAT). In regular CAT, online calibration is commonly used to calibrate the new items continuously. However, until now no reference has publicly become available about online calibration for CD-CAT. Thus, this study investigates the possibility to extend some current strategies used in CAT to CD-CAT. Three representative online calibration methods were investigated: Method A (Stocking in Scale drift in on-line calibration. Research Rep. 88-28, 1988), marginal maximum likelihood estimate with one EM cycle (OEM) (Wainer & Mislevy In H. Wainer (ed.) Computerized adaptive testing: A primer, pp. 65–102, 1990) and marginal maximum likelihood estimate with multiple EM cycles (MEM) (Ban, Hanson, Wang, Yi, & Harris in J. Educ. Meas. 38:191–212, 2001). The objective of the current paper is to generalize these methods to the CD-CAT context under certain theoretical justifications, and the new methods are denoted as CD-Method A, CD-OEM and CD-MEM, respectively. Simulation studies are conducted to compare the performance of the three methods in terms of item-parameter recovery, and the results show that all three methods are able to recover item parameters accurately and CD-Method A performs best when the items have smaller slipping and guessing parameters. This research is a starting point of introducing online calibration in CD-CAT, and further studies are proposed for investigations such as different sample sizes, cognitive diagnostic models, and attribute-hierarchical structures.

Appendices

Appendix A. CD-Method A Online Calibration Method for the DINA Model in CD-CAT

To obtain the conditional maximum likelihood estimates of g _j and s _j , we take the derivatives of lnL _j with respect to g _j and s _j

(A.1)

(A.2)

Equations (A.1) and (A.2) can be simplified as follows:

(A.3)

(A.4)

Thus, maximizations of lnL _j with respect to g _j and s _j are equivalent to solving for g _j and s _j in the following equations:

(A.5)

(A.6)

Equation (A.5) gives the estimator $\hat{g}_{j} = \frac{n_{2}}{n_{1} +n_{2}}$, where n ₁ is the number of examinees who take the new item j and satisfy the condition “$\hat{\eta} _{ij} = 0$ and u _ij =0”, n ₂ is the number of examinees who answer the new item j and meet the condition “$\hat{\eta} _{ij} = 0$ and u _ij =1”; Similarly, Equation (A.6) results in the estimator $\hat{s}_{j} =\frac{n_{3}}{n_{3} + n_{4}}$, where n ₃ is the number of examinees who take the new item j and satisfy the condition “$\hat{\eta} _{ij}= 1$ and u _ij =0”, n ₄ is the number of examinees who answer the new item j and meet the condition “$\hat{\eta} _{ij} = 1$ and u _ij =1”. Note that if n ₁+n ₂=0 or n ₃+n ₄=0, we let $\hat{g}_{j} = \frac{\mathit{Lower}\_\mathit{Bound} +\mathit{Upper}\_\mathit{Bound}}{2}$ or $\hat{s}_{j}= \frac{\mathit{Lower}\_\mathit{Bound} + \mathit{Upper}\_\mathit{Bound}}{2}$.

Therefore, for every new item, once the n ₁,n ₂,n ₃ and n ₄ values are calculated, the estimators $\hat{g}_{j}$ and $\hat{s}_{j}$ can be easily computed.

Appendix B. CD-OEM Online Calibration Method for the DINA Model in CD-CAT

To obtain the estimates of g _j and s _j , we take the derivative of l _j (u _j ) with respect to Δ _j (Δ _j =g _j ,s _j ):

(B.1)

Because

(B.2)

Equation (B.1) becomes

(B.3)

where is the expected number of examinees who take the new item j and have the KS α _l ; is the expected number of examinees who correctly answer the new item j and have the KS α _l .

In addition, all possible KSs can be divided into two categories by the new item j, the first category is “” (examinees whose KS α _l does not include mastery of all the required attributes of the new item j), and the second one is “” (examinees whose KS α _l includes mastery of all the required attributes of the new item j). Consequently, Equation (B.3) turns into

(B.4)

Since

(B.5)

Equation (B.4) changes into

(B.6)

where

is the number of examinees who take the new item j and lack mastery of at least one of the required attributes of the new item j, is the number of examinees who correctly answer the new item j and lack mastery of at least one of the required attributes of new item j; Similarly, , and they belong to the examinees who have mastered all the required attributes of the new item j. Equation $I_{j}^{(0)} + I_{j}^{(1)} = n_{j}$ (j=1,2,…,m) holds for every new item.

Equation (B.6) results in the estimators of g _j and s _j

$$ \hat{g}_{j} = \frac{R_{j}^{(0)}}{I_{j}^{(0)}}\quad \mbox{and}\quad \hat{s}_{j} =\frac{I_{j}^{(1)} - R_{j}^{(1)}}{I_{j}^{(1)}}.$$

(B.7)

Therefore, for the new item j, once the $I_{j}^{(0)}$, $R_{j}^{(0)}$, $I_{j}^{(1)}$ and $R_{j}^{(1)}$ values are calculated, the estimators $\hat{g}_{j}$ and $\hat{s}_{j}$ can be easily computed. However, it should be noted that calculating the above four values needs to use the item parameters of the new items. Consequently, this study sets the initial guessing and slipping parameters to all $\frac{\mathit{Lower}\_\mathit{Bound} +\mathit{Upper}\_\mathit{Bound}}{2}$.

Appendix C. Attributes for the Level-2 English Proficiency Test