Online Calibration Methods for the DINA Model with Independent Attributes in CD-CAT
- 949 Downloads
Item replenishing is essential for item bank maintenance in cognitive diagnostic computerized adaptive testing (CD-CAT). In regular CAT, online calibration is commonly used to calibrate the new items continuously. However, until now no reference has publicly become available about online calibration for CD-CAT. Thus, this study investigates the possibility to extend some current strategies used in CAT to CD-CAT. Three representative online calibration methods were investigated: Method A (Stocking in Scale drift in on-line calibration. Research Rep. 88-28, 1988), marginal maximum likelihood estimate with one EM cycle (OEM) (Wainer & Mislevy In H. Wainer (ed.) Computerized adaptive testing: A primer, pp. 65–102, 1990) and marginal maximum likelihood estimate with multiple EM cycles (MEM) (Ban, Hanson, Wang, Yi, & Harris in J. Educ. Meas. 38:191–212, 2001). The objective of the current paper is to generalize these methods to the CD-CAT context under certain theoretical justifications, and the new methods are denoted as CD-Method A, CD-OEM and CD-MEM, respectively. Simulation studies are conducted to compare the performance of the three methods in terms of item-parameter recovery, and the results show that all three methods are able to recover item parameters accurately and CD-Method A performs best when the items have smaller slipping and guessing parameters. This research is a starting point of introducing online calibration in CD-CAT, and further studies are proposed for investigations such as different sample sizes, cognitive diagnostic models, and attribute-hierarchical structures.
Key wordscognitive diagnostic computerized adaptive testing online calibration DINA model independent attribute new item
This study was conducted when the first author was a visiting scholar at the University of Illinois at Urbana-Champaign (UIUC). He would like to thank the China Scholarship Council (CSC) for the opportunity and one year financial support. Part of the paper was originally presented in 2010 annual meeting of the Psychometric Society, Athens, GA. The authors are indebted to the editor, associate editor and three anonymous reviewers for their constructive suggestions and comments on the earlier manuscript.
- Ban, J.-C., Hanson, B.H., Wang, T., Yi, Q., & Harris, D.J. (2001). A comparative study of on-line pretest item-calibration/scaling methods in computerized adaptive testing. Journal of Educational Measurement, 38, 191–212. Google Scholar
- Ban, J.-C., Hanson, B.H., Yi, Q., & Harris, D.J. (2002). Data sparseness and online pretest item calibration/scaling methods in CAT (ACT Research Report 02-01). Iowa City, IA, ACT, Inc. Available at http://www.eric.ed.gov/ERICDocs/data/ericdocs2sql/content_storage_01/0000019b/80/19/da/e9.pdf.
- Cheng, Y., & Chang, H. (2007). The modified maximum global discrimination index method for cognitive diagnostic computerized adaptive testing. Paper presented at the 2007 GMAC Conference on Computerized Adaptive Testing, McLean, USA, June. Google Scholar
- Dibello, L.V., Stout, W.F., & Roussos, L.A. (1995). Unified cognitive/psychometric diagnostic assessment likelihood-based classification techniques. In P. Nichols, S. Chipman, & R. Brennan (Eds.), Cognitively diagnostic assessments (pp. 361–389). Hillsdale: Erlbaum. Google Scholar
- Embretson, S., & Reise, S. (2000). Item response theory for psychologists. Mahwah: Erlbaum. Google Scholar
- Fedorov, V.V. (1972). Theory of optimal design. New York: Academic Press. Google Scholar
- Hartz, S.M. (2002). A Bayesian framework for the unified model for assessing cognitive abilities: Blending theory with practicality (Unpublished doctoral dissertation). University of Illinois at Urbana-Champaign, Urbana-Champaign, IL. Google Scholar
- Liu, H., You, X., Wang, W., Ding, S., & Chang, H. (2010). Large-scale applications of cognitive diagnostic computerized adaptive testing in China. Paper presented at the annual meeting of National Council on Measurement in Education, Denver, CO, April. Google Scholar
- Macready, G.B., & Dayton, C.M. (1977). The use of probabilistic models in the assessment of mastery. Journal of Educational Statistics, 33, 379–416. Google Scholar
- Makransky, G. (2009). An automatic online calibration design in adaptive testing. Paper presented at the 2007 GMAC Conference on Computerized Adaptive Testing, McLean, USA, June. Google Scholar
- McGlohen, M.K. (2004). The application of cognitive diagnosis and computerized adaptive testing to a large-scale assessment. Unpublished doctoral thesis, University of Texas at Austin. Google Scholar
- Silvey, S.D. (1980). Optimal design. London: Chapman and Hall. Google Scholar
- Stocking, M.L. (1988). Scale drift in on-line calibration (Research Rep. 88-28). Princeton, NJ: ETS. Google Scholar
- Tatsuoka, K.K. (1995). Architecture of knowledge structures and cognitive diagnosis: a statistical pattern classification approach. In P. Nichols, S. Chipman, & R. Brennan (Eds.), Cognitively diagnostic assessments (pp. 327–359). Hillsdale: Erlbaum. Google Scholar
- Wainer, H. (1990). Computerized adaptive testing: A primer. Hillsdale: Erlbaum. Google Scholar
- Wainer, H., & Mislevy, R.J. (1990). Item response theory, item calibration, and proficiency estimation. In H. Wainer (Ed.), Computerized adaptive testing: A primer (pp. 65–102). Hillsdale: Erlbaum. Google Scholar
- Xu, X., Chang, H., & Douglas, J. (2003). A simulation study to compare CAT strategies for cognitive diagnosis. Paper presented at the annual meeting of National Council on Measurement in Education, Chicago, IL, April. Google Scholar