Advertisement

Psychometrika

, Volume 77, Issue 2, pp 201–222 | Cite as

Online Calibration Methods for the DINA Model with Independent Attributes in CD-CAT

  • Ping ChenEmail author
  • Tao Xin
  • Chun Wang
  • Hua-Hua Chang
Article

Abstract

Item replenishing is essential for item bank maintenance in cognitive diagnostic computerized adaptive testing (CD-CAT). In regular CAT, online calibration is commonly used to calibrate the new items continuously. However, until now no reference has publicly become available about online calibration for CD-CAT. Thus, this study investigates the possibility to extend some current strategies used in CAT to CD-CAT. Three representative online calibration methods were investigated: Method A (Stocking in Scale drift in on-line calibration. Research Rep. 88-28, 1988), marginal maximum likelihood estimate with one EM cycle (OEM) (Wainer & Mislevy In H. Wainer (ed.) Computerized adaptive testing: A primer, pp. 65–102, 1990) and marginal maximum likelihood estimate with multiple EM cycles (MEM) (Ban, Hanson, Wang, Yi, & Harris in J. Educ. Meas. 38:191–212, 2001). The objective of the current paper is to generalize these methods to the CD-CAT context under certain theoretical justifications, and the new methods are denoted as CD-Method A, CD-OEM and CD-MEM, respectively. Simulation studies are conducted to compare the performance of the three methods in terms of item-parameter recovery, and the results show that all three methods are able to recover item parameters accurately and CD-Method A performs best when the items have smaller slipping and guessing parameters. This research is a starting point of introducing online calibration in CD-CAT, and further studies are proposed for investigations such as different sample sizes, cognitive diagnostic models, and attribute-hierarchical structures.

Key words

cognitive diagnostic computerized adaptive testing online calibration DINA model independent attribute new item 

Notes

Acknowledgements

This study was conducted when the first author was a visiting scholar at the University of Illinois at Urbana-Champaign (UIUC). He would like to thank the China Scholarship Council (CSC) for the opportunity and one year financial support. Part of the paper was originally presented in 2010 annual meeting of the Psychometric Society, Athens, GA. The authors are indebted to the editor, associate editor and three anonymous reviewers for their constructive suggestions and comments on the earlier manuscript.

References

  1. Ban, J.-C., Hanson, B.H., Wang, T., Yi, Q., & Harris, D.J. (2001). A comparative study of on-line pretest item-calibration/scaling methods in computerized adaptive testing. Journal of Educational Measurement, 38, 191–212. Google Scholar
  2. Ban, J.-C., Hanson, B.H., Yi, Q., & Harris, D.J. (2002). Data sparseness and online pretest item calibration/scaling methods in CAT (ACT Research Report 02-01). Iowa City, IA, ACT, Inc. Available at http://www.eric.ed.gov/ERICDocs/data/ericdocs2sql/content_storage_01/0000019b/80/19/da/e9.pdf.
  3. Chang, Y.-C.I., & Lu, H. (2010). Online calibration via variable length computerized adaptive testing. Psychometrika, 75, 140–157. CrossRefGoogle Scholar
  4. Cheng, Y. (2009). When cognitive diagnosis meets computerized adaptive testing. Psychometrika, 74, 619–632. CrossRefGoogle Scholar
  5. Cheng, Y., & Chang, H. (2007). The modified maximum global discrimination index method for cognitive diagnostic computerized adaptive testing. Paper presented at the 2007 GMAC Conference on Computerized Adaptive Testing, McLean, USA, June. Google Scholar
  6. Dibello, L.V., Stout, W.F., & Roussos, L.A. (1995). Unified cognitive/psychometric diagnostic assessment likelihood-based classification techniques. In P. Nichols, S. Chipman, & R. Brennan (Eds.), Cognitively diagnostic assessments (pp. 361–389). Hillsdale: Erlbaum. Google Scholar
  7. de la Torre, J. (2009). DINA model and parameter estimation: a didactic. Journal of Educational and Behavioral Statistics, 34, 115–130. CrossRefGoogle Scholar
  8. de la Torre, J., & Douglas, J.A. (2004). Higher-order latent trait models for cognitive diagnosis. Psychometrika, 69, 333–353. CrossRefGoogle Scholar
  9. Doignon, J.P., & Falmagne, J.C. (1999). Knowledge spaces. New York: Springer. CrossRefGoogle Scholar
  10. Embretson, S. (1984). A general latent trait model for response processes. Psychometrika, 49, 175–186. CrossRefGoogle Scholar
  11. Embretson, S., & Reise, S. (2000). Item response theory for psychologists. Mahwah: Erlbaum. Google Scholar
  12. Fedorov, V.V. (1972). Theory of optimal design. New York: Academic Press. Google Scholar
  13. Haertel, E.H. (1989). Using restricted latent class models to map the skill structure of achievement items. Journal of Educational Measurement, 26, 333–352. CrossRefGoogle Scholar
  14. Hartz, S.M. (2002). A Bayesian framework for the unified model for assessing cognitive abilities: Blending theory with practicality (Unpublished doctoral dissertation). University of Illinois at Urbana-Champaign, Urbana-Champaign, IL. Google Scholar
  15. Junker, B.W., & Sijtsma, K. (2001). Cognitive assessment models with few assumptions, and connections with nonparametric item response theory. Applied Psychological Measurement, 25, 258–272. CrossRefGoogle Scholar
  16. Leighton, J.P., Gierl, M.J., & Hunka, S.M. (2004). The attribute hierarchy method for cognitive assessment: a variation on Tatsuoka’s rule-space approach. Journal of Educational Measurement, 41, 205–237. CrossRefGoogle Scholar
  17. Liu, H., You, X., Wang, W., Ding, S., & Chang, H. (2010). Large-scale applications of cognitive diagnostic computerized adaptive testing in China. Paper presented at the annual meeting of National Council on Measurement in Education, Denver, CO, April. Google Scholar
  18. Macready, G.B., & Dayton, C.M. (1977). The use of probabilistic models in the assessment of mastery. Journal of Educational Statistics, 33, 379–416. Google Scholar
  19. Makransky, G. (2009). An automatic online calibration design in adaptive testing. Paper presented at the 2007 GMAC Conference on Computerized Adaptive Testing, McLean, USA, June. Google Scholar
  20. Maris, E. (1999). Estimating multiple classification latent class models. Psychometrika, 64, 187–212. CrossRefGoogle Scholar
  21. McGlohen, M.K. (2004). The application of cognitive diagnosis and computerized adaptive testing to a large-scale assessment. Unpublished doctoral thesis, University of Texas at Austin. Google Scholar
  22. McGlohen, M.K., & Chang, H. (2008). Combining computer adaptive testing technology with cognitively diagnostic assessment. Behavior Research Methods, 40, 808–821. PubMedCrossRefGoogle Scholar
  23. Rupp, A., & Templin, J. (2008). The effects of Q-matrix misspecification on parameter estimates and classification accuracy in the DINA model. Educational and Psychological Measurement, 68, 78–96. CrossRefGoogle Scholar
  24. Silvey, S.D. (1980). Optimal design. London: Chapman and Hall. Google Scholar
  25. Stocking, M.L. (1988). Scale drift in on-line calibration (Research Rep. 88-28). Princeton, NJ: ETS. Google Scholar
  26. Tatsuoka, K.K. (1995). Architecture of knowledge structures and cognitive diagnosis: a statistical pattern classification approach. In P. Nichols, S. Chipman, & R. Brennan (Eds.), Cognitively diagnostic assessments (pp. 327–359). Hillsdale: Erlbaum. Google Scholar
  27. Tatsuoka, C. (2002). Data analytic methods for latent partially ordered classification models. Journal of the Royal Statistical Society. Series C, Applied Statistics, 51, 337–350. CrossRefGoogle Scholar
  28. Tatsuoka, K.K., & Tatsuoka, M.M. (1997). Computerized cognitive diagnostic adaptive testing: effect on remedial instruction as empirical validation. Journal of Educational Measurement, 34, 3–20. CrossRefGoogle Scholar
  29. Templin, J., & Henson, R. (2006). Measurement of psychological disorders using cognitive diagnosis models. Psychological Methods, 11, 287–305. PubMedCrossRefGoogle Scholar
  30. Wainer, H. (1990). Computerized adaptive testing: A primer. Hillsdale: Erlbaum. Google Scholar
  31. Wainer, H., & Mislevy, R.J. (1990). Item response theory, item calibration, and proficiency estimation. In H. Wainer (Ed.), Computerized adaptive testing: A primer (pp. 65–102). Hillsdale: Erlbaum. Google Scholar
  32. Weiss, D.J. (1982). Improving measurement quality and efficiency with adaptive testing. Applied Psychological Measurement, 6, 473–492. CrossRefGoogle Scholar
  33. Xu, X., Chang, H., & Douglas, J. (2003). A simulation study to compare CAT strategies for cognitive diagnosis. Paper presented at the annual meeting of National Council on Measurement in Education, Chicago, IL, April. Google Scholar

Copyright information

© The Psychometric Society 2012

Authors and Affiliations

  1. 1.National Key Laboratory of Cognitive Neuroscience and LearningBeijing Normal UniversityBeijingChina
  2. 2.University of Illinois at Urbana-ChampaignUrbana-ChampaignUSA

Personalised recommendations