Psychometrika

, Volume 80, Issue 2, pp 468–490 | Cite as

A Rate Function Approach to Computerized Adaptive Testing for Cognitive Diagnosis

Article
First Online:
Received:

Abstract

Computerized adaptive testing (CAT) is a sequential experiment design scheme that tailors the selection of experiments to each subject. Such a scheme measures subjects’ attributes (unknown parameters) more accurately than the regular prefixed design. In this paper, we consider CAT for diagnostic classification models, for which attribute estimation corresponds to a classification problem. After a review of existing methods, we propose an alternative criterion based on the asymptotic decay rate of the misclassification probabilities. The new criterion is then developed into new CAT algorithms, which are shown to achieve the asymptotically optimal misclassification rate. Simulation studies are conducted to compare the new approach with existing methods, demonstrating its effectiveness, even for moderate length tests.

Key words

computerized adaptive testing cognitive diagnosis large deviation classification 
This is a preview of subscription content, log in to check access.

Notes

Acknowledgements

We would like to thank the editors and the reviewers for providing valuable comments. This research is supported in part by NSF CMMI-1069064, NSF SES-1323977, and NIH 5R37GM047845.

References

  1. Chang, H.-H., & Ying, Z. (1996). A global information approach to computerized adaptive testing. Applied Psychological Measurement, 20, 213–229. CrossRefGoogle Scholar
  2. Cheng, Y. (2009). When cognitive diagnosis meets computerized adaptive testing CD-CAT. Psychometrika, 74, 619–632. CrossRefGoogle Scholar
  3. Chiu, C., Douglas, J., & Li, X. (2009). Cluster analysis for cognitive diagnosis: theory and applications. Psychometrika, 74, 633–665. CrossRefGoogle Scholar
  4. Cox, D., & Hinkley, D. (2000). Theoretical statistics. London: Chapman & Hall. Google Scholar
  5. de la Torre, J., & Douglas, J. (2004). Higher order latent trait models for cognitive diagnosis. Psychometrika, 69, 333–353. CrossRefGoogle Scholar
  6. DiBello, L.V., Stout, W.F., & Roussos, L.A. (1995). Unified cognitive psychometric diagnostic assessment likelihood-based classification techniques. In P.D. Nichols, S.F. Chipman, & R.L. Brennan (Eds.), Cognitively diagnostic assessment (pp. 361–390). Hillsdale: Erlbaum Associates. Google Scholar
  7. Edelsbrunner, H., & Grayson, D.R. (2000). Edgewise subdivision of a simplex. Discrete & Computational Geometry, 24, 707–719. CrossRefGoogle Scholar
  8. Hartz, S.M. (2002). A Bayesian framework for the unified model for assessing cognitive abilities: blending theory with practicality. Unpublished doctoral dissertation, University of Illinois, Urbana-Champaign. Google Scholar
  9. Junker, B. (2007). Using on-line tutoring records to predict end-of-year exam scores: experience with the ASSISTments project and MCAS 8th grade mathematics. In R.W. Lissitz (Ed.), Assessing and modeling cognitive development in school: intellectual growth and standard settings. Maple Grove: JAM Press. Google Scholar
  10. Junker, B., & Sijtsma, K. (2001). Cognitive assessment models with few assumptions, and connections with nonparametric item response theory. Applied Psychological Measurement, 25, 258–272. CrossRefGoogle Scholar
  11. Leighton, J.P., Gierl, M.J., & Hunka, S.M. (2004). The attribute hierarchy model for cognitive assessment: a variation on Tatsuoka’s rule-space approach. Journal of Educational Measurement, 41, 205–237. CrossRefGoogle Scholar
  12. Lord, F.M. (1971). Robbins–Monro procedures for tailored testing. Educational and Psychological Measurement, 31, 3–31. CrossRefGoogle Scholar
  13. Lord, F.M. (1980). Applications of item response theory to practical testing problems. Hillsdale: Erlbaum. Google Scholar
  14. Owen, R.J. (1975). Bayesian sequential procedure for quantal response in context of adaptive mental testing. Journal of the American Statistical Association, 70, 351–356. CrossRefGoogle Scholar
  15. Rupp, A.A., Templin, J., & Henson, R.A. (2010). Diagnostic measurement: theory, methods, and applications. New York: Guilford Press. Google Scholar
  16. Serfling, R.J. (1980). Approximation theorems of mathematical statistics. New York: Wiley-Interscience (W. Shewhart & S. Wilks (Eds.)). CrossRefGoogle Scholar
  17. Tatsuoka, K.K. (1983). Rule space: an approach for dealing with misconceptions based on item response theory. Journal of Educational Measurement, 20, 345–354. CrossRefGoogle Scholar
  18. Tatsuoka, K.K. (1985). A probabilistic model for diagnosing misconceptions in the pattern classification approach. Journal of Educational Statistics, 12, 55–73. Google Scholar
  19. Tatsuoka, K. (1991). Boolean algebra applied to determination of the universal set of misconception states (ONR-Technical Report No. RR-91-44). Princeton: Educational Testing Services. Google Scholar
  20. Tatsuoka, C. (1996). Sequential classification on partially ordered sets. Doctoral dissertation, Cornell University. Google Scholar
  21. Tatsuoka, C. (2002). Data-analytic methods for latent partially ordered classification models. Applied Statistics, 51, 337–350. Google Scholar
  22. Tatsuoka, K.K. (2009). Cognitive assessment: an introduction to the rule space method. New York: Routledge. Google Scholar
  23. Tatsuoka, C., & Ferguson, T. (2003). Sequential classification on partially ordered sets. Journal of the Royal Statistical Society, Series B, Statistical Methodology, 65, 143–157. CrossRefGoogle Scholar
  24. Templin, J., & Henson, R.A. (2006). Measurement of psychological disorders using cognitive diagnosis models. Psychological Methods, 11, 287–305. PubMedCrossRefGoogle Scholar
  25. Templin, J., He, X., Roussos, L.A., & Stout, W.F. (2003). The pseudo-item method: a simple technique for analysis of polytomous data with the fusion model (External Diagnostic Research Group Technical Report). Google Scholar
  26. Thissen, D., & Mislevy, R.J. (2000). Testing algorithms. In H. Wainer et al. (Eds.), Computerized adaptive testing: a primer (2nd ed., pp. 101–133). Mahwah: Lawrence Erlbaum Associates. Google Scholar
  27. van der Linden, W.J. (1998). Bayesian item selection criteria for adaptive testing. Psychometrika, 63, 201–216. CrossRefGoogle Scholar
  28. von Davier, M. (2005). A general diagnosis model applied to language testing data (Research report). Princeton: Educational Testing Service. Google Scholar
  29. Xu, X., Chang, H.-H., & Douglas, J. (2003). A simulation study to compare CAT strategies for cognitive diagnosis. Paper presented at the annual meeting of the American Educational Research Association, Chicago, April 2003. Google Scholar

Copyright information

© The Psychometric Society 2013

Authors and Affiliations

  1. 1.Columbia UniversityNew YorkUSA

Personalised recommendations