Incorrect options for multiple-choice questions are often intentionally included so that they may be selected by an examinee who possesses a misconception. Determining whether an examinee possess a misconception is useful for educational purposes. In the present paper, two statistical models that can estimate examinees’ possession of misconceptions by analyzing multiple-choice data, which are unscored data were developed. By converting multiple-choice data to binary data, which are scored data (\(1=\) correct, \(0=\) incorrect), the Bug-DINO model can estimate examinees’ possession of misconceptions. However, converting multiple-choice data to binary data causes a loss in information, because which incorrect option an examinee chooses is important information for an examinee’s knowledge state. The three models (two developed models and the Bug-DINO model) are compared in a simulation study, and the developed models are applied to the Reading Skill Test data.
This is a preview of subscription content, log in to check access.
Buy single article
Instant access to the full article PDF.
Price includes VAT for USA
Arai NH, Todo N, Arai T, Bunji K, Sugawara S, Inuzuka M, Matsuzaki T, Ozaki K (2017) Reading skill test to diagnose basic language skills in comparison to machines. In: Proceedings of the 39th annual cognitive science society meeting (CogSci 2017), pp 1556–1561
Chen J (2017) A residual-based approach to validate Q-Matrix specifications. Appl Psychol Meas 41(4):277–293
de la Torre J, Douglas J (2004) Higher-order latent trait models for cognitive diagnosis. Psychometrika 69(3):333–353
de la Torre J (2009) A cognitive diagnosis model for cognitively based multiple-choice options. Appl Psychol Meas 33(3):163–183
de la Torre J (2011) The generalized DINA model framework. Psychometrika 76(2):179–199
de la Torre J, Chiu C-Y (2016) A general method of empirical Q-matrix validation. Psychometrika 81(2):253–273
DiBello LV, Henson RA, Stout WF (2015) A family of generalized diagnostic classification models for multiple choice option-based scoring. Appl Psychol Meas 39(1):62–79
Gelman A, Rubin DB (1992) Inference from iterative simulation using mutiple sequences. Stat Sci 7(4):457–472
Hartz S (2002) A Bayesian framework for the unified model for assessing cognitive abilities: blending theory with practicality (Doctoral dissertation). University of Illinois, Urbana-Champaign
Hastings WK (1970) Monte carlo sampling methods using markov chains and their applications. Biometrika 57(1):97–109
Im S, Corter JE (2011) Statistical consequences of attribute misspecification in the rule spece method. Educ Psychol Meas 71(4):712–731
Junker BW, Sijtsma K (2001) Cognitive assessment models with few assumptions, and connections with nonparametric item response theory. Appl Psychol Meas 25(3):258–272
Köhn HF, Chiu CY (2017) A procedure for assessing the completeness of the Q-Matrices of cognitively diagnostic tests. Psychometrika 82(1):112–132
Kuo B-C, Chen C-H, Yang C-W, Mok MMC (2016) Cognitive diagnostic models for tests with multiple-choice and constructed-response items. Educ Psychol 36(6):1115–1133
Kuo B-C, Chen C-H, de la Torre J (2018) A cognitive diagnosis model for identifying coexisting skills and misconceptions. Appl Psychol Meas 42(3):179–191
Maris E (1999) Estimating multiple classification latent class models. Psychometrika 64(2):187–212
Minchen ND, de la Torre J, Liu Y (2017) A cognitive diagnosis model for continuous response. J Educ Behav Stat 42(6):651–677
Ozaki K (2015) DINA models for multiple-choice items with few parameters: considering incorrect answers. Appl Psychol Meas 39(6):431–447
Richards JC, Schmidt R (2002) Dictionary of language teaching and applied linguistics, 3rd edn. Longman, London
Rupp AA, Templin JL (2008) The effects of Q-matrix misspecification on parameter estimates and classification accuracy in the DINA model. Educ Psychol Meas 68(1):78–96
Templin J, Henson R (2006) Measurement of psychological disorders using cognitive diagnosis models. Psychol Methods 11(3):287–305
This research was funded by Grant-in-Aid for Scientific Research(C) 18K03057.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Communicated by Russell George Almond.
Electronic supplementary material
Below is the link to the electronic supplementary material.
About this article
Cite this article
Ozaki, K., Sugawara, S. & Arai, N. Cognitive diagnosis models for estimation of misconceptions analyzing multiple-choice data. Behaviormetrika 47, 19–41 (2020). https://doi.org/10.1007/s41237-019-00100-9
- Multiple-choice item
- Cognitive diagnosis model
- DINO model