Skip to main content
SpringerLink
Log in

Diagnostic Classification Models for Testlets: Methods and Theory

  • Theory & Methods
  • Published:
Psychometrika Aims and scope Submit manuscript

Abstract

Diagnostic classification models (DCMs) have seen wide applications in educational and psychological measurement, especially in formative assessment. DCMs in the presence of testlets have been studied in recent literature. A key ingredient in the statistical modeling and analysis of testlet-based DCMs is the superposition of two latent structures, the attribute profile and the testlet effect. This paper extends the standard testlet DINA (T-DINA) model to accommodate the potential correlation between the two latent structures. Model identifiability is studied and a set of sufficient conditions are proposed. As a byproduct, the identifiability of the standard T-DINA is also established. The proposed model is applied to a dataset from the 2015 Programme for International Student Assessment. Comparisons are made with DINA and T-DINA, showing that there is substantial improvement in terms of the goodness of fit. Simulations are conducted to assess the performance of the new method under various settings.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

References

  • Allman, E. S., Matias, C., & Rhodes, J. A. (2009). Identifiability of parameters in latent structure models with many observed variables. The Annals of Statistics, 37(6A), 3099–3132.

    Article  Google Scholar 

  • Bradlow, E. T., Wainer, H., & Wang, X. (1999). A Bayesian random effects model for testlets. Psychometrika, 64(2), 153–168.

    Article  Google Scholar 

  • Cai, L. (2010). A two-tier full-information item factor analysis model with applications. Psychometrika, 75(4), 581–612.

    Article  Google Scholar 

  • Chen, Y., Liu, J., Xu, G., & Ying, Z. (2015). Statistical analysis of q-matrix based diagnostic classification models. Journal of the American Statistical Association, 110(510), 850–866.

    Article  PubMed  Google Scholar 

  • Chen, Y., Liu, Y., & Xu, S. (2018). Mutual information reliability for latent class analysis. Applied Psychological Measurement, 42(6), 460–477.

    Article  PubMed  PubMed Central  Google Scholar 

  • Chiu, C.-Y., Douglas, J. A., & Li, X. (2009). Cluster analysis for cognitive diagnosis: Theory and applications. Psychometrika, 74, 633–665.

    Article  Google Scholar 

  • Coelho, P. S., & Pereira, L. N. (2011). A spatial unit level model for small area estimation. REVSTAT-Statistical Journal, 9(2), 155–180.

    Google Scholar 

  • Culpepper, S. A. (2015). Bayesian estimation of the DINA model with Gibbs sampling. Journal of Educational and Behavioral Statistics, 40(5), 454–476.

    Article  Google Scholar 

  • Culpepper, S. A. (2019). An exploratory diagnostic model for ordinal responses with binary attributes: Identifiability and estimation. Psychometrika, 84(4), 921–940.

    Article  PubMed  Google Scholar 

  • de la Torre, J. (2011). The generalized DINA model framework. Psychometrika, 76(2), 179–199.

    Article  Google Scholar 

  • de la Torre, J., & Douglas, J. A. (2004). Higher-order latent trait models for cognitive diagnosis. Psychometrika, 69(3), 333–353.

    Article  Google Scholar 

  • de La Torre, J., & Karelitz, T. M. (2009). Impact of diagnosticity on the adequacy of models for cognitive diagnosis under a linear attribute structure: A simulation study. Journal of Educational Measurement, 46(4), 450–469.

    Article  Google Scholar 

  • DeMars, C. E. (2006). Application of the bi-factor multidimensional item response theory model to testlet-based tests. Journal of Educational Measurement, 43(2), 145–168.

    Article  Google Scholar 

  • Fang, G., Liu, J., & Ying, Z. (2019). On the identifiability of diagnostic classification models. Psychometrika, 84(1), 19–40.

    Article  PubMed  Google Scholar 

  • Fang, G., Guo, J., Xu, X., Ying, Z., & Zhang, S. (2021). Identifiability of bifactor models. Statistica Sinica, 31, 2309–2330.

    Google Scholar 

  • Gibbons, R. D., & Hedeker, D. R. (1992). Full-information item bi-factor analysis. Psychometrika, 57(3), 423–436.

    Article  Google Scholar 

  • Gu, Y. (2020). Statistical analysis of structured latent attribute models (Unpublished doctoral dissertation).

  • Gu, Y., & Xu, G. (2019b). The sufficient and necessary condition for the identifiability and estimability of the DINA model. Psychometrika, 84(2), 468–483.

    Article  PubMed  Google Scholar 

  • Gu, Y., & Xu, G. (2020). Partial identifiability of restricted latent class models. The Annals of Statistics, 48(4), 2082–2107.

    Article  Google Scholar 

  • Gu, Y., & Xu, G. (2021). Sufficient and necessary conditions for the identifiability of the q-matrix. Statistica Sinica.

  • Gu, Y., & Xu, G. (2022). Generic identifiability of the DINA model and blessing of latent dependence. Psychometrika.

  • Hansen, M. (2013). Hierarchical item response models for cognitive diagnosis. University of California.

    Book  Google Scholar 

  • Hansen, M., Cai, L., Monroe, S., & Li, Z. (2016). Limited-information goodness-of-fit testing of diagnostic classification item response models. British Journal of Mathematical and Statistical Psychology, 69(3), 225–252.

    Article  PubMed  Google Scholar 

  • Henson, R., Templin, J., & Willse, J. (2009). Defining a family of cognitive diagnosis models using log-linear models with latent variables. Psychometrika, 74(2), 191–210.

    Article  Google Scholar 

  • Jennrich, R. I., & Bentler, P. M. (2012). Exploratory bi-factor analysis: The oblique case. Psychometrika, 77(3), 442–454.

    Article  PubMed  Google Scholar 

  • Junker, B. W., & Sijtsma, K. (2001). Cognitive assessment models with few assumptions, and connections with nonparametric item response theory. Applied Psychological Measurement, 25(3), 258–272.

    Article  Google Scholar 

  • Kang, H.-A., Liu, J., & Ying, Z. (2017). A graphical diagnostic classification model. Preprint at arXiv:1707.06318

  • Kish, L., Namboodiri, N. K., & Pillai, R. K. (1962). The ratio bias in surveys. Journal of the American Statistical Association, 57(300), 863–876.

    Article  Google Scholar 

  • Köhn, H.-F., & Chiu, C.-Y. (2016). A proof of the duality of the DINA model and the DINO model. Journal of Classification, 33, 171–184.

    Article  Google Scholar 

  • Ma, W. (2019). A diagnostic tree model for polytomous responses with multiple strategies. British Journal of Mathematical and Statistical Psychology, 72(1), 61–82.

    Article  PubMed  Google Scholar 

  • Ma, W., & de la Torre, J. (2016). A sequential cognitive diagnosis model for polytomous responses. British Journal of Mathematical and Statistical Psychology, 69(3), 253–275.

    Article  PubMed  Google Scholar 

  • Ma, W., Wang, C., & Xiao, J. (2023). A testlet diagnostic classification model with attribute hierarchies. Applied Psychological Measurement, 01466216231165315.

  • Macready, G. B., & Dayton, C. M. (1977). The use of probabilistic models in the assessment of mastery. Journal of Educational Statistics, 2(2), 99–120.

    Article  Google Scholar 

  • Meng, X.-L. (1993). On the absolute bias ratio of ratio estimators. Statistics & Probability Letters, 18(5), 345–348.

    Article  Google Scholar 

  • OECD. (2016). Pisa 2015 assessment and analytical framework: Science, reading, mathematic and financial literacy. Author Paris.

  • Rupp, A. A., Templin, J., & Henson, R. A. (2010). Diagnostic measurement: Theory, methods, and applications. Guilford Press.

    Google Scholar 

  • Sha, S. (2016). Nonparametric diagnostic classification analysis for testlet-based tests (Unpublished doctoral dissertation). The University of North Carolina at Greensboro.

  • Sireci, S. G., Thissen, D., & Wainer, H. (1991). On the reliability of testlet-based tests. Journal of Educational Measurement, 28(3), 237–247.

    Article  Google Scholar 

  • Tatsuoka, K. K. (1983). Rule space: An approach for dealing with misconceptions based on item response theory. Journal of educational measurement, 20(4), 345–354.

    Article  Google Scholar 

  • Templin, J. L., & Henson, R. A. (2006). Measurement of psychological disorders using cognitive diagnosis models. Psychological Methods, 11(3), 287.

    Article  PubMed  Google Scholar 

  • Wainer, H., Bradlow, E. T., & Wang, X. (2007). Testlet response theory and its applications. Cambridge University Press.

    Book  Google Scholar 

  • Xu, G. (2017). Identifiability of restricted latent class models with binary responses. The Annals of Statistics, 45(2), 675–707.

    Article  Google Scholar 

  • Xu, G., & Zhang, S. (2016). Identifiability of diagnostic classification models. Psychometrika, 81(3), 625–649.

    Article  PubMed  Google Scholar 

  • Yavuz, E., & Atar, H. Y. (2020). An examination of Turkish students’ PISA 2015 collaborative problem-solving competencies. International Journal of Assessment Tools in Education, 7(4), 588–606.

    Article  Google Scholar 

  • Zhan, P., Li, X., Wang, W.-C., Bian, Y., & Wang, L. (2015). The multidimensional testlet-effect cognitive diagnostic models. Acta Psychologica Sinica.

  • Zhan, P., Liao, M., & Bian, Y. (2018). Joint testlet cognitive diagnosis modeling for paired local item dependence in response times and response accuracy. Frontiers in Psychology, 9, 607.

    Article  PubMed  PubMed Central  Google Scholar 

Download references

Funding

This project is supported in part by the National Science Foundation (DMS-2015417), China Postdoctoral Science Foundation (2021M700466), and the China National Natural Science Foundation (12301376, 12371263).

Author information

Authors and Affiliations

  1. Beijing Normal University, Beijing, China

    Xin Xu

  2. Fudan University, Shanghai, China

    Guanhua Fang

  3. Minzu University of China, Beijing, China

    Jinxin Guo

  4. Columbia University, New York, USA

    Zhiliang Ying

  5. University of Illinois Urbana-Champaign, Champaign, USA

    Susu Zhang

Authors
  1. Xin Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Guanhua Fang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jinxin Guo
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Zhiliang Ying
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Susu Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Susu Zhang.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

This project is supported in part by the National Science Foundation (DMS-2015417), China Postdoctoral Science Foundation (2021M700466), and the China National Natural Science Foundation (12301376, 12371263). The authors have no competing interests to declare that are relevant to the content of this article. The datasets analyzed during the current study are publicly available in the PISA 2015 Database.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Xu, X., Fang, G., Guo, J. et al. Diagnostic Classification Models for Testlets: Methods and Theory. Psychometrika (2024). https://doi.org/10.1007/s11336-024-09962-9

Download citation

  • Received:

  • Published:

  • DOI: https://doi.org/10.1007/s11336-024-09962-9

Keywords

Search

Navigation