Bayesian Estimation of the DINA Q matrix

Psychometrika volume 83, pages89–108(2018)Cite this article

Abstract

Cognitive diagnosis models are partially ordered latent class models and are used to classify students into skill mastery profiles. The deterministic inputs, noisy “and” gate model (DINA) is a popular psychometric model for cognitive diagnosis. Application of the DINA model requires content expert knowledge of a Q matrix, which maps the attributes or skills needed to master a collection of items. Misspecification of Q has been shown to yield biased diagnostic classifications. We propose a Bayesian framework for estimating the DINA Q matrix. The developed algorithm builds upon prior research (Chen, Liu, Xu, & Ying, in J Am Stat Assoc 110(510):850–866, 2015) and ensures the estimated Q matrix is identified. Monte Carlo evidence is presented to support the accuracy of parameter recovery. The developed methodology is applied to Tatsuoka’s fraction-subtraction dataset.

This is a preview of subscription content, log in to check access.

Fig. 1
Fig. 2

References

  1. Chen, Y., Liu, J., Xu, G., & Ying, Z. (2015). Statistical analysis of Q-matrix based diagnostic classification models. Journal of the American Statistical Association, 110(510), 850–866.

    Article  PubMed  Google Scholar 

  2. Chiu, C.-Y. (2013). Statistical refinement of the Q-matrix in cognitive diagnosis. Applied Psychological Measurement, 37(8), 598–618.

    Article  Google Scholar 

  3. Chiu, C.-Y., Douglas, J. A., & Li, X. (2009). Cluster analysis for cognitive diagnosis: Theory and applications. Psychometrika, 74(4), 633–665.

    Article  Google Scholar 

  4. Chiu, C.-Y., & Köhn, H.-F. (2016). The reduced RUM as a logit model: Parameterization and constraints. Psychometrika, 81(2), 350–370.

    Article  PubMed  Google Scholar 

  5. Chiu, C.-Y., Köhn, H.-F., & Wu, H.-M. (2016). Fitting the reduced RUM with Mplus: A tutorial. International Journal of Testing, 16, 1–21.

    Article  Google Scholar 

  6. Chung, M. (2014). Estimating the Q-matrix for Cognitive Diagnosis Models in a Bayesian Framework (Unpublished doctoral dissertation). Columbia University.

  7. Cowles, M. K., & Carlin, B. P. (1996). Markov chain Monte Carlo convergence diagnostics: A comparative review. Journal of the American Statistical Association, 91, 883–904.

  8. Culpepper, S. A. (2015). Bayesian estimation of the DINA model with Gibbs sampling. Journal of Educational and Behavioral Statistics, 40(5), 454–476.

    Article  Google Scholar 

  9. DeCarlo, L. T. (2010). On the analysis of fraction subtraction data: The DINA model, classification, latent class sizes, and the Q-matrix. Applied Psychological Measurement, 35, 8–26.

    Article  Google Scholar 

  10. DeCarlo, L. T. (2012). Recognizing uncertainty in the Q-matrix via a Bayesian extension of the DINA model. Applied Psychological Measurement, 36, 447–468.

    Article  Google Scholar 

  11. de la Torre, J. (2008). An empirically based method of Q-matrix validation for the DINA model: Development and applications. Journal of Educational Measurement, 45(4), 343–362.

    Article  Google Scholar 

  12. de la Torre, J. (2009). Estimation code for the G-DINA model. In Presentation at the meeting of the American Educational Research Association. San Diego, CA.

  13. de la Torre, J., & Chiu, C.-Y. (2016). A general method of empirical Q-matrix validation. Psychometrika, 81(2), 253–273.

    Article  PubMed  Google Scholar 

  14. de la Torre, J., & Douglas, J. A. (2004). Higher-order latent trait models for cognitive diagnosis. Psychometrika, 69(3), 333–353.

    Article  Google Scholar 

  15. de la Torre, J., & Douglas, J. A. (2008). Model evaluation and multiple strategies in cognitive diagnosis: An analysis of fraction subtraction data. Psychometrika, 73(4), 595–624.

    Article  Google Scholar 

  16. Geweke, J. (1992). Evaluating the accuracy of sampling-based approaches to the calculation of posterior moments. Bayesian Statistics, 4, 169–188.

  17. Henson, R., & Templin, J. (2007). Importance of Q-matrix construction and its effects cognitive diagnosis model results. In Annual meeting of the national council on measurement in education. Chicago, IL.

  18. Henson, R., Templin, J., & Willse, J. (2009). Defining a family of cognitive diagnosis models using log-linear models with latent variables. Psychometrika, 74, 191–210.

    Article  Google Scholar 

  19. Huff, K., & Goodman, D. P. (2007). The demand for cognitive diagnostic assessment. In J. P. Leighton & M. J. Gierl (Eds.), Cognitive diagnostic assessment for education: Theory and applications (pp. 19–60). Cambridge: Cambridge University Press.

    Google Scholar 

  20. Junker, B. W., & Sijtsma, K. (2001). Cognitive assessment models with few assumptions, and connections with nonparametric item response theory. Applied Psychological Measurement, 25(3), 258–272.

    Article  Google Scholar 

  21. Leighton, J. P., & Gierl, M. J. (2007). Why cognitive diagnostic assessment. In J. P. Leighton & M. J. Gierl (Eds.), Cognitive diagnostic assessment for education: Theory and applications (pp. 3–18). Cambridge: Cambridge University Press.

    Google Scholar 

  22. Liu, J. (2017). On the consistency of Q-matrix estimation: A commentary. Psychometrika, 82(2), 523–527.

    Article  PubMed  Google Scholar 

  23. Liu, J., Xu, G., & Ying, Z. (2012). Data-driven learning of Q-matrix. Applied Psychological Measurement, 36(7), 548–564.

    Article  PubMed  PubMed Central  Google Scholar 

  24. Liu, J., Xu, G., & Ying, Z. (2013). Theory of the self-learning Q-matrix. Bernoulli, 19(5A), 1790–1817.

    Article  PubMed  PubMed Central  Google Scholar 

  25. Mislevy, R. J., & Wilson, M. (1996). Marginal maximum likelihood estimation for a psychometric model of discontinuous development. Psychometrika, 61(1), 41–71.

    Article  Google Scholar 

  26. Norris, S. P., Macnab, J. S., & Phillips, L. M. (2007). Cognitive modeling of performance on diagnostic achievement tests. In J. P. Leighton & M. J. Gierl (Eds.), Cognitive diagnostic assessment for education: Theory and applications (pp. 61–84). Cambridge: Cambridge University Press.

    Google Scholar 

  27. Rupp, A. A. (2009). Software for calibrating diagnostic classification models: An overview of the current state-of-the-art. In Symposium conducted at the meeting of the American Educational Research Association, San Diego, CA.

  28. Rupp, A. A., & Templin, J. L. (2008). The effects of Q-matrix misspecification on parameter estimates and classification accuracy in the DINA model. Educational and Psychological Measurement, 68(1), 78–96.

    Article  Google Scholar 

  29. Rupp, A. A., Templin, J. L., & Henson, R. A. (2010). Diagnostic measurement: Theory, methods, and applications. New York: Guilford Press.

    Google Scholar 

  30. Tatsuoka, C. (2002). Data analytic methods for latent partially ordered classification models. Journal of the Royal Statistical Society: Series C (Applied Statistics), 51(3), 337–350.

    Article  Google Scholar 

  31. Tatsuoka, K. K. (1984). Analysis of errors in fraction addition and subtraction problems. Computer-Based Education Research Laboratory, University of Illinois at Urbana-Champaign.

  32. Templin, J. L., & Henson, R. A. (2006). A Bayesian method for incorporating uncertainty into Q-matrix estimation in skills assessment. In Symposium conducted at the meeting of the American Educational Research Association, San Diego, CA.

  33. Templin, J. L., & Hoffman, L. (2013). Obtaining diagnostic classification model estimates using Mplus. Educational Measurement: Issues and Practice, 32(2), 37–50.

    Article  Google Scholar 

  34. von Davier, M. (2014). The DINA model as a constrained general diagnostic model: Two variants of a model equivalency. British Journal of Mathematical and Statistical Psychology, 67(1), 49–71.

  35. Xiang, R. (2013). Nonlinear Penalized Estimation of True Q-matrix in Cognitive Diagnostic Models (Unpublished doctoral dissertation). Columbia University.

  36. Xu, G., & Zhang, S. (2016). Identifiability of diagnostic classification models. Psychometrika, 81(3), 625–649.

    Article  PubMed  Google Scholar 

Download references

Author information

Affiliations

  1. Department of Mathematics & Statistics, University of Nevada, Reno, 1664 N. Virginia Street, Reno, NV, 89557 , USA

    Yinghan Chen

  2. Department of Statistics, University of Illinois at Urbana-Champaign, 725 South Wright Street, Champaign, IL, 61820 , USA

    Steven Andrew Culpepper, Yuguo Chen & Jeffrey Douglas

Authors
  1. Yinghan Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Steven Andrew Culpepper
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yuguo Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jeffrey Douglas
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Steven Andrew Culpepper.

Electronic supplementary material

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Chen, Y., Culpepper, S.A., Chen, Y. et al. Bayesian Estimation of the DINA Q matrix. Psychometrika 83, 89–108 (2018). https://doi.org/10.1007/s11336-017-9579-4

Download citation

Keywords

  • cognitive diagnosis models
  • deterministic inputs
  • noisy “and” gate (DINA) model
  • Q matrix
  • Bayesian statistics
  • fraction-subtraction data