Consistency Theory for the General Nonparametric Classification Method

Abstract

Parametric likelihood estimation is the prevailing method for fitting cognitive diagnosis models—also called diagnostic classification models (DCMs). Nonparametric concepts and methods that do not rely on a parametric statistical model have been proposed for cognitive diagnosis. These methods are particularly useful when sample sizes are small. The general nonparametric classification (GNPC) method for assigning examinees to proficiency classes can accommodate assessment data conforming to any diagnostic classification model that describes the probability of a correct item response as an increasing function of the number of required attributes mastered by an examinee (known as the “monotonicity assumption”). Hence, the GNPC method can be used with any model that can be represented as a general DCM. However, the statistical properties of the estimator of examinees’ proficiency class are currently unknown. In this article, the consistency theory of the GNPC proficiency-class estimator is developed and its statistical consistency is proven.

This is a preview of subscription content, log in to check access.

References

  1. Ayers, E., Nugent, R., & Dean, N. (2008). Skill set profile clustering based on student capability vectors computed from online tutoring data. In R. S. J. de Baker, T. Barnes, & J. E. Beck (Eds.), Educational data mining 2008: Proceedings of the 1st international conference on educational data mining, Montreal, Quebec, Canada (pp. 210–217). Retrieved from http://www.educationaldatamining.org/EDM2008/uploads/proc/full%20proceedings.pdf.

  2. Chiu, C.-Y. (2008). Cluster analysis for cognitive diagnosis: Theory and applications (Doctoral dissertation). Available from ProQuest Dissertations and Theses database. (UMI No. 3337778).

  3. Chiu, C.-Y., & Douglas, J. A. (2013). A nonparametric approach to cognitive diagnosis by proximity to ideal response profiles. Journal of Classification, 30, 225–250.

    Article  Google Scholar 

  4. Chiu, C.-Y., Douglas, J. A., & Li, X. (2009). Cluster analysis for cognitive diagnosis: Theory and applications. Psychometrika, 74, 633–665.

    Article  Google Scholar 

  5. Chiu, C.-Y., Sun, Y., & Bian, Y. (2018). Cognitive diagnosis for small educational programs: The general nonparametric classification method. Psychometrika, 83, 355–375.

    Article  PubMed  Google Scholar 

  6. de la Torre, J. (2009). DINA model and parameter estimation: A didactic. Journal of Educational and Behavioral Statistics, 34, 115–130.

    Article  Google Scholar 

  7. de la Torre, J. (2011). The generalized DINA model framework. Psychometrika, 76, 179–199.

    Article  Google Scholar 

  8. de la Torre, J., & Douglas, J. A. (2004). Higher-order latent trait models for cognitive diagnosis. Psychometrika, 69, 333–353.

    Article  Google Scholar 

  9. DiBello, L. V., Roussos, L. A., & Stout, W. F. (2007). Review of cognitively diagnostic assessment and a summary of psychometric models. In C. R. Rao & S. Sinharay (Eds.), Handbook of statistics: Volume 26. Psychometrics (pp. 979–1030). Amsterdam: Elsevier.

    Google Scholar 

  10. Haberman, S. J., & von Davier, M. (2007). Some notes on models for cognitively based skill diagnosis. In C. R. Rao & S. Sinharay (Eds.), Handbook of Statistics: Volume 26. Psychometrics (pp. 1031–1038). Amsterdam: Elsevier.

    Google Scholar 

  11. Haertel, E. H. (1989). Using restricted latent class models to map the skill structure of achievement items. Journal of Educational Measurement, 26, 333–352.

    Article  Google Scholar 

  12. Henson, R. A., Templin, J. L., & Willse, J. T. (2009). Defining a family of cognitive diagnosis models using log-linear models with latent variables. Psychometrika, 74, 191–210.

    Article  Google Scholar 

  13. Hoeffding, W. (1963). Probability inequalities for sums of bounded random variables. Journal of the American Statistical Association, 58(301), 13–30.

    Article  Google Scholar 

  14. Junker, B. W., & Sijtsma, K. (2001). Cognitive assessment models with few assumptions, and connections with nonparametric item response theory. Applied Psychological Measurement, 25, 258–272.

    Article  Google Scholar 

  15. Köhn, H.-F., & Chiu, C.-Y. (2016a). Conditions of completeness of the Q-matrix of tests for cognitive diagnosis. In A. L. van der Ark, D. M. Bolt, W.-C. Wang, J. A. Douglas, & M. Wiberg (Eds.), Quantitative psychology research: The 80th annual meeting of the psychometric society, Beijing, 2015 (pp. 255–264). New York: Springer.

  16. Köhn, H.-F., & Chiu, C.-Y. (2016b). A proof of the duality of the DINA model and the DINO model. Journal of Classification, 33, 171–184.

    Article  Google Scholar 

  17. Köhn, H.-F., & Chiu, C.-Y. (2017). A procedure for assessing completeness of the Q-matrix of cognitive diagnosis models. Psychometrika, 82, 112–132.

    Article  PubMed  Google Scholar 

  18. Leighton, J., & Gierl, M. (2007). Cognitive diagnostic assessment for education: Theory and applications. Cambridge: Cambridge University Press.

    Google Scholar 

  19. Ma, W., & de la Torre, J. (2019). GDINA: The generalized DINA model framework. R package version 2.3.2. Retrieved from the Comprehensive R Archive Network [CRAN] website https://cran.r-project.org/web/packages/GDINA/.

  20. Macready, G. B., & Dayton, C. M. (1977). The use of probabilistic models in the assessment of mastery. Journal of Educational Statistics, 33, 379–416.

    Google Scholar 

  21. Maris, E. (1999). Estimating multiple classification latent class models. Psychometrika, 64, 187–212.

    Article  Google Scholar 

  22. Nichols, P. D., Chipman, S. E., & Brennan, R. L. (1995). Cognitively diagnostic assessment. Mahwah, NJ: Erlbaum.

    Google Scholar 

  23. Park, Y. S., & Lee, Y.-S. (2011). Diagnostic cluster analysis of mathematics skills. IERI Monograph Series: Issues and Methodologies in Large-Scale Assessments, 4, 75–107.

    Google Scholar 

  24. Robitzsch, A., Kiefer, T., George, A. C., & Uenlue, A. (2018). CDM: Cognitive diagnosis modeling. R package version 7.1-20. Retrieved from the Comprehensive R Archive Network [CRAN] website https://cran.r-project.org/web/packages/CDM/.

  25. Rupp, A. A., Templin, J. L., & Henson, R. A. (2010). Diagnostic Measurement. Theory, Methods, and Applications. New York: Guilford.

    Google Scholar 

  26. Stout, W. (2002). Psychometrics: From practice to theory and back. Psychometrika, 67, 485–518.

    Article  Google Scholar 

  27. Tatsuoka, K. K. (1985). A probabilistic model for diagnosing misconception in the pattern classification approach. Journal of Educational and Behavioral Statistics, 12, 55–73.

    Article  Google Scholar 

  28. Tatsuoka, K. K. (2009). Cognitive assessment. An introduction to the rule space method. New York: Routledge/Taylor & Francis.

    Google Scholar 

  29. Templin, J. L., & Henson, R. A. (2006). Measurement of psychological disorders using cognitive diagnosis models. Psychological Methods, 11, 287–305.

    Article  PubMed  Google Scholar 

  30. von Davier, M. (2005). A general diagnostic model applied to language testing data (Research report No. RR-05-16). Princeton, NJ: Educational Testing Service.

  31. von Davier, M. (2008). A general diagnostic model applied to language testing data. British Journal of Mathematical and Statistical Psychology, 61, 287–301.

    Article  Google Scholar 

  32. Wang, S., & Douglas, J. (2015). Consistency of nonparametric classification in cognitive diagnosis. Psychometrika, 80, 85–100.

    Article  PubMed  Google Scholar 

  33. Willse, J., Henson, R., & Templin, J. (2007). Using sum scores or IRT in place of cognitive diagnosis models: Can existing or more familiar models do the job? Paper presented at the Annual Meeting of the National Council on Measurement in Education, Chicago, IL.

  34. Zheng, Y., & Chiu, C.-Y. (2014). NPCD: Nonparametric methods for cognitive diagnosis. R package version 1.0-5. Retrieved from the Comprehensive R Archive Network [CRAN] website http://CRAN.R-project.org/package=NPCD.

Download references

Acknowledgements

Funding was provided by National Science Foundation (Grant No. 1552563).

Author information

Affiliations

Authors

Corresponding author

Correspondence to Chia-Yi Chiu.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Chiu, C., Köhn, H. Consistency Theory for the General Nonparametric Classification Method. Psychometrika 84, 830–845 (2019). https://doi.org/10.1007/s11336-019-09660-x

Download citation

Keywords

  • cognitive diagnosis
  • Q-matrix
  • DINA model
  • DINO model
  • general DCM
  • G-DINA model
  • nonparametric classification
  • general nonparametric classification method