Abstract
Cognitive diagnostic assessment seeks to promote targeted instruction designed to address learners’ strengths and weaknesses within a specific domain. Most of the previous CDA applications, however, have merely implemented a single CDM. Few studies have investigated the applicability of multi-CDM to educational assessment data. To this end, the present study attempted to diagnose a sample of 740 college freshmen’s performance on a reading comprehension test designed by the PELDiaG (Personalized English Learning: Diagnosis & Guidance) research team from a key university in China with the multi-CDM. The model-data fit results showed that the multi-CDM outperformed any single CDMs and enhanced the interpretation of the inter-skill relationship as well. Consequently, both group and individual level diagnostic information were extracted and further synthesized into a fine-grained diagnostic feedback. The findings provided further evidence pertinent to the practicality of multi-CDM in reading comprehension tests. Finally, limitations and suggestions for further research were presented.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Alderson, J. C. (2000). Assessing reading. Cambridge: Cambridge University Press.
Bachman, L. F. (1990). Fundamental considerations in language testing. New York, NY: Oxford University Press.
Bernhardt, E. B. (2005). Process and procrastination in second language reading. Annual Review of Applied Linguistics, 25, 133–150. https://doi.org/10.1017/S0267190505000073.
Chen, H., & Chen, J. (2015). Exploring reading comprehension skill relationships through the G-DINA model. Educational Psychology, 36(6), 1–20. https://doi.org/10.1080/01443410.2015.1076764.
Chen, J., de la Torre, J., & Zhang, Z. (2013). Relative and absolute fit evaluation in cognitive diagnosis modeling. Journal of Educational Measurement, 50(2), 123–140. https://doi.org/10.1111/j.1745-3984.2012.00185.x.
Chen, W. H., & Thissen, D. (1997). Local dependence indexes for item pairs using item response theory. Journal of Educational and Behavioral Statistics, 22(3), 265–289. https://doi.org/10.3102/10769986022003265.
de la Torre, J. (2011). The generalized DINA model framework. Psychometrika, 76(2), 179–199. https://doi.org/10.1007/s11336-011-9207-7.
de la Torre, J., & Chiu, C. Y. (2016). A general method of empirical Q-matrix validation. Psychometrika, 81(2), 253–273. https://doi.org/10.1007/s11336-015-9467-8.
de la Torre, J., & Lee, Y. S. (2013). Evaluating the Wald test for item-level comparison of saturated and reduced models in cognitive diagnosis. Journal of Educational Measurement, 50(4), 355–373. https://doi.org/10.1111/jedm.12022.
DiBello, L. V., Roussos, L. A., & Stout, W. (2007). Review of cognitively diagnostic assessment and a summary of psychometric models. In C. R. Rao & S. Sinharay (Eds.), Handbook of statistics (Vol. 26, pp. 970–1030)., Psychometrics Amsterdam: Elsevier.
Ericsson, K. A., & Simon, H. A. (1993). Protocol analysis: Verbal reports as data. Cambridge, MA: The MIT Press.
Gough, P. B., & Tunmer, W. E. (1986). Decoding, reading, and reading disability. Remedial and Special Education, 7(1), 6–10. https://doi.org/10.1177/074193258600700104.
Grabe, W. (2009). Reading in a second language: Moving from theory to practice. Cambridge: Cambridge University Press.
Harding, L., Alderson, C., & Brunfaut, T. (2015). Diagnostic assessment of reading and listening in a second or foreign language: Elaborating on diagnostic principles. Language Testing, 32(3), 317–336. https://doi.org/10.1177/0265532214564505.
Hartz, S. (2002). A Bayesian framework for the unified model for assessing cognitive abilities: Blending theory with practicality (Unpublished doctoral dissertation). University of Illinois at Urbana-Champaign, IL.
Henson, R. A., Templin, J., & Willse, J. (2009). Defining a family of cognitive diagnosis models using log-linear models with latent variables. Psychometrika, 74(2), 191–210. https://doi.org/10.1007/s11336-008-9089-5.
Jang, E. E. (2009). Cognitive diagnostic assessment of L2 reading comprehension ability: Validity arguments for fusion model application to LanguEdge assessment. Language Testing, 26(1), 31–73. https://doi.org/10.1177/0265532208097336.
Jang, E. E., Dunlop, M., Park, G., & van der Boom, E. H. (2015). How do young learners with different profiles of reading skill mastery, perceived ability, and goal orientation respond to holistic diagnostic feedback? Language Testing, 32(3), 359–383. https://doi.org/10.1177/0265532215570924.
Jang, E. E., Dunlop, M., Wagner, M., Kim, Y. H., & Gu, Z. (2013). Elementary school ELLs’ reading skill profiles using cognitive diagnosis modeling: Roles of length of residence and home language environment. Language Learning, 63(3), 400–436. https://doi.org/10.1111/lang.12016.
Javidanmehr, Z., & Anani Sarab, M. R. (2019). Retrofitting non-diagnostic reading comprehension assessment: Application of the G-DINA model to a high stakes reading comprehension test. Language Assessment Quarterly, 16(3), 294–311. https://doi.org/10.1080/15434303.2019.1654479.
Junker, B. W., & Sijtsma, K. (2001). Cognitive assessment models with few assumptions, and connections with nonparametric item response theory. Applied Psychological Measurement, 25(3), 258–272. https://doi.org/10.1177/01466210122032064.
Kim, A. (2015). Exploring ways to provide diagnostic feedback with an ESL placement test: Cognitive diagnostic assessment of L2 reading ability. Language Testing, 32(2), 227–258. https://doi.org/10.1177/0265532214558457.
Kintsch, W. (1988). The role of knowledge in discourse processing: A construction-integration model. Psychological Review, 95(2), 163–182. https://doi.org/10.1037/0033-295x.95.2.163.
Kintsch, W., & Rawson, K. A. (2005). Comprehension. In M. J. Snowling & C. Hulme (Eds.), The science of reading: A handbook (pp. 209–226). Malden, MA: Blackwell Publishing.
Koda, K. (2005). Insights into second language reading: A cross-linguistic approach. New York, NY: Cambridge University Press.
Landis, J. R., & Koch, G. G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33(1), 159–174. https://doi.org/10.2307/2529310.
Lee, Y. W. (2004). Examining passage-related local item dependence (LID) and measurement construct using Q3 statistics in an EFL reading comprehension test. Language Testing, 21(1), 74–100. https://doi.org/10.1191/0265532204lt260oa.
Lee, Y. W. (2015). Diagnosing diagnostic language assessment. Language Testing, 32(3), 299–316. https://doi.org/10.1177/0265532214565387.
Lee, Y. W., & Sawaki, Y. (2009). Application of three cognitive diagnosis models to ESL reading and listening assessments. Language Assessment Quarterly, 6(3), 239–263. https://doi.org/10.1080/15434300903079562.
Lei, P. W., & Li, H. (2016). Performance of fit indices in choosing correct cognitive diagnostic models and Q-matrices. Applied Psychological Measurement, 40(6), 405–417. https://doi.org/10.1177/0146621616647954.
Leighton, J. P., & Gierl, M. J. (2007). Cognitive diagnostic assessment for education: Theory and applications. New York, NY: Cambridge University Press.
Li, H., Hunter, C. V., & Lei, P. W. (2016). The selection of cognitive diagnostic models for a reading test. Language Testing, 33(3), 391–409. https://doi.org/10.1177/0265532215590848.
Li, H., & Suen, H. (2013). Constructing and validating a Q-matrix for cognitive diagnostic analyses of a reading test. Educational Assessment, 18(1), 1–25. https://doi.org/10.1080/10627197.2013.761522.
Madison, M. J., & Bradshaw, L. P. (2018). Assessing growth in a diagnostic classification model framework. Psychometrika, 83(4), 963–990. https://doi.org/10.1007/s11336-018-9638-5.
McNeil, L. (2012). Extending the compensatory model of second language reading. System, 40(1), 64–76. https://doi.org/10.1016/j.system.2012.01.011.
Nichols, P. D. (1994). A framework for developing cognitively diagnostic assessments. Review of Educational Research, 64(4), 575–603. https://doi.org/10.2307/1170588.
Ravand, H. (2016). Application of a cognitive diagnostic model to a high-stakes reading comprehension test. Journal of Psychoeducational Assessment, 34(8), 782–799. https://doi.org/10.1177/0734282915623053.
Ravand, H., & Baghaei, P. (2020). Diagnostic classification models: Recent developments, practical issues, and prospects. International Journal of Testing, 20(1), 24–56. https://doi.org/10.1080/15305058.2019.1588278.
Ravand, H., & Robitzsch, A. (2018). Cognitive diagnostic model of best choice: A study of reading comprehension. Educational Psychology, 38(10), 1255–1277. https://doi.org/10.1080/01443410.2018.1489524.
Roberts, M. R., & Gierl, M. G. (2010). Developing score reports for cognitive diagnostic assessments. Educational Measurement: Issues and Practice, 29(3), 25–38. https://doi.org/10.1111/j.1745-3992.2010.00181.x.
Robitzsch, A., Kiefer, T., George, A. C., & Uenlue, A. (2018). CDM: Cognitive diagnosis modeling. R package version 6.3-45. https://CRAN.R-project.org/package=CDM.
Rojas, G., de la Torre, J., & Olea, J. (2012, April). Choosing between general and specific cognitive diagnosis models when the sample size is small. Paper presented at the meeting of the National Council on Measurement in Education, Vancouver, Canada.
Rupp, A. A., Templin, J., & Henson, R. (2010). Diagnostic measurement: Theory, methods, and applications. New York, NY: The Guildford Press.
Sadoski, M., & Paivio, A. (2007). Toward a unified theory of reading. Scientific Studies of Reading, 11(4), 337–356. https://doi.org/10.1080/10888430701530714.
Stanovich, K. E. (1980). Toward an interactive⁃compensatory model of individual differences in the development of reading fluency. Reading Research Quarterly, 16(1), 32–71. https://doi.org/10.2307/747348.
Templin, J., & Henson, R. A. (2006). Measurement of psychological disorders using cognitive diagnosis models. Psychological Methods, 11(3), 287–305. https://doi.org/10.1037/1082-989X.11.3.287.
Urquhart, S., & Weir, C. J. (1998). Reading in a second language: Process, product and practice. New York, NY: Longman.
von Davier, M. (2005). A general diagnostic model applied to language testing data (RR-05-16). Princeton, NJ: Educational Testing Service.
von Davier, M. (2014). The DINA model as a constrained general diagnostic model: Two variants of a model equivalency. British Journal of Mathematical and Statistical Psychology, 67(1), 49–71. https://doi.org/10.1111/bmsp.12003.
Wang, C., & Gierl, M. J. (2011). Using the attribute hierarchy method to make diagnostic inferences about examinees’ cognitive skills in critical reading. Journal of Educational Measurement, 48(2), 165–187. https://doi.org/10.1111/j.1745-3984.2011.00142.x.
Weir, C., Yang, H. Z., & Jin, Y. (2000). An empirical investigation of the componentiality of L2 reading in English for academic purposes. Cambridge: Cambridge University Press.
Yen, W. M. (1984). Effects of local item dependence on the fit and equating performance of the three parameter logistic model. Applied Psychological Measurement, 8(2), 125–145. https://doi.org/10.1177/014662168400800201.
Zhang, J. (2013). Relationships between missing responses and skill mastery profiles of cognitive diagnostic assessment (Unpublished doctoral dissertation). University of Toronto, Toronto, ON, Canada.
Acknowledgements
We would like to thank all the team members of PELDiaG for the great efforts they have made on constructing the diagnostic reading test prior to this study. We also want to express our sincere gratitude to the two anonymous reviewers for their constructive comments and suggestions for the refinement of this manuscript.
Funding
This work was supported by the National Social Science Fund of China under Grant No. 17BYY015, and the National Education Examinations Authority—British Council English Assessment Research under Grant No. EARG2020004.
Author information
Authors and Affiliations
Corresponding authors
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Du, W., Ma, X. Probing what’s behind the test score: application of multi-CDM to diagnose EFL learners’ reading performance. Read Writ 34, 1441–1466 (2021). https://doi.org/10.1007/s11145-021-10124-x
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11145-021-10124-x