Skip to main content
SpringerLink
Log in

DIAGNOSTIC Classification Analysis of Problem-Solving Competence using Process Data: An Item Expansion Method

  • Application Reviews and Case Studies
  • Published:
Psychometrika Aims and scope Submit manuscript

Abstract

Process data refer to data recorded in computer-based assessments (CBAs) that reflect respondents’ problem-solving processes and provide greater insight into how respondents solve problems, in addition to how well they solve them. Using the rich information contained in process data, this study proposed an item expansion method to analyze action-level process data from the perspective of diagnostic classification in order to comprehensively understand respondents’ problem-solving competence. The proposed method cannot only estimate respondents’ problem-solving ability along a continuum, but also classify respondents according to their problem-solving skills. To illustrate the application and advantages of the proposed method, a Programme for International Student Assessment (PISA) problem-solving item was used. The results indicated that (a) the estimated latent classes provided more detailed diagnoses of respondents’ problem-solving skills than the observed score categories; (b) although only one item was used, the estimated higher-order latent ability reflected the respondents’ problem-solving ability more accurately than the unidimensional latent ability estimated from the outcome data; and (c) interactions among problem-solving skills followed the conjunctive condensation rule, which indicated that the specific action sequence appeared only when a respondent mastered all required problem solving skills. In conclusion, the proposed diagnostic classification approach is feasible and promising analyzing process data.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

Notes

  1. More details about this item can be found in https://www.oecd.org/pisa/test-2012/testquestions/question5/ retrieved on July 11th, 2020.

References

  • Akaike, H. (1981). Likelihood of a model and information criteria. Journal of Econometrics, 16(1), 3–14.

    Article  Google Scholar 

  • Bock, R. D., & Aitkin, M. (1981). Marginal maximum likelihood estimation of item parameters: Application of an EM algorithm. Psychometrika, 46, 443–459.

    Article  Google Scholar 

  • Bozdogan, H. (1987). Model selection and Akaike’s information criterion (AIC): The general theory and its analytical extensions. Psychometrika, 52(3), 345–370. https://doi.org/10.1007/BF02294361

    Article  Google Scholar 

  • Chen, J., de la Torre, J., & Zhang, Z. (2013). Relative and absolute fit evaluation in cognitive diagnosis modeling. Journal of Educational Measurement, 50(2), 123–140.

    Article  Google Scholar 

  • Chen, Y., Culpepper, S. A., Chen, Y., & Douglas, J. (2018). Bayesian estimation of the DINA Q matrix. Psychometrika, 83(1), 89–108.

    Article  PubMed  Google Scholar 

  • Chen, Y., Culpepper, S. A., Wang, S., & Douglas, J. (2018). A hidden Markov model for learning trajectories in cognitive diagnosis with application to spatial rotation skills. Applied Psychological Measurement, 42(1), 5–23.

    Article  PubMed  Google Scholar 

  • Chen, Y., Li, X., Liu, J., & Ying, Z. (2017). Recommendation system for adaptive learning. Applied Psychological Measurement, 42(1), 24–41. https://doi.org/10.1177/0146621617697959

    Article  PubMed  PubMed Central  Google Scholar 

  • Chiu, C.-Y. (2013). Statistical refinement of the Q-matrix in cognitive diagnosis. Applied Psychological Measurement, 37(8), 598–618. https://doi.org/10.1177/0146621613488436

    Article  Google Scholar 

  • de la Torre, J. (2011). The generalized DINA model framework. Psychometrika, 76(2), 179–199.

    Article  Google Scholar 

  • de la Torre, J., & Chiu, C.-Y. (2016). A general method of empirical Q-matrix validation. Psychometrika, 81(2), 253–273. https://doi.org/10.1007/s11336-015-9467-8

    Article  PubMed  Google Scholar 

  • de la Torre, J., & Douglas, J. (2004). Higher-order latent trait models for cognitive diagnosis. Psychometrika, 69(3), 333–353. https://doi.org/10.1007/BF02295640

    Article  Google Scholar 

  • de la Torre, J., Hong, Y., & Deng, W. (2010). Factors affecting the item parameter estimation and classification accuracy of the DINA model. Journal of Educational Measurement, 47(2), 227–249.

    Article  Google Scholar 

  • DiCerbo, K. E., Liu, J., Rutstein, D. W., Choi, Y., & Behrens, J. T. (2011). Visual analysis of sequential log data from complex performance assessments,” [Paper presentation]. In Annual meeting of the American Educational Research Association, New Orleans, LA, USA.

  • Greiff, S., Niepel, C., Scherer, R., & Martin, R. (2016). Understanding students’ performance in a computer-based assessment of complex problem solving: An analysis of behavioral data from computer-generated log files. Computers in Human Behavior, 61, 36–46. https://doi.org/10.1016/j.chb.2016.02.095

    Article  Google Scholar 

  • Gu, Y., & Xu, G. (2019). Partial identifiability of restricted latent class models. arXiv preprint arXiv:1803.04353

  • He, Q., & von Davier, M. (2016). Analyzing process data from problem-solving items with n-grams: Insights from a computer-based large-scale assessment. In Y. Rosen, S. Ferrara, & M. Mosharraf (Eds.), Handbook of research on technology tools for real-world skill development (Vol. 2, pp. 749–776). Hershey: Information Science Reference. https://doi.org/10.4018/978-1-4666-9441-5.ch029

  • Howard, L., Johnson, J., & Neitzel, C. (2010). Examining learner control in a structured inquiry cycle using process mining. In Proceedings of the 3rd international conference on educational data mining (pp. 71–80). Available online at: https://files.eric.ed.gov/fulltext/ED538834.pdf#page=83 (Accessed August 26, 2018).

  • Junker, B. W., & Sijtsma, K. (2001). Cognitive assessment models with few assumptions, and connections with nonparametric item response theory. Applied Psychological Measurement, 25(3), 258–272.

    Article  Google Scholar 

  • Jiao, H., Liao, D., & Zhan, P. (2019). Utilizing process data for cognitive diagnosis. In M. von Davier & Y.-S. Lee (Eds.), Handbook of diagnostic classification models. New York: Springer.

    Google Scholar 

  • Kroehne, U., & Goldhammer, F. (2018). How to conceptualize, represent, and analyze log data from technology-based assessments? A generic framework and an application to questionnaire items. Behaviormetrika, 45(2), 527–563. https://doi.org/10.1007/s41237-018-0063-y

    Article  Google Scholar 

  • LaMar, M. M. (2018). Markov decision process measurement model. Psychometrika, 83(1), 67–88. https://doi.org/10.1007/s11336-017-9570-0

    Article  PubMed  Google Scholar 

  • Leighton, J. P., Gierl, M. J., & Hunka, S. M. (2004). The attribute hierarchy model: An approach for integrating cognitive theory with assessment practice. Journal of Educational Measurement, 41(3), 205–237. https://doi.org/10.1111/j.1745-3984.2004.tb01163.x

    Article  Google Scholar 

  • Levy, R. (2014). Dynamic Bayesian network modeling of game based diagnostic assessments (CRESST Report No. 837). Los Angeles, CA: University of California, National Center for Research on Evaluation, Standards, and Student Testing (CRESST), Center for Studies in Education, UCLA. Retrieved August 26, 2018, from https://files.eric.ed.gov/fulltext/ED555714.pdf

  • Liao, D., He, Q., & Jiao, H. (2019). Mapping background variables with sequential patterns in problem-solving environments: An investigation of us adults’ employment status in PIAAC. Frontiers in Psychology, 10, 646.

    Article  PubMed  PubMed Central  Google Scholar 

  • Liu, H., Liu, Y., & Li, M. (2018). Analysis of process data of PISA 2012 computer-based problem solving: Application of the modified multilevel mixture IRT model. Frontiers in Psychology, 9, 1372. https://doi.org/10.3389/fpsyg.2018.01372

    Article  PubMed  PubMed Central  Google Scholar 

  • Liu, J., Xu, G., & Ying, Z. (2012). Data-driven learning of Q-matrix. Applied Psychological Measurement, 36(7), 548–564.

    Article  PubMed  Google Scholar 

  • Ma, W., & de la Torre, J. (2020a). Choosing between CDM and unidimensional IRT: The proportional reasoning test case. Measurement Interdisciplinary Research and Perspectives, 18(2), 87–96. https://doi.org/10.1080/15366367.2019.1697122

    Article  Google Scholar 

  • Ma, W., & de la Torre, J. (2020b). GDINA: An R package for cognitive diagnosis modeling. Journal of Statistical Software, 93(14), 1–26. https://doi.org/10.18637/jss.v093.i14

    Article  Google Scholar 

  • Maris, E. (1999). Estimating multiple classification latent class models. Psychometrika, 64(2), 187–212.

    Article  Google Scholar 

  • Masters, G. N. (1982). A Rasch model for partial credit model: Application of an EM algorithm. Applied Psychological Measurement, 16(2), 159–174.

    Google Scholar 

  • Maydeu-Olivares, A. (2013). Goodness-of-fit assessment of item response theory models. Measurement, 11(3), 71–101.

    Google Scholar 

  • OECD. (2014). PISA 2012 results: Creative problem solving: Students’ skills in tackling real-life problems (Volume V). PISA, OECD Publishing: https://doi.org/10.1787/9789264208070-en

  • Qiao, X., & Jiao, H. (2018). Data mining techniques in analyzing process data: A didactic. Frontiers in Psychology, 9, 2231.

    Article  PubMed  PubMed Central  Google Scholar 

  • Rabiner, L. (1989). A tutorial on hidden Markov models and selected applications in speech recognition. Proceedings of IEEE, 77(2), 257–285.

    Article  Google Scholar 

  • Ravand, H. (2016). Application of a cognitive diagnostic model to a high-stakes reading comprehension test. Journal of Psychoeducational Assessment, 34(8), 782–799. https://doi.org/10.1177/0734282915623053

    Article  Google Scholar 

  • Robitzsch, A., Kiefer, T., & Wu, M. (2020). TAM: Test analysis modules. [R package version 3.5-19]. Retrieved from http://CRAN.R-project.org/package=TAM.

  • Roduta Roberts, M., Alves, C. B., Chu, M.-W., Thompson, M., Bahry, L. M., & Gotzmann, A. (2014). Testing expert-based versus student-based cognitive models for a grade 3 diagnostic mathematics assessment. Applied Measurement in Education, 27(3), 173–195. https://doi.org/10.1080/08957347.2014.905787

    Article  Google Scholar 

  • Sao Pedro, M. A., Baker, R. S. J. d., & Gobert, J. D. (2012). Improving construct validity yields better models of systematic inquiry, even with less information. In J. Masthoff, B. Mobasher, M. C. Desmarais, & R. Nkambou (Eds.), User modeling, adaptation, and personalization: Proceedings of the 20th UMAP conference (pp. 249–260). Berlin, Germany: Springer.

  • Schwarz, G. (1978). Estimating the dimension of a model. The Annals of Statistics, 6(2), 461–464.

    Article  Google Scholar 

  • Sclove, S. L. (1987). Application of model-selection criteria to some problems in multivariate analysis. Psychometrika, 52(3), 333–343. https://doi.org/10.1007/BF02294360

    Article  Google Scholar 

  • Shu, Z., Bergner, Y., Zhu, M., Hao, J., & von Davier, A. A. (2017). An item response theory analysis of problem-solving processes in scenario-based tasks. Psychological Test and Assessment Modeling, 59(1), 109–131.

    Google Scholar 

  • Tang, F., & Zhan, P. (2021). Does diagnostic feedback promote learning? Evidence from a longitudinal cognitive diagnostic assessment. AERA Open. https://doi.org/10.13140/RG.2.2.23511.19365

  • Tang, X., Wang, Z., Liu, J., & Ying, Z. (2020). An exploratory analysis of the latent structure of process data via action sequence autoencoders. British Journal of Mathematical and Statistical Psychology, 74(1), 1–33. https://doi.org/10.1111/bmsp.12203

    Article  PubMed  Google Scholar 

  • Tatsuoka, K. K. (1983). Rule-space: An approach for dealing with misconceptions based on item response theory. Journal of Educational Measurement, 20(4), 345–354. https://doi.org/10.1111/j.1745-3984.1983.tb00212.x

    Article  Google Scholar 

  • Templin, J., & Henson, R. A. (2006). Measurement of psychological disorders using cognitive diagnosis models. Psychological Methods, 11(3), 287–305.

    Article  PubMed  Google Scholar 

  • van der Linden, W. J. (2007). A hierarchical framework for modeling speed and accuracy on test items. Psychometrika, 72(3), 287–308. https://doi.org/10.1007/s11336-006-1478-z

    Article  Google Scholar 

  • von Davier, M., & Lee, Y.-S. (2019). Handbook of diagnostic classification models: Models and model extensions, applications, software packages. Springer.

  • Wang, S., & Chen, Y. (2020). Using response times and response accuracy to measure fluency within cognitive diagnosis models. Psychometrika, 85, 600–629.

    Article  PubMed  Google Scholar 

  • Wang, W., Song, L., Chen, P., Meng, Y., & Ding, S. (2015). Attribute-level and pattern-level classification consistency and accuracy indices for cognitive diagnostic assessment. Journal of Educational Measurement, 52(4), 457–476. https://doi.org/10.1111/jedm.12096.

    Article  Google Scholar 

  • Wu, H.-M. (2019). Online individualised tutor for improving mathematics learning: A cognitive diagnostic model approach. Educational Psychology, 39(10), 1218–1232. https://doi.org/10.1080/01443410.2018.1494819

    Article  Google Scholar 

  • Xu, H., Fang, G., & Ying, Z. (2020). A latent topic model with Markovian transition for process data. British Journal of Mathematical and Statistical Psychology. https://doi.org/10.1111/bmsp.12197

    Article  PubMed  Google Scholar 

  • Zhan, P. (2020). A Markov estimation strategy for longitudinal learning diagnosis: Providing timely diagnostic feedback. Educational and Psychological Measurement, 80(6), 1145–1167.

    Article  PubMed  PubMed Central  Google Scholar 

  • Zhan, P., Jiao, H., & Liao, D. (2018). Cognitive diagnosis modelling incorporating item response times. British Journal of Mathematical and Statistical Psychology, 71(2), 262–286.

    Article  PubMed  Google Scholar 

  • Zhan, P., Jiao, H., Liao, D., & Li, F. (2019). A longitudinal higher-order diagnostic classification model. Journal of Educational and Behavioral Statistics, 44(3), 251–281.

    Article  Google Scholar 

  • Zhu, M., Shu, Z., & von Davier, A. A. (2016). Using networks to visualize and analyze process data for educational assessment. Journal of Educational Measurement, 53(2), 190–211.

    Article  Google Scholar 

Download references

Funding

This work was supported by the National Natural Science Foundation of China (Grant No. 31900795).

Author information

Authors and Affiliations

  1. Department of Psychology, College of Teacher Education, Zhejiang Normal University, Jinhua, China

    Peida Zhan

  2. Measurement, Statistics, and Evaluation, Department of Human Development and Quantitative Methodology, University of Maryland, 1230 Benjamin Building, College Park, MD, 20742, USA

    Xin Qiao

Authors
  1. Peida Zhan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xin Qiao
    View author publications

    You can also search for this author in PubMed Google Scholar

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Peida Zhan and Xin Qiao are co-first authors.

Supplementary Information

Below is the link to the electronic supplementary material.

Supplementary file 1 (pdf 971 KB)

Supplementary file 2 (zip 21098 KB)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhan, P., Qiao, X. DIAGNOSTIC Classification Analysis of Problem-Solving Competence using Process Data: An Item Expansion Method. Psychometrika 87, 1529–1547 (2022). https://doi.org/10.1007/s11336-022-09855-9

Download citation

  • Received:

  • Revised:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11336-022-09855-9

Keywords

Search

Navigation