Abstract
Process data recorded by computer-based assessments reflect how respondents solve problems and thus contain rich information about respondents as well as tasks. Considering that different respondents may exhibit different behavioral characteristics during problem-solving process, in this study, we propose a mixture one-parameter state response (Mix1P-SR) measurement model. This model assumes that respondents belong to discrete latent classes with different propensities towards responses to task states during the problem-solving process, and the varying response propensities are captured by different state parameters across classes. A Markov Chain Monte Carlo algorithm for the estimation of model parameters and classification of respondents is described. The simulation study shows that the Mix1P-SR model could recover parameters well on the premise that the average sequence length was not too short. Moreover, larger sample size, longer sequences, more uniform mixing proportions, and lower interclass similarity facilitated model convergence, model selection, and parameter estimation accuracy, with sequence length being particularly important. Based on the empirical data from PISA 2012, the Mix1P-SR model identified two latent classes of respondents. They had different patterns of state easiness parameters and exhibited different state response patterns, which affected their problem solving results. Implications for model application and future research directions are discussed.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Data and Code Availability
The data that support the findings of this study are openly available from the OECD website: http://www.oecd.org/pisa/pisaproducts/database-cbapisa2012.htm. The R code for simulating data and implementing the MCMC algorithm is available at https://osf.io/pqbxr/?view_only=d798407027424f95aae5ea9824259254.
Notes
We also used WinBUGS to implement the MCMC algorithm for the Mix1P-SR model with three classes. Results showed that although the model converged, only two meaningful classes were identified, while no observations were assigned to the third class and its mixing proportion was zero. In fact, in the preliminary analysis, we found that for homogeneous data (with a single class), the Mix1P-SR model with more than one class did not converge at all. Also, we generated data with three classes and found that the three-class model with our algorithm could converge, but the four-class model failed. Therefore, we infer that the current mixture model with more than k classes using the current algorithm could not converge when fitted to data with k classes.
Note that in the comparison of the two models, results of the one-class model were summarized based on only the replications where the two-class model converged successfully. Indeed, these results were very close to the average results of the one-class model over all 50 replications in each condition.
References
Akaike, H. 1974. A new look at the statistical model identification. IEEE Transactions on Automatic Control 19: 716–723. https://doi.org/10.1109/TAC.1974.1100705.
Arieli-Attali, M., L. Ou, and V.R. Simmering. 2019. Understanding test Takers’ choices in a self-adapted test: A hidden Markov modeling of process data. Frontiers in Psychology 10: 83. https://doi.org/10.3389/fpsyg.2019.00083.
Bergner, Y., Shu, Z., and von Davier, A. 2014. Visualization and confirmatory clustering of sequence data from a simulation-based assessment task. In Proceedings of the 7th International Conference on Educational Data Mining, pp.177–184. International Educational Data Mining Society. https://pdfs.semanticscholar.org/b0da/eb27f2982ced8ab9fc6646edc33abad04742.pdf
Bolt, D.M., A.S. Cohen, and J.A. Wollack. 2001. A mixture item response model for multiple-choice data. Journal of Educational and Behavioral Statistics 26 (4): 3811–4409. https://doi.org/10.3102/1076998602600438.
Brooks, S.P., and A. Gelman. 1998. General methods for monitoring convergence of iterative simulations. Journal of Computational and Graphical Statistics 7 (4): 434–455. https://doi.org/10.1080/10618600.1998.10474787.
Buchner, A., and J. Funke. 1993. Finite-state automata: Dynamic task environments in problem-solving research. The Quarterly Journal of Experimental Psychology 46 (1): 83–118. https://doi.org/10.1080/14640749308401068.
Chen, Y. 2020. A continuous-time dynamic choice measurement model for problem-solving process data. Psychometrika 85 (4): 1052–1075. https://doi.org/10.1007/s11336-020-09734-1.
Chen, Yi., Jingru Zhang, Yi. Yang, and Young‐Sun. Lee. 2022. Latent space model for process data. Journal of Educational Measurement 59 (4): 517–535. https://doi.org/10.1111/jedm.12337.
Chib, S., and E. Greenberg. 1995. Understanding the metropolis-hastings algorithm. The American Statistician 49: 327–335. https://doi.org/10.1080/00031305.1995.10476177.
Cho, S.J., A.S. Cohen, and S.H. Kim. 2013. Markov chain Monte Carlo estimation of a mixture item response theory model. Journal of Statistical Computation and Simulation 83 (2): 278–306. https://doi.org/10.1080/00949655.2011.603090.
Choi, Y.J., and A.S. Cohen. 2020. Comparison of scale identification methods in mixture IRT models. Journal of Modern Applied Statistical Methods 18 (1): 31. https://doi.org/10.22237/jmasm/1556669700.
Choi, Y. J. 2014. Metric identification in mixture IRT models (Doctoral dissertation). University of Georgia.
Cohen, Jacob. 1988. Statistical power analysis for the behavioral sciences, 2nd ed. Mahwah: Lawrence Erlbaum Associates.
Eichmann, B., S. Greiff, J. Naumann, L. Brandhuber, and F. Goldhammer. 2020. Exploring behavioural patterns during complex problem-solving. Journal of Computer Assisted Learning 36 (6): 933–956. https://doi.org/10.1111/jcal.12451.
Funke, J. 2001. Dynamic systems as tools for analysing human judgement. Thinking and Reasoning 7: 69–89. https://doi.org/10.1080/13546780042000046.
Geisser, S., and W.F. Eddy. 1979. A predictive approach to model selection. Journal of the American Statistical Association 74: 153–160. https://doi.org/10.1080/01621459.1979.10481632.
Gelfand, A.E., and D.K. Dey. 1994. Bayesian model choice: Asymptotics and exact calculations. Journal of the Royal Statistical Society, Series B 56: 501–514. https://doi.org/10.1111/j.2517-6161.1994.tb01996.x.
Gelman, A., and D.B. Rubin. 1992. Inference from iterative simulation using multiple sequences. Statistical Science 7: 457–511. https://doi.org/10.1214/ss/1177011136.
Gilks, W.R., S. Richardson, and D.J. Spiegelhalter. 1996. Markov chain Monte Carlo in practice. Boca Raton: Chapman & Hall.
Gnaldi, M., S. Bacci, T. Kunze, and S. Greiff. 2020. Students’ complex problem solving profiles. Psychometrika 85 (2): 469–501. https://doi.org/10.1007/s11336-020-09709-2.
Greiff, S., S. Wüstenberg, and F. Avvisati. 2015. Computer-generated log-file analyses as a window into students’ minds? A showcase study based on the PISA 2012 assessment of problem solving. Computers and Education 91: 92–105. https://doi.org/10.1016/j.compedu.2015.10.018.
Greiff, S., C. Niepel, R. Scherer, and R. Martin. 2016. Understanding students’ performance in a computer-based assessment of complex problem solving: An analysis of behavioral data from computer-generated log files. Computers in Human Behavior 61: 36–46. https://doi.org/10.1016/j.chb.2016.02.095.
Gu, H. 2023. Data, big tech, and the new concept of sovereignty. Journal of Chinese Political Science. https://doi.org/10.1007/s11366-023-09855-1.
Han, Y., H. Liu, and F. Ji. 2022. A sequential response model for analyzing process data on technology-based problem-solving tasks. Multivariate Behavioral Research 57 (6): 960–977. https://doi.org/10.1080/00273171.2021.1932403.
He, Q., and M. von Davier. 2015. Identifying feature sequences from process data in problem-solving items with n-grams. In Quantitative psychology research, ed. L.A. van der Ark, D.M. Bolt, W.-C. Wang, J.A. Douglas, and S.-M. Chow, 173–190. Berlin: Springer. https://doi.org/10.1007/978-3-319-19977-1_13.
He, Q., and M. von Davier. 2016. Analyzing process data from problem-solving items with ngrams: Insights from a computer-based large-scale assessment. In Handbook of research on technology tools for real-world skill development, ed. Y. Rosen, S. Ferrara, and M. Mosharraf, 750–777. Hershey: IGI Global.
He, Q., D. Liao, and H. Jiao. 2019. Clustering behavioral patterns using process data in PIAAC Problem-Solving items. In Theoretical and practical advances in computer-based educational measurement, ed. B.P. Veldkamp and C. Sluijter, 189–212. Berlin: Springer.
Huang, Y., and M. Mayer. 2023. Power in the age of datafication: Exploring China’s global data power. Journal of Chinese Political Science 28: 25–49. https://doi.org/10.1007/s11366-022-09816-0.
Junker, B.W., R.J. Patz, and N.M. VanHoudnos. 2016. Markov chain Monte Carlo for item response models. In Handbook of item response theory, volume two: Statistical tools, ed. W.J. van der Linden, 271–325. Boca Raton: CRC Press.
Kerr, D., Chung, G., & Iseli, M. (2011). The feasibility of using cluster analysis to examine log data from educational video games (CRESST Report No. 790). University of California, National Center for Research on Evaluation, Standards, and Student Testing (CRESST), Center for Studies in Education, UCLA. https://files.eric.ed.gov/fulltext/ED520531.pdf
Kim, J.S., and D.M. Bolt. 2007. Estimating item response theory models using Markov chain Monte Carlo methods. Educational Measurement: Issues and Practice 26 (4): 38–51. https://doi.org/10.1111/j.1745-3992.2007.00107.x.
LaMar, M.M. 2018. Markov decision process measurement model. Psychometrika 83 (1): 67–88. https://doi.org/10.1007/s11336-017-9570-0.
Levy, R., and R.J. Mislevy. 2016. Bayesian psychometric modeling. Boca Raton: CRC Press.
Li, F., A.S. Cohen, S.-H. Kim, and S.-J. Cho. 2009. Model selection methods for mixture dichotomous IRT models. Applied Psychological Measurement 33: 353–373. https://doi.org/10.1177/0146621608326422.
Li, T., H. Jiao, and G. Macready. 2016. Different approaches to covariate inclusion in the mixture Rasch model. Educational and Psychological Measurement 76 (5): 848–872. https://doi.org/10.1177/0013164415610380.
Liao, D., Q. He, and H. Jiao. 2019. Mapping background variables with sequential patterns in problem-solving environments: An investigation of US Adults’ employment status in PIAAC. Frontiers in Psychology 10: 646. https://doi.org/10.3389/fpsyg.2019.00646.
Liu, H., Y. Liu, and M. Li. 2018. Analysis of process data of PISA 2012 computer-based problem solving: Application of the modified multilevel mixture IRT model. Frontiers in Psychology 9: 1372. https://doi.org/10.3389/fpsyg.2018.01372.
McLachlan, G., S. Lee, and S. Rathnayake. 2019. Finite mixture models. Annual Review of Statistics and Its Application 6: 355–378. https://doi.org/10.1146/annurev-statistics-031017-100325.
Mislevy, R.J. 2019. Advances in measurement and cognition. The ANNALS of the American Academy of Political and Social Science 683 (1): 164–182. https://doi.org/10.1177/0002716219843816.
Mislevy, R.J., and N. Verhelst. 1990. Modeling item responses when different subjects employ different solution strategies. Psychometrika 55 (2): 195–215. https://doi.org/10.1007/BF02295283.
Mosel, J.N., and J.B. Roberts. 1954. The comparability of measures of profile similarity: An empirical study. Journal of Consulting Psychology 18 (1): 61–66. https://doi.org/10.1037/h0058323.
OECD. (2014). PISA 2012 results: Creative problem solving: Students’ skills in tackling real-life problems, vol. V, (OECD Publishing), http://www.oecd.org/education/pisa-2012-results-volumev.htm
Patz, R.J., and B.W. Junker. 1999. A straightforward approach to Markov chain Monte Carlo methods for item response models. Journal of Educational Behavioral Statistics 24 (2): 146–178. https://doi.org/10.3102/10769986024002146.
Qiao, X., and H. Jiao. 2018. Data mining techniques in analyzing process data: A didactic. Frontiers in Psychology 9: 2231. https://doi.org/10.3389/fpsyg.2018.02231.
Rosenthal, J.S. 2011. Optimal proposal distributions and adaptive MCMC. In Handbook of Markov chain Monte Carlo, ed. S. Brooks, A. Gelman, G.L. Jones, and X.-L. Meng, 93–111. Boca Raton: Chapman and Hall/CRC.
Rost, J. 1990. Rasch models in latent classes: An integration of two approaches to item analysis. Applied Psychological Measurement 14 (3): 271–282. https://doi.org/10.1177/014662169001400305.
Schwarz, G. 1978. Estimating the dimension of a model. Annals of Statistics 6: 461–464.
Sclove, L.S. 1987. Application of model-selection criteria to some problems in multivariate analysis. Psychometrika 52: 333–343. https://doi.org/10.1007/BF02294360.
Shu, Z., Y. Bergner, M. Zhu, J. Hao, and A.A. von Davier. 2017. An item response theory analysis of problem-solving processes in scenario-based tasks. Psychological Test and Assessment Modeling 59 (1): 109–131.
Spiegelhalter, D. J., Best, N. G., Carlin, B. P., and Van der Linde, A. 1998. Bayesian deviance, the effective number of parameters, and the comparison of arbitrarily complex models. Technical report, MRC Biostatistics Unit.
Stadler, M., S. Hofer, and S. Greiff. 2020. First among equals: log data indicates ability differences despite equal scores. Computers in Human Behavior 111: 106442. https://doi.org/10.1016/j.chb.2020.106442.
Tang, X., Z. Wang, Q. He, J. Liu, and Z. Ying. 2020. Latent feature extraction for process data via multidimensional scaling. Psychometrika 85 (2): 378–397. https://doi.org/10.1007/s11336-020-09708-3.
Tang, X., Z. Wang, J. Liu, and Z. Ying. 2021. An exploratory analysis of the latent structure of process data via action sequence autoencoders. British Journal of Mathematical and Statistical Psychology 74 (1): 1–33. https://doi.org/10.1111/bmsp.12203.
Xiao, Y., and H. Liu. 2024. A state response measurement model for problem-solving process data. Behavior Research Methods 56: 258–277. https://doi.org/10.3758/s13428-022-02042-9.
Xiao, Y., Q. He, B. Veldkamp, and H. Liu. 2021. Exploring latent states of problem-solving competence using hidden Markov model on process data. Journal of Computer Assisted Learning 37 (5): 1232–1247. https://doi.org/10.1111/jcal.12559.
Xiao, Y., B. Veldkamp, and H. Liu. 2022. Combining process information and item response modeling to estimate problem-solving ability. Educational Measurement: Issues and Practice 41 (2): 36–54. https://doi.org/10.1111/emip.12474.
Yu, X. 2021. Allocating personal data rights: Toward resolving conflicts of interest over personal data. Fudan Journal of the Humanities and Social Sciences 14: 549–563. https://doi.org/10.1007/s40647-021-00330-w.
Zhan, P., and X. Qiao. 2022. Diagnostic classification analysis of problem-solving competency using process data: An item expansion method. Psychometrika 87: 1529–1547. https://doi.org/10.1007/s11336-022-09855-9.
Acknowledgements
The authors thank the OECD-PISA (Program for International Student Assessment) team for granting access to the data source and instruments in this study. This work was supported by National Natural Science Foundation of China (Grant 32300938).
Funding
This work was supported by National Natural Science Foundation of China (Grant 32300938).
Author information
Authors and Affiliations
Contributions
Conceptualization: YX and HL; methodology: YX and HL; formal analysis and investigation: YX; writing—original draft preparation: YX; writing—review and editing: YX and HL; supervision: HYL.
Corresponding author
Ethics declarations
Conflict of interest
The authors claim there has no known conflict of interest to disclose.
Appendices
Appendix 1
See Appendix Tables
5,
6 and
7.
Appendix 2
See Appendix Tables
8 and
9.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Xiao, Y., Liu, H. A Mixture Modeling Approach to Detect Different Behavioral Patterns for Process Data. Fudan J. Hum. Soc. Sci. (2024). https://doi.org/10.1007/s40647-024-00405-4
Received:
Accepted:
Published:
DOI: https://doi.org/10.1007/s40647-024-00405-4