Abstract
Computational advancements in the last couple of decades have brought forth a new era of international large-scale assessments (ILSAs) in which the administration of computer-based tests is becoming the norm. Beyond collecting the correct/incorrect answers for each item, computer-based assessments are also able to collect a range of actions performed by the respondents in the computer testing application during the course of the test administration. Both respondents’ actions – starting a unit, clicking a button, spending time until inputting or submitting an answer, and so forth – and their overall behavior – what they do with their keyboards, mice, and even their own eyes – are recorded as a new set of data called “process data,” which are normally time-stamped and can be stored in the so-called log files. There may be plenty of useful insight into the respondent’s cognitive process, and process data can potentially become a relevant element in the scoring process of an assessment, validate test score interpretations, to name a few possibilities for the analysis of such data. This chapter aims to contribute to the body of knowledge in this area by offering (1) an introduction to process data, what kind of data it contains, its relation to the cognitive process, and how it can be organized into a proposed six-layered ecological framework that facilitates its analysis; (2) a literature review of 37 seminal and state-of-the-art publications produced in the last decade on the topic, which are then analyzed both in their chronological perspective as well as in how they fit into the ecological framework; and (3) a discussion of the potential and limitations of using process data in the assessment framework, including a view of what could be the next steps in the analysis of process data from ILSAs.
References
Azzolini, D., Bazoli, N., Lievore, I., Schizzerotto, A., & Vergolini, L. (2019). Beyond achievement. A comparative look into 15-year-olds’ school engagement, effort and perseverance in the European Union. In E. Union (Ed.), Entrepreneurship as extreme experience. https://doi.org/10.2766/98129
Chen, M. Y., & Zumbo, B. D. (2017). Ecological framework of item responding as validity evidence: An application of multilevel DIF modeling using PISA data. In Understanding and investigating response processes in validation research (pp. 53–68). Springer.
Chen, Y., Li, X., Liu, J., & Ying, Z. (2019). Statistical analysis of complex problem-solving process data: An event history analysis approach. Frontiers in Psychology, 10, 486. https://doi.org/10.3389/FPSYG.2019.00486
De Boeck, P., & Scalise, K. (2019). Collaborative problem solving: Processing actions, time, and performance. Frontiers in Psychology, 10, 1280. https://doi.org/10.3389/fpsyg.2019.01280
Engelhardt, L., & Goldhammer, F. (2019). Validating test score interpretations using time information. Frontiers in Psychology, 10, 1131. https://doi.org/10.3389/fpsyg.2019.01131
Goldhammer, F., Martens, T., Christoph, G., & Lüdtke, O. (2016). Test-taking engagement in PIAAC (No. 133; OECD Education Working Papers). https://doi.org/10.1787/5jlzfl6fhxs2-en
Goldhammer, F., Martens, T., & Lüdtke, O. (2017). Conditioning factors of test-taking engagement in PIAAC: An exploratory IRT modelling approach considering person and item characteristics. Large-Scale Assessments in Education, 5, 18. https://doi.org/10.1186/s40536-017-0051-9
Goldhammer, F., Naumann, J., Stelter, A., Tóth, K., Rölke, H., & Klieme, E. (2014). The time on task effect in reading and problem solving is moderated by task difficulty and skill: Insights from a computer-based large-scale assessment. Journal of Educational Psychology, 106(3), 608–626. https://doi.org/10.1037/a0034716
Goldhammer, F., & Zehner, F. (2017). What to make of and how to interpret process data. Measurement: Interdisciplinary Research and Perspectives, 15(3–4), 128–132. https://doi.org/10.1080/15366367.2017.1411651
Greiff, S., Wüstenberg, S., & Avvisati, F. (2015). Computer-generated log-file analyses as a window into students’ minds? A showcase study based on the PISA 2012 assessment of problem solving. Computers and Education, 91, 92–105. https://doi.org/10.1016/j.compedu.2015.10.018
Hahnel, C., Goldhammer, F., Kröhne, U., & Naumann, J. (2017). Reading digital text involves working memory updating based on task characteristics and reader behavior. Learning and Individual Differences, 59(October), 149–157. https://doi.org/10.1016/j.lindif.2017.09.001
Hahnel, C., Goldhammer, F., Kröhne, U., & Naumann, J. (2018). The role of reading skills in the evaluation of online information gathered from search engine environments. Computers in Human Behavior, 78, 223–234. https://doi.org/10.1016/j.chb.2017.10.004
Hahnel, C., Goldhammer, F., Naumann, J., & Kröhne, U. (2016). Effects of linear reading, basic computer skills, evaluating online information, and navigation on reading digital text. Computers in Human Behavior, 55, 486–500. https://doi.org/10.1016/j.chb.2015.09.042
Han, Z., He, Q., & von Davier, M. (2019). Predictive feature generation and selection from process data in PISA simulation-based environment: An implementation of tree-based ensemble methods. Frontiers in Psychology, 10, 2461. https://doi.org/10.3389/fpsyg.2019.02461
He, Q., Borgonovi, F., & Paccagnella, M. (2019). Using process data to understand adults’ problem-solving behaviours in PIAAC: Identifying generalised patterns across multiple tasks with sequence mining (No. 205; OECD Education Working Papers). OECD.
He, Q., & von Davier, M. (2015). Identifying feature sequences from process data in problem-solving items with N-grams. In Quantitative psychology research: The 79th annual meeting of the Psychometric Society (pp. 173–190). Springer.
Herde, C. N., Wüstenberg, S., & Greiff, S. (2016). Assessment of complex problem solving: What we know and what we don’t know. Applied Measurement in Education, 29(4), 265–277. https://doi.org/10.1080/08957347.2016.1209208
Hu, Y., Wu, B., & Gu, X. (2017). An eye tracking study of high-and low-performing students in solving interactive and analytical problems. Journal of Educational Technology & Society, 20(4), 300–311. https://doi.org/10.2307/26229225
Hubley, A. M., & Zumbo, B. D. (2017). Response processes in the context of validity: Setting the stage. In Understanding and investigating response processes in validation research (pp. 1–12). Springer.
Klotzke, K., & Fox, J. P. (2019). Bayesian covariance structure modelling of responses and process data. Frontiers in Psychology, 10, 1675. https://doi.org/10.3389/fpsyg.2019.01675
Kroehne, U., & Goldhammer, F. (2018). How to conceptualize, represent, and analyze log data from technology-based assessments? A generic framework and an application to questionnaire items. Behaviormetrika, 45, 527–563. https://doi.org/10.1007/s41237-018-0063-y
Kroehne, U., Hahnel, C., & Goldhammer, F. (2019). Invariance of the response processes between gender and modes in an assessment of reading. Frontiers in Applied Mathematics and Statistics, 5, 2. https://doi.org/10.3389/fams.2019.00002
Krstić, K., Šoškić, A., Ković, V., & Holmqvist, K. (2018). All good readers are the same, but every low-skilled reader is different: An eye-tracking study using PISA data. European Journal of Psychology of Education, 33, 521–541. https://doi.org/10.1007/s10212-018-0382-0
Lee, Y.-H., & Haberman, S. J. (2016). Investigating test-taking behaviors using timing and process data. International Journal of Testing, 16(3), 240–267. https://doi.org/10.1080/15305058.2015.1085385
Liao, D., He, Q., & Jiao, H. (2019). Mapping background variables with sequential patterns in problem-solving environments: An investigation of United States adults’ employment status in PIAAC. Frontiers in Psychology, 10, 646. https://doi.org/10.3389/fpsyg.2019.00646
Liu, H., Liu, Y., & Li, M. (2018). Analysis of process data of PISA 2012 computer-based problem solving: Application of the modified multilevel mixture IRT model. Frontiers in Psychology, 9, 1372. https://doi.org/10.3389/fpsyg.2018.01372
Maddox, B. (2017). Talk and gesture as process data. Measurement: Interdisciplinary Research and Perspectives, 15(3–4), 113–127. https://doi.org/10.1080/15366367.2017.1392821
Maddox, B., Bayliss, A. P., Fleming, P., Engelhardt, P. E., Edwards, S. G., & Borgonovi, F. (2018). Observing response processes with eye tracking in international large-scale assessments: Evidence from the OECD PIAAC assessment. European Journal of Psychology of Education, 33(3), 543–558. https://doi.org/10.1007/s10212-018-0380-2
Mislevy, R. J. (1994). Evidence and inference in educational assessment. Psychometrika, 59(4), 439–483. https://doi.org/10.1007/BF02294388
Mullis, I. V. S., & Martin, M. O. (2017). TIMSS 2019 assessment frameworks. Boston College, TIMSS & PIRLS International Study Center. http://timssandpirls.bc.edu/timss2019/frameworks/
Naumann, J. (2015). A model of online reading engagement: Linking engagement, navigation, and performance in digital reading. Computers in Human Behavior, 53, 263–277. https://doi.org/10.1016/j.chb.2015.06.051
Naumann, J. (2019). The skilled, the knowledgeable, and the motivated: Investigating the strategic allocation of time on task in a computer-based assessment. Frontiers in Psychology, 10, 1429. https://doi.org/10.3389/fpsyg.2019.01429
Naumann, J., & Goldhammer, F. (2017). Time-on-task effects in digital reading are non-linear and moderated by persons’ skills and tasks’ demands. Learning and Individual Differences, 53, 1–16. https://doi.org/10.1016/j.lindif.2016.10.002
OECD. (2010). PISA computer-based assessment of student skills in science. OECD Publishing. http://www.sourceoecd.org/education/9789264082021
OECD. (2013). LogDataAnalyzer. PIAAC Log File Website. https://www.oecd.org/skills/piaac/log-file/
OECD. (2017). PISA 2015 technical report. OECD Publishing. https://doi.org/10.1787/9789264255425-en
OECD. (2019). Beyond proficiency: Using log files to understand respondent behaviour in the survey of adult skills. OECD Publishing. https://doi.org/10.1787/0b1414ed-en
Ouzzani, M., Hammady, H., Fedorowicz, Z., & Elmagarmid, A. (2016). Rayyan – A web and mobile app for systematic reviews. Systematic Reviews, 5(4). https://doi.org/10.1186/s13643-016-0384-4
Pejic, A., & Molcer, P. S. (2016). Exploring data mining possibilities on computer based problem solving data. SISY 2016 – IEEE 14th International Symposium on Intelligent Systems and Informatics, Proceedings, 171–176. https://doi.org/10.1109/SISY.2016.7601491
Pokropek, A. (2016). Grade of membership response time model for detecting guessing behaviors. Journal of Educational and Behavioral Statistics, 41(3), 300–325. https://doi.org/10.3102/1076998616636618
Qiao, X., & Jiao, H. (2018). Data mining techniques in analyzing process data: A didactic. Frontiers in Psychology, 9, 2231. https://doi.org/10.3389/fpsyg.2018.02231
Ramalingam, D., & Adams, R. J. (2018). How can the use of data from computer-delievered assessments improve the measurement of twenty-first century skills? In Assessment and teaching of 21st century skills (pp. 225–238). Springer. https://doi.org/10.1007/978-3-319-65368-6
Reis Costa, D., & Leoncio, W. (2019). LOGAN: An R package for log file analysis in international large-scale assessments. R Package.
Ren, Y., Luo, F., Ren, P., Bai, D., & Li, X. (2019). Exploring multiple goals balancing in complex problem solving based on log data. Frontiers in Psychology, 10, 1975. https://doi.org/10.3389/fpsyg.2019.01975
Stone, J., & Zumbo, B. D. (2016). Validity as a pragmatist project: A global concern with local application. In V. Aryadoust & J. Fox (Eds.), Trends in language assessment research and practice (pp. 555–573). Cambridge Scholars.
Teig, N. (2019). Scientific inquiry in TIMSS and PISA 2015: Inquiry as an instructional approach and the assessment of inquiry as an instructional outcome in science. University of Oslo. http://urn.nb.no/URN:NBN:no-74775
Ulitzsch, E., von Davier, M., & Pohl, S. (2020a). A hierarchical latent response model for inferences about examinee engagement in terms of guessing and item-level non-response. British Journal of Mathematical and Statistical Psychology, 73(S1), 83–112. https://doi.org/10.1111/bmsp.12188
Ulitzsch, E., von Davier, M., & Pohl, S. (2020b). Using response times for joint modeling of response and omission behavior. Multivariate Behavioral Research, 55(3), 425–453. https://doi.org/10.1080/00273171.2019.1643699
Valdman, J. (2001). Log file analysis. In Department of Computer Science and Engineering (FAV UWB) (Technical Report No. DCSE/TR-2001-04) http://www.kiv.zcu.cz/vyzkum/publikace/technicke-zpravy/2001/tr-2001-04.pdf
van Damme, D. (2014). How closely is the distribution of skills related to countries’ overall level of social inequality and economic prosperity? OECD Education Working Papers, 105(105), 1–23.
Vista, A., Care, E., & Awwal, N. (2017). Visualising and examining sequential actions as behavioural paths that can be interpreted as markers of complex behaviours. Computers in Human Behavior, 76, 656–671. https://doi.org/10.1016/j.chb.2017.01.027
Vörös, Z., & Rouet, J. F. (2016). Laypersons’ digital problem solving: Relationships between strategy and performance in a large-scale international survey. Computers in Human Behavior, 64, 108–116. https://doi.org/10.1016/j.chb.2016.06.018
Xu, H., Fang, G., Chen, Y., Liu, J., & Ying, Z. (2018). Latent class analysis of recurrent events in problem-solving items. Applied Psychological Measurement, 42(6), 476–498. https://doi.org/10.1177/0146621617748325
Zhan, P., Jiao, H., & Liao, D. (2018). Cognitive diagnosis modelling incorporating item response times. British Journal of Mathematical and Statistical Psychology, 71(2), 262–286. https://doi.org/10.1111/bmsp.12114
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Section Editor information
Rights and permissions
Copyright information
© 2022 Springer Nature Switzerland AG
About this entry
Cite this entry
Reis Costa, D., Leoncio Netto, W. (2022). Process Data Analysis in ILSAs. In: Nilsen, T., Stancel-Piątak, A., Gustafsson, JE. (eds) International Handbook of Comparative Large-Scale Studies in Education. Springer International Handbooks of Education. Springer, Cham. https://doi.org/10.1007/978-3-030-38298-8_60-1
Download citation
DOI: https://doi.org/10.1007/978-3-030-38298-8_60-1
Received:
Accepted:
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-38298-8
Online ISBN: 978-3-030-38298-8
eBook Packages: Springer Reference EducationReference Module Humanities and Social SciencesReference Module Education