Identifying and Comparing Writing Process Patterns Using Keystroke Logs
There is a growing literature on the use of process data in digitally delivered assessments. In this study, we analyzed students’ essay writing processes using keystroke logs. Using four basic writing performance indicators, writers were grouped into four clusters, representing groups from fluent to struggling. The clusters differed significantly on the mean essay score, mean total time spent on task, and mean total number of words in the final submissions. Two of the four clusters were significantly different on the aforementioned three dimensions but not on typing skill. The higher scoring group even showed signs of less fluency than the lower scoring group, suggesting that task engagement and writing efforts might play an important role in generating better quality text. The four identified clusters further showed distinct sequential patterns over the course of the writing session on three process characteristics and, as well, differed on their editing behaviors during the writing process.
KeywordsWriting process Keystroke logs CBAL Sequential pattern Editing behavior
- Alves, R. A., Castro, S. L., de Sousa, L., & Stromqvist, S. (2007). Influence of typing skill on pauseexecution cycles in written composition. In M. Torrance, L. van Waes, & D. Galbraith (Eds.), Writing and Cognition: Research and Applications (pp. 55–65). Amsterdam: Elsevier.Google Scholar
- Cleveland, William S. (1979). Robust locally weighted regression and smoothing scatterplots. Journal of the American Statistical Association., 74, 829–836. https://pdfs.semanticscholar.org/414e/5d1f5a75e2327d99b5bbb93f2e4e241c5acc.pdf.MathSciNetCrossRefGoogle Scholar
- Cooley, W. W., & Lohnes, P. R. (1971). Multivariate data analysis. John Wiley and Sons.Google Scholar
- Deane, P., Feng, G., Zhang, M., Hao, J., Bergner, Y., Flor, M., Wagner, M., Lederer. N.: Generating scores and feedback for writing assessment and instruction using electronic process logs. US Patent and Trademark Office. Application No. 14/937,164 (2016).Google Scholar
- Ercikan, K., & Pellegrino, J. W. (2017). Validation of score meaning for the next generation of assessments: The use of response processes. Taylor & Francis.Google Scholar
- Guo, H., Deane, P., van Rijn, P., Zhang, M., & Bennett, R. (2018). Exploring the heavy-tailed key-stroke data in writing processes. Journal of Educational Measurement, 194–216,Google Scholar
- Murtagh, F., Legendre, P.: Ward’s hierarchical clustering method: Clustering criterion and agglomerative algorithm. Accessed in October 2018: http://arxiv.org/abs/1111.6285.pdf (2011)
- Rencher, A. C. (1992). Interpretation of canonical discriminant functions, canonical variates, and principal components. The American Statistician, 46, 217–225.Google Scholar
- Sinharay, S., Zhang, M., Deane, P.: Application of data mining for predicting essay scores from writing process and product features. Applied Measurement in Education (2019). https://www.tandfonline.com/doi/full/10.1080/08957347.2019.1577245.
- Stevenson, M., Schoonen, R., & de Glopper, K. (2006). Revising in two languages: A multi-dimensional comparison of online writing revisions in L1 and FL. Journal of Second Language Writing, 201–233,Google Scholar
- van Rijn, P., Yan-Koo, Y.: Statistical results from the 2013 CBAL English Language Arts multistate study: Parallel forms for argumentative writing. RM-16-15. Princeton, NJ: Educational Testing Service (2016).Google Scholar
- van Rijn, P., Chen, J., Yan-Koo, Y.: Statistical results from the 2013 CBALTM English Language Arts multistate study: Parallel forms for policy recommendation writing. RR-16-01. Princeton, NJ: Educational Testing Service (2016).Google Scholar
- Zhang, M., Bennett, R., Deane, P., & van Rijn, P. (2019). Are there gender differences in how students write their essays? An analysis of writing processes. Educational Measurement: Issues and Practice, Online First.Google Scholar
- Zhang, M., Feng, G., Deane, P., H, Guo.: Investigating an approach to evaluating keyboarding fluency. To be submitted for publication (2018).Google Scholar
- Zhang, M., Hao, J., Li, C., & Deane, P. (2016). Classification of writing patterns using keystroke logs. In L. A. van der Ark, D. M. Bolt, W.-C. Wang, J. A. Douglas, & M. Wiberg (Eds.), Quantitative Psychology Research. New York: Springer.Google Scholar
- Zhang, M., Hao, J., Li, C., Deane, P.: Defining personalized writing burst measures of translation using keystroke logs. In proceedings of the 2018 Educational Data Mining Conference, 549 - 552 (2018).Google Scholar
- Zhu, M., Zhang, M., Deane, P.: Analysis of keystroke sequences in writing logs. RR-xx-xx. Princeton, NJ: Educational Testing Service (2019). https://onlinelibrary.wiley.com/doi/10.1002/ets2.12247.