Abstract
The quality of students’ writing skills continues to concern educators. Because writing is essential to success in both college and career, poor writing can have lifelong consequences. Writing is now primarily done digitally, but students receive limited explicit instruction in digital writing. This lack of instruction means that students fail to take advantage of the affordances of digital tools. The writing process is shaped by the tools used, which makes digital writing, to an extent, different from writing with pen and paper. To better understand students’ digital writing skills, we take advantage of the information provided by computer-based assessments—keyboard and mouse activity data. We examine the relationship between students’ use of the keyboard and mouse during the assessment and students’ writing achievement. Our data comes from the first national computer-based writing assessment in the United States, the 2011 National Assessment of Educational Progress (NAEP) assessment. Using data from over 24,100 eighth-grade students, we found that the number of keypresses had a distinct and direct effect on writing achievement scores, controlling for word count. We also identified several different patterns of keyboard and mouse activity on the computer-based NAEP assessment.
Similar content being viewed by others
Notes
We note that NAEP reports plausible values to improve the ability to draw inferences about a population’s true proficiency level (Rogers et al. 2013). Our analysis is not intended to be representative of the population’s proficiency level, but instead to understand the relationship of this micro-level keyboard and mouse activity on an individual, human-scored writing test. We acknowledge that the writing tasks were varied across the sample.
References
Allen, N. L., & Donoghue, J. R. (1996). Applying the Mantel–Haenszel procedure to complex samples of items. Journal of Educational Measurement, 33(2), 231–251. https://doi.org/10.1111/j.1745-3984.1996.tb00491.x.
Allen, N. L., Donoghue, J. R., & Schoeps, T. L. (2001). The NAEP 1998 Technical Report. NCES 2001-509. Washington, DC: National Center for Education Statistics.
Almond, R., Deane, P., Quinlan, T., Wagner, M., & Sydorenko, T. (2012). A preliminary analysis of keystroke log data from a timed writing task. ETS Research Report Series, 2012(2), i–61.
Applebee, A. N. (2007). Issues in large-scale writing assessment perspectives from the National Assessment of Educational Progress. Journal of Writing Assessment, 3(2), 81–98. Retrieved from http://www.journalofwritingassessment.org/archives/3-2.2pdf.
Applebee, A. N., & Langer, J. A. (2011). A snapshot of writing instruction in middle schools and high schools. English Journal, 100(6), 14–27.
Bazerman, C., Applebee, A., Berninger, V., Brandt, D., Graham, S., Jeffery, J., et al. (Eds.). (2018). The lifespan development of writing. Urbana, IL: National Council of English.
Beaton, A., Rogers, A., Gonzalez, E., Hanly, M., Kolstad, A., Rust, K., et al. (2011). The NAEP Primer. Washington, DC: National Center for Educational Statistics.
Beaton, A. E., & Zwick, R. (1992). Chapter 1: Overview of the National Assessment of Educational Progress. Journal of Educational Statistics, 17(2), 95–109. https://doi.org/10.3102/10769986017002095.
Benjamini, Y., & Hochberg, Y. (1995). Controlling the false discovery rate: A practical and powerful approach to multiple testing. Journal of the Royal Statistical Society: Series B (Methodological), 57(1), 289–300.
Bergman, L. R., Magnusson, D., & El-Khouri, B. M. (2002). Studying individual development in an interindividual context. A person-oriented approach. Mahwah, New Jersey, London: Lawrence-Erlbaum Associates.
Chen, E., Niemi, D., Wang, J., Wang, H., & Mirocha, J. (2007). Examining the generalizability of direct writing assessment tasks. CSE technical report 718. Los Angeles, CA: National Center for Research on Evaluation, Standards, and Student Testing (CRESST). https://doi.org/10.1037/e643812011-001.
Chun, D., Smith, B., & Kern, R. (2016). Technology in language use, language teaching, and language learning. Modern Language Journal, 100, 64–80. https://doi.org/10.1111/modl.12302.
Cochran-Smith, M. (1991). Word processing and writing in elementary classrooms: A critical review of related literature. Review of Educational Research, 61(1), 107–155. https://doi.org/10.3102/00346543061001107.
Common Core State Standards Initiative (CCSSI). (2017). Language progressive skills. Retrieved June 07, 2017, from http://www.corestandards.org/ELA-Literacy/L/language-progressive-skills/.
Crossley, S. A., Weston, J., McLain Sullivan, S. T., & McNamara, D. S. (2011). The development of writing proficiency as a function of grade level: A linguistic analysis. Written Communication, 28(3), 282–311. https://doi.org/10.1177/0741088311410188.
Daiute, C. (1986). Physical and cognitive factors in revising: Insights from studies with computers. Research in the Teaching of English, 20(2), 141–159.
De La Paz, S., & Graham, S. (2002). Explicitly teaching strategies, skills, and knowledge: Writing instruction in middle school classrooms. Journal of Educational Psychology, 94(4), 687. https://doi.org/10.1037/0022-0663.94.4.687.
Deane, P. (2011). Writing assessment and cognition. ETS Research Report Series, 1, i–60. https://doi.org/10.1002/j.2333-8504.2011.tb02250.x.
DeVoss, D. N., Eidman-Aadahl, E., & Hicks, T. (2010). Because digital writing matters: Improving student writing in online and multimedia environments. Hoboken: Wiley.
Eccles, J., & Midgley, C. (1990). Changes in academic motivation and self-perception during early adolescence. In R. Montemayor (Ed.), Early adolescence as a time of transition (pp. 1–29). Beverly Hills, CA: Sage Publishing.
Engeström, Y. (1987). Learning by expanding: An activity-theoretical approach to developmental research. Helsinki: Orienta-Konsultit Oy.
Fang, Z., Schleppegrell, M. J., & Cox, B. E. (2006). Understanding the language demands of schooling: Nouns in academic registers. Journal of Literacy Research, 38(3), 247–273. https://doi.org/10.1207/s15548430jlr3803_1.
Goldfine, R. (2001). Making word processing more effective in the composition classroom. Teaching English in the Two-Year College, 28(3), 307–315.
Graham, S. (2018). A writer(s) within community model of writing. In C. Bazerman, A. Applebee, V. Berninger, D. Brandt, S. Graham, J. Jeffery, P. Matsuda, S. Murphy, D. Rowe, M. Schleppegrell, & K. Wilcox (Eds.), The lifespan development of writing (pp. 271–325). Urbana, IL: National Council of English. https://doi.org/10.1080/00461520.2018.1481406.
Graham, S., & Perin, D. (2007). A meta-analysis of writing instruction for adolescent students. Journal of Educational Psychology, 99(3), 445–476. https://doi.org/10.1037/0022-0663.99.3.445.
Greenleaf, C. (1994). Technological indeterminacy: The role of classroom writing practices and pedagogy in shaping student use of the computer. Written Communication, 11(1), 85–130. https://doi.org/10.1177/0741088394011001005.
Haertel, E. (1984). An application of latent class models to assessment data. Applied Psychological Measurement, 8(3), 333–346.
Halliday, M. A. K. (1993). Towards a language-based theory of learning. Lingquistics and Education, 5, 93–116. https://doi.org/10.1016/0898-5898(93)90026-7.
Horkay, N., Bennett, R. E., Allen, N., Kaplan, B., Yan, F. (2006). Does it matter if I take my writing test on computer? An empirical study of mode effects in NAEP. Journal of Technology, Learning, and Assessment, 5(2), 4–48.
Johnson, E. G. (1992). The design of the National Assessment of Educational Progress. Journal of Educational Measurement, 29(2), 95–110. https://doi.org/10.1111/j.1745-3984.1992.tb00369.x.
KewalRamani, A., Zhang, J., Wang, X., Rathbun, A., Corcoran, L., Diliberti, M., et al. (2018). Student access to digital learning resources outside of the classroom (NCES 2017-098). U.S. Department of Education. Washington, DC: National Center for Education Statistics. Retrieved April 6, 2018 from https://nces.ed.gov/pubsearch/pubsinfo.asp?pubid=2017098.
Kim, Y. S. G., Schatschneider, C., Wanzek, J., Gatlin, B., & Al Otaiba, S. (2017). Writing evaluation: rater and task effects on the reliability of writing scores for children in Grades 3 and 4. Reading and Writing, 30(6), 1287–1310. https://doi.org/10.1007/s11145-017-9724-6.
Leontyev, A. N. (1978). Activity, consciousness, and personality. Englewood Cliffs, NJ: Prentice-Hall.
Leontyev, A. N. (1981). Problems of the development of mind. Moscow: Progress.
Macarthur, C. A. (1999). Overcoming barriers to writing: Computer support for basic writing skills. Reading & Writing Quarterly, 15(2), 169–192. https://doi.org/10.1080/105735699278251.
MacArthur, C. A. (2009). Reflections on research on writing and technology for struggling writers. Learning Disabilities Research & Practice, 24(2), 93–103.
McCutchen, D. (1996). A capacity theory of writing: Working memory in composition. Educational Psychology Review, 8(3), 299–325. https://doi.org/10.1007/BF01464076.
McNamara, D. S., Crossley, S. A., & Roscoe, R. (2013). Natural language processing in an intelligent writing strategy tutoring system. Behavior Research Methods, 45(2), 499–515. https://doi.org/10.3758/s13428-012-0258-1.
Mo, Y., & Troia, G. (2017). Similarities and differences in constructs represented by U.S. States’ middle school writing tests and the 2007 National Assessment of Educational Progress writing assessment. Assessing Writing, 33(July 17), 48–67. https://doi.org/10.1016/j.asw.2017.06.001.
Morphy, P., & Graham, S. (2012). Word processing programs and weaker writers/readers: A meta-analysis of research findings. Reading and Writing, 25(3), 641–678. https://doi.org/10.1007/s11145-010-9292-5.
Murphy, S., & Smith, M. A. (2018). The faraway stick cannot kill the nearby snake. In C. Bazerman, A. Applebee, V. Berninger, D. Brandt, S. Graham, J. Jeffery, P. Matsuda, S. Murphy, D. Rowe, M. Schleppegrell, & K. Wilcox (Eds.), The lifespan development of writing (pp. 210–243). Urbana, IL: National Council of English.
National Assessment Governing Board (NAGB). (2010). Writing framework for the 2011 National Assessment of Educational Progress. Washington, DC: U.S. Department of Education.
National Center for Education Statistics (NCES). (2012). The Nation’s report card: Writing 2011. Washington, DC: Institute of Education Sciences, U.S. Department of Education.
National Center for Education Statistics (NCES). (2014a). Lessons learned from the 2012 grade 4 writing computer-based assessment study. Washington, DC: Institute of Education Sciences, U.S. Department of Education. Retrieved February 28, 2019 from, http://nces.ed.gov/nationsreportcard/writing/lessons/default.aspx.
National Center for Education Statistics (NCES). (2014b). NAEP 2012 pilot writing computer-based assessment study-grade 4: Technical memorandum: Institute of Education Sciences. Institute of Education Sciences, U.S. Department of Education: Washington, DC.
National Center for Education Statistics (NCES). (n.d.). NAEP technical documentation: Calculation and use of replicate weights. Washington, D.C.: Institute of Education Sciences, U.S. Department of Education. Retrieved April 19, 2018 from https://nces.ed.gov/nationsreportcard/tdw/weighting/2000_2001/2000state_repweights.aspx.
Olinghouse, N., Santangelo, T., & Wilson, J. (2012). Examining the validity of single-occasion, single-genre, holistically scored writing assessments. In E. V. Steendam, M. Tillema, G. Rijlaarsdam, & H. V. D. Bergh (Eds.), Measuring writing: Recent insights into theory, methodology and practices (pp. 55–82). New York: Guilford.
Porter, A., McMaken, J., Hwang, J., & Yang, R. (2011). Common core standards: The new US intended curriculum. Educational Researcher, 40(3), 103–116. https://doi.org/10.3102/0013189X11405038.
Purcell, K., Buchanan, J., & Friedrich, L. (2013a). The impact of digital tools on student writing and how writing is taught in schools. Washington, DC: Pew Research Center’s Interent & American Life Project. Retrieved February 28, 2019 from http://pewinternet.org/Reports/2013/Teachers-technology-and-writing.
Purcell, K., Heaps, A., Buchanan, J., & Friedrich, L. (2013b). How teachers are using technology at home and in their classrooms. Washington, DC: Pew Research Center’s Interent & American Life Project. Retrieved February 28, 2019 from http://pewinternet.org/Reports/2013/Teachers-and-technology.
Rogers, A., Stoeckel, J., & Sikali, E. (2013). National Assessment of Educational Progress (NAEP) 2011 writing assessment restricted-use data files data companion. Washington, DC: National Center for Education Statistics.
Russell, M., & Haney, W. (1997). Testing writing on computers. Education Policy Analysis Archives, 5(3), 1–20. https://doi.org/10.14507/epaa.v5n3.1997.
Russell, M., & Plati, T. (2002). Does it matter with what I write? Comparing performance on paper, computer and portable writing devices. Current Issues in Education, 5(4), 1–15.
Sandene, B., Horkay, N., Bennett, R. E., Allen, N., Braswell, J., Kaplan, B., et al. (2005). Online assessment in mathematics and writing: Reports from the NAEP Technology-Based Assessment Project, Research and Development Series. NCES 2005-457. Washington, DC: Institute of Education Sciences, U.S. Department of Education.
Tate, T., Warschauer, M., & Abedi, J. (2016). The effects of prior computer use on computer-based writing: The 2011 NAEP writing assessment. Computers & Education, 101, 115–131. https://doi.org/10.1016/j.compedu.2016.06.001.
Tate, T., Warschauer, M., & Kim, Y.-S. G. (2019). Learning to compose digitally: The effect of prior computer use and keyboard activity on NAEP writing. Reading & Writing. https://doi.org/10.1007/s11145-019-09940-z.
Vargha, A., Torma, B., & Bergman, L. R. (2015). ROPstat: A general statistical package useful for conducting person-oriented analyses. Journal for Person-Oriented Research, 1(1-2), 87–98. https://doi.org/10.17505/jpor.2015.09.
Vygotsky, L. S. (1962). Thought and language. Cambridge, MA: MIT Press. https://doi.org/10.1037/11193-000.
Vygotsky, L. S. (1978). Mind and society. Cambridge, MA: MIT Press.
Vygotsky, L. S. (1981). The genesis of higher mental functions. In J. V. Wertsch (Ed.), The concept of activity in Soviet psychology (pp. 144–188). Armonk, NY: M.E. Sharpe.
Wang, Q., & Pomerantz, E. M. (2009). The motivational landscape of early adolescence in the United States and China: A longitudinal investigation. Child Development, 80(4), 1272–1287. https://doi.org/10.1111/j.1467-8624.2009.01331.x.
Warschauer, M. (2006). Laptops and literacy: Learning in the wireless classroom. New York: Teachers College Press.
Way, W. D., Davis, L. L., & Strain-Seymour, E. (2008). The validity case for assessing direct writing by computer. A Pearson Assessments & Information White Paper.
Wenglinsky, H. (2005). Using technology wisely: The keys to success in schools. New York, NY: Teachers College Press.
Wertsch, J. V. (1979). From social interaction to higher psychological processes. A clarification and application of Vygotsky’s theory. Human Development, 22(1), 1–22.
Wertsch, J. V. (1991). Voices of the mind: A sociocultural approach to mediated action. Cambridge, MA: Harvard University Press.
White, S., Kim, Y. Y., Chen, J., & Liu, F. (2015). Performance of fourth-grade students in the 2012 NAEP computer-based writing pilot assessment: Scores, text length, and use of editing tools. NCES 2015-119. Washington, DC: Institute of Education Sciences, US Department of Education.
Wigfield, A., & Eccles, J. S. (2000). Expectancy–value theory of achievement motivation. Contemporary Educational Psychology, 25(1), 68–81. https://doi.org/10.1006/ceps.1999.1015.
Williamson, G. L., Fitzgerald, J., & Stenner, A. J. (2013). The Common Core State Standards’ quantitative text complexity trajectory: Figuring out how much complexity is enough. Educational Researcher, 42(2), 59–69. https://doi.org/10.3102/0013189X12466695.
Zheng, B., Lawrence, J., Warschauer, M., & Lin, C. H. (2015). Middle school students’ writing and feedback in a cloud-based classroom environment. Technology, Knowledge and Learning, 20(2), 201–229.
Zwick, R. (1987). Assessing the dimensionality of NAEP reading data. Journal of Educational Measurement, 24(4), 293–308.
Acknowledgements
This work has been supported by funding from the Spencer Foundation, Grant No. 201500153.
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Tate, T.P., Warschauer, M. Keypresses and Mouse Clicks: Analysis of the First National Computer-Based Writing Assessment. Tech Know Learn 24, 523–543 (2019). https://doi.org/10.1007/s10758-019-09412-x
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10758-019-09412-x