Skip to main content

Keypresses and Mouse Clicks: Analysis of the First National Computer-Based Writing Assessment

Technology, Knowledge and Learning volume 24, pages 523–543 (2019)Cite this article

Abstract

The quality of students’ writing skills continues to concern educators. Because writing is essential to success in both college and career, poor writing can have lifelong consequences. Writing is now primarily done digitally, but students receive limited explicit instruction in digital writing. This lack of instruction means that students fail to take advantage of the affordances of digital tools. The writing process is shaped by the tools used, which makes digital writing, to an extent, different from writing with pen and paper. To better understand students’ digital writing skills, we take advantage of the information provided by computer-based assessments—keyboard and mouse activity data. We examine the relationship between students’ use of the keyboard and mouse during the assessment and students’ writing achievement. Our data comes from the first national computer-based writing assessment in the United States, the 2011 National Assessment of Educational Progress (NAEP) assessment. Using data from over 24,100 eighth-grade students, we found that the number of keypresses had a distinct and direct effect on writing achievement scores, controlling for word count. We also identified several different patterns of keyboard and mouse activity on the computer-based NAEP assessment.

This is a preview of subscription content, access via your institution.

Access options

Buy single article

Instant access to the full article PDF.

USD 39.95

Price includes VAT (USA)
Tax calculation will be finalised during checkout.

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Notes

  1. We note that NAEP reports plausible values to improve the ability to draw inferences about a population’s true proficiency level (Rogers et al. 2013). Our analysis is not intended to be representative of the population’s proficiency level, but instead to understand the relationship of this micro-level keyboard and mouse activity on an individual, human-scored writing test. We acknowledge that the writing tasks were varied across the sample.

References

  • Allen, N. L., & Donoghue, J. R. (1996). Applying the Mantel–Haenszel procedure to complex samples of items. Journal of Educational Measurement, 33(2), 231–251. https://doi.org/10.1111/j.1745-3984.1996.tb00491.x.

    Article  Google Scholar 

  • Allen, N. L., Donoghue, J. R., & Schoeps, T. L. (2001). The NAEP 1998 Technical Report. NCES 2001-509. Washington, DC: National Center for Education Statistics.

  • Almond, R., Deane, P., Quinlan, T., Wagner, M., & Sydorenko, T. (2012). A preliminary analysis of keystroke log data from a timed writing task. ETS Research Report Series, 2012(2), i–61.

  • Applebee, A. N. (2007). Issues in large-scale writing assessment perspectives from the National Assessment of Educational Progress. Journal of Writing Assessment, 3(2), 81–98. Retrieved from http://www.journalofwritingassessment.org/archives/3-2.2pdf.

  • Applebee, A. N., & Langer, J. A. (2011). A snapshot of writing instruction in middle schools and high schools. English Journal, 100(6), 14–27.

    Google Scholar 

  • Bazerman, C., Applebee, A., Berninger, V., Brandt, D., Graham, S., Jeffery, J., et al. (Eds.). (2018). The lifespan development of writing. Urbana, IL: National Council of English.

    Google Scholar 

  • Beaton, A., Rogers, A., Gonzalez, E., Hanly, M., Kolstad, A., Rust, K., et al. (2011). The NAEP Primer. Washington, DC: National Center for Educational Statistics.

    Google Scholar 

  • Beaton, A. E., & Zwick, R. (1992). Chapter 1: Overview of the National Assessment of Educational Progress. Journal of Educational Statistics, 17(2), 95–109. https://doi.org/10.3102/10769986017002095.

    Article  Google Scholar 

  • Benjamini, Y., & Hochberg, Y. (1995). Controlling the false discovery rate: A practical and powerful approach to multiple testing. Journal of the Royal Statistical Society: Series B (Methodological), 57(1), 289–300.

    Google Scholar 

  • Bergman, L. R., Magnusson, D., & El-Khouri, B. M. (2002). Studying individual development in an interindividual context. A person-oriented approach. Mahwah, New Jersey, London: Lawrence-Erlbaum Associates.

    Google Scholar 

  • Chen, E., Niemi, D., Wang, J., Wang, H., & Mirocha, J. (2007). Examining the generalizability of direct writing assessment tasks. CSE technical report 718. Los Angeles, CA: National Center for Research on Evaluation, Standards, and Student Testing (CRESST). https://doi.org/10.1037/e643812011-001.

  • Chun, D., Smith, B., & Kern, R. (2016). Technology in language use, language teaching, and language learning. Modern Language Journal, 100, 64–80. https://doi.org/10.1111/modl.12302.

    Article  Google Scholar 

  • Cochran-Smith, M. (1991). Word processing and writing in elementary classrooms: A critical review of related literature. Review of Educational Research, 61(1), 107–155. https://doi.org/10.3102/00346543061001107.

    Article  Google Scholar 

  • Common Core State Standards Initiative (CCSSI). (2017). Language progressive skills. Retrieved June 07, 2017, from http://www.corestandards.org/ELA-Literacy/L/language-progressive-skills/.

  • Crossley, S. A., Weston, J., McLain Sullivan, S. T., & McNamara, D. S. (2011). The development of writing proficiency as a function of grade level: A linguistic analysis. Written Communication, 28(3), 282–311. https://doi.org/10.1177/0741088311410188.

    Article  Google Scholar 

  • Daiute, C. (1986). Physical and cognitive factors in revising: Insights from studies with computers. Research in the Teaching of English, 20(2), 141–159.

    Google Scholar 

  • De La Paz, S., & Graham, S. (2002). Explicitly teaching strategies, skills, and knowledge: Writing instruction in middle school classrooms. Journal of Educational Psychology, 94(4), 687. https://doi.org/10.1037/0022-0663.94.4.687.

    Article  Google Scholar 

  • Deane, P. (2011). Writing assessment and cognition. ETS Research Report Series, 1, i–60. https://doi.org/10.1002/j.2333-8504.2011.tb02250.x.

    Article  Google Scholar 

  • DeVoss, D. N., Eidman-Aadahl, E., & Hicks, T. (2010). Because digital writing matters: Improving student writing in online and multimedia environments. Hoboken: Wiley.

    Google Scholar 

  • Eccles, J., & Midgley, C. (1990). Changes in academic motivation and self-perception during early adolescence. In R. Montemayor (Ed.), Early adolescence as a time of transition (pp. 1–29). Beverly Hills, CA: Sage Publishing.

    Google Scholar 

  • Engeström, Y. (1987). Learning by expanding: An activity-theoretical approach to developmental research. Helsinki: Orienta-Konsultit Oy.

    Google Scholar 

  • Fang, Z., Schleppegrell, M. J., & Cox, B. E. (2006). Understanding the language demands of schooling: Nouns in academic registers. Journal of Literacy Research, 38(3), 247–273. https://doi.org/10.1207/s15548430jlr3803_1.

    Article  Google Scholar 

  • Goldfine, R. (2001). Making word processing more effective in the composition classroom. Teaching English in the Two-Year College, 28(3), 307–315.

    Google Scholar 

  • Graham, S. (2018). A writer(s) within community model of writing. In C. Bazerman, A. Applebee, V. Berninger, D. Brandt, S. Graham, J. Jeffery, P. Matsuda, S. Murphy, D. Rowe, M. Schleppegrell, & K. Wilcox (Eds.), The lifespan development of writing (pp. 271–325). Urbana, IL: National Council of English. https://doi.org/10.1080/00461520.2018.1481406.

    Chapter  Google Scholar 

  • Graham, S., & Perin, D. (2007). A meta-analysis of writing instruction for adolescent students. Journal of Educational Psychology, 99(3), 445–476. https://doi.org/10.1037/0022-0663.99.3.445.

    Article  Google Scholar 

  • Greenleaf, C. (1994). Technological indeterminacy: The role of classroom writing practices and pedagogy in shaping student use of the computer. Written Communication, 11(1), 85–130. https://doi.org/10.1177/0741088394011001005.

    Article  Google Scholar 

  • Haertel, E. (1984). An application of latent class models to assessment data. Applied Psychological Measurement, 8(3), 333–346.

    Article  Google Scholar 

  • Halliday, M. A. K. (1993). Towards a language-based theory of learning. Lingquistics and Education, 5, 93–116. https://doi.org/10.1016/0898-5898(93)90026-7.

    Article  Google Scholar 

  • Horkay, N., Bennett, R. E., Allen, N., Kaplan, B., Yan, F. (2006). Does it matter if I take my writing test on computer? An empirical study of mode effects in NAEP. Journal of Technology, Learning, and Assessment, 5(2), 4–48.

    Google Scholar 

  • Johnson, E. G. (1992). The design of the National Assessment of Educational Progress. Journal of Educational Measurement, 29(2), 95–110. https://doi.org/10.1111/j.1745-3984.1992.tb00369.x.

    Article  Google Scholar 

  • KewalRamani, A., Zhang, J., Wang, X., Rathbun, A., Corcoran, L., Diliberti, M., et al. (2018). Student access to digital learning resources outside of the classroom (NCES 2017-098). U.S. Department of Education. Washington, DC: National Center for Education Statistics. Retrieved April 6, 2018 from https://nces.ed.gov/pubsearch/pubsinfo.asp?pubid=2017098.

  • Kim, Y. S. G., Schatschneider, C., Wanzek, J., Gatlin, B., & Al Otaiba, S. (2017). Writing evaluation: rater and task effects on the reliability of writing scores for children in Grades 3 and 4. Reading and Writing, 30(6), 1287–1310. https://doi.org/10.1007/s11145-017-9724-6.

    Article  Google Scholar 

  • Leontyev, A. N. (1978). Activity, consciousness, and personality. Englewood Cliffs, NJ: Prentice-Hall.

    Google Scholar 

  • Leontyev, A. N. (1981). Problems of the development of mind. Moscow: Progress.

    Google Scholar 

  • Macarthur, C. A. (1999). Overcoming barriers to writing: Computer support for basic writing skills. Reading & Writing Quarterly, 15(2), 169–192. https://doi.org/10.1080/105735699278251.

    Article  Google Scholar 

  • MacArthur, C. A. (2009). Reflections on research on writing and technology for struggling writers. Learning Disabilities Research & Practice, 24(2), 93–103.

    Article  Google Scholar 

  • McCutchen, D. (1996). A capacity theory of writing: Working memory in composition. Educational Psychology Review, 8(3), 299–325. https://doi.org/10.1007/BF01464076.

    Article  Google Scholar 

  • McNamara, D. S., Crossley, S. A., & Roscoe, R. (2013). Natural language processing in an intelligent writing strategy tutoring system. Behavior Research Methods, 45(2), 499–515. https://doi.org/10.3758/s13428-012-0258-1.

    Article  Google Scholar 

  • Mo, Y., & Troia, G. (2017). Similarities and differences in constructs represented by U.S. States’ middle school writing tests and the 2007 National Assessment of Educational Progress writing assessment. Assessing Writing, 33(July 17), 48–67. https://doi.org/10.1016/j.asw.2017.06.001.

    Article  Google Scholar 

  • Morphy, P., & Graham, S. (2012). Word processing programs and weaker writers/readers: A meta-analysis of research findings. Reading and Writing, 25(3), 641–678. https://doi.org/10.1007/s11145-010-9292-5.

    Article  Google Scholar 

  • Murphy, S., & Smith, M. A. (2018). The faraway stick cannot kill the nearby snake. In C. Bazerman, A. Applebee, V. Berninger, D. Brandt, S. Graham, J. Jeffery, P. Matsuda, S. Murphy, D. Rowe, M. Schleppegrell, & K. Wilcox (Eds.), The lifespan development of writing (pp. 210–243). Urbana, IL: National Council of English.

    Google Scholar 

  • National Assessment Governing Board (NAGB). (2010). Writing framework for the 2011 National Assessment of Educational Progress. Washington, DC: U.S. Department of Education.

    Google Scholar 

  • National Center for Education Statistics (NCES). (2012). The Nation’s report card: Writing 2011. Washington, DC: Institute of Education Sciences, U.S. Department of Education.

    Google Scholar 

  • National Center for Education Statistics (NCES). (2014a). Lessons learned from the 2012 grade 4 writing computer-based assessment study. Washington, DC: Institute of Education Sciences, U.S. Department of Education. Retrieved February 28, 2019 from, http://nces.ed.gov/nationsreportcard/writing/lessons/default.aspx.

  • National Center for Education Statistics (NCES). (2014b). NAEP 2012 pilot writing computer-based assessment study-grade 4: Technical memorandum: Institute of Education Sciences. Institute of Education Sciences, U.S. Department of Education: Washington, DC.

    Google Scholar 

  • National Center for Education Statistics (NCES). (n.d.). NAEP technical documentation: Calculation and use of replicate weights. Washington, D.C.: Institute of Education Sciences, U.S. Department of Education. Retrieved April 19, 2018 from https://nces.ed.gov/nationsreportcard/tdw/weighting/2000_2001/2000state_repweights.aspx.

  • Olinghouse, N., Santangelo, T., & Wilson, J. (2012). Examining the validity of single-occasion, single-genre, holistically scored writing assessments. In E. V. Steendam, M. Tillema, G. Rijlaarsdam, & H. V. D. Bergh (Eds.), Measuring writing: Recent insights into theory, methodology and practices (pp. 55–82). New York: Guilford.

    Google Scholar 

  • Porter, A., McMaken, J., Hwang, J., & Yang, R. (2011). Common core standards: The new US intended curriculum. Educational Researcher, 40(3), 103–116. https://doi.org/10.3102/0013189X11405038.

    Article  Google Scholar 

  • Purcell, K., Buchanan, J., & Friedrich, L. (2013a). The impact of digital tools on student writing and how writing is taught in schools. Washington, DC: Pew Research Center’s Interent & American Life Project. Retrieved February 28, 2019 from http://pewinternet.org/Reports/2013/Teachers-technology-and-writing.

  • Purcell, K., Heaps, A., Buchanan, J., & Friedrich, L. (2013b). How teachers are using technology at home and in their classrooms. Washington, DC: Pew Research Center’s Interent & American Life Project. Retrieved February 28, 2019 from http://pewinternet.org/Reports/2013/Teachers-and-technology.

  • Rogers, A., Stoeckel, J., & Sikali, E. (2013). National Assessment of Educational Progress (NAEP) 2011 writing assessment restricted-use data files data companion. Washington, DC: National Center for Education Statistics.

    Google Scholar 

  • Russell, M., & Haney, W. (1997). Testing writing on computers. Education Policy Analysis Archives, 5(3), 1–20. https://doi.org/10.14507/epaa.v5n3.1997.

    Article  Google Scholar 

  • Russell, M., & Plati, T. (2002). Does it matter with what I write? Comparing performance on paper, computer and portable writing devices. Current Issues in Education, 5(4), 1–15.

    Google Scholar 

  • Sandene, B., Horkay, N., Bennett, R. E., Allen, N., Braswell, J., Kaplan, B., et al. (2005). Online assessment in mathematics and writing: Reports from the NAEP Technology-Based Assessment Project, Research and Development Series. NCES 2005-457. Washington, DC: Institute of Education Sciences, U.S. Department of Education.

  • Tate, T., Warschauer, M., & Abedi, J. (2016). The effects of prior computer use on computer-based writing: The 2011 NAEP writing assessment. Computers & Education, 101, 115–131. https://doi.org/10.1016/j.compedu.2016.06.001.

    Article  Google Scholar 

  • Tate, T., Warschauer, M., & Kim, Y.-S. G. (2019). Learning to compose digitally: The effect of prior computer use and keyboard activity on NAEP writing. Reading & Writing. https://doi.org/10.1007/s11145-019-09940-z.

    Article  Google Scholar 

  • Vargha, A., Torma, B., & Bergman, L. R. (2015). ROPstat: A general statistical package useful for conducting person-oriented analyses. Journal for Person-Oriented Research, 1(1-2), 87–98. https://doi.org/10.17505/jpor.2015.09.

    Article  Google Scholar 

  • Vygotsky, L. S. (1962). Thought and language. Cambridge, MA: MIT Press. https://doi.org/10.1037/11193-000.

    Book  Google Scholar 

  • Vygotsky, L. S. (1978). Mind and society. Cambridge, MA: MIT Press.

    Google Scholar 

  • Vygotsky, L. S. (1981). The genesis of higher mental functions. In J. V. Wertsch (Ed.), The concept of activity in Soviet psychology (pp. 144–188). Armonk, NY: M.E. Sharpe.

    Google Scholar 

  • Wang, Q., & Pomerantz, E. M. (2009). The motivational landscape of early adolescence in the United States and China: A longitudinal investigation. Child Development, 80(4), 1272–1287. https://doi.org/10.1111/j.1467-8624.2009.01331.x.

    Article  Google Scholar 

  • Warschauer, M. (2006). Laptops and literacy: Learning in the wireless classroom. New York: Teachers College Press.

    Google Scholar 

  • Way, W. D., Davis, L. L., & Strain-Seymour, E. (2008). The validity case for assessing direct writing by computer. A Pearson Assessments & Information White Paper.

  • Wenglinsky, H. (2005). Using technology wisely: The keys to success in schools. New York, NY: Teachers College Press.

    Google Scholar 

  • Wertsch, J. V. (1979). From social interaction to higher psychological processes. A clarification and application of Vygotsky’s theory. Human Development, 22(1), 1–22.

    Article  Google Scholar 

  • Wertsch, J. V. (1991). Voices of the mind: A sociocultural approach to mediated action. Cambridge, MA: Harvard University Press.

    Google Scholar 

  • White, S., Kim, Y. Y., Chen, J., & Liu, F. (2015). Performance of fourth-grade students in the 2012 NAEP computer-based writing pilot assessment: Scores, text length, and use of editing tools. NCES 2015-119. Washington, DC: Institute of Education Sciences, US Department of Education.

  • Wigfield, A., & Eccles, J. S. (2000). Expectancy–value theory of achievement motivation. Contemporary Educational Psychology, 25(1), 68–81. https://doi.org/10.1006/ceps.1999.1015.

    Article  Google Scholar 

  • Williamson, G. L., Fitzgerald, J., & Stenner, A. J. (2013). The Common Core State Standards’ quantitative text complexity trajectory: Figuring out how much complexity is enough. Educational Researcher, 42(2), 59–69. https://doi.org/10.3102/0013189X12466695.

    Article  Google Scholar 

  • Zheng, B., Lawrence, J., Warschauer, M., & Lin, C. H. (2015). Middle school students’ writing and feedback in a cloud-based classroom environment. Technology, Knowledge and Learning, 20(2), 201–229.

    Article  Google Scholar 

  • Zwick, R. (1987). Assessing the dimensionality of NAEP reading data. Journal of Educational Measurement, 24(4), 293–308.

    Article  Google Scholar 

Download references

Acknowledgements

This work has been supported by funding from the Spencer Foundation, Grant No. 201500153.

Author information

Authors and Affiliations

  1. School of Education, University of California, 3200 Education, Irvine, CA, 92697-5500, USA

    Tamara P. Tate & Mark Warschauer

Authors
  1. Tamara P. Tate
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mark Warschauer
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Tamara P. Tate.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Tate, T.P., Warschauer, M. Keypresses and Mouse Clicks: Analysis of the First National Computer-Based Writing Assessment. Tech Know Learn 24, 523–543 (2019). https://doi.org/10.1007/s10758-019-09412-x

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10758-019-09412-x

Keywords

  • NAEP
  • Writing
  • Secondary students
  • Digital writing
  • Log data
  • Clickstream data
  • Trace data