Keypresses and Mouse Clicks: Analysis of the First National Computer-Based Writing Assessment

Abstract

The quality of students’ writing skills continues to concern educators. Because writing is essential to success in both college and career, poor writing can have lifelong consequences. Writing is now primarily done digitally, but students receive limited explicit instruction in digital writing. This lack of instruction means that students fail to take advantage of the affordances of digital tools. The writing process is shaped by the tools used, which makes digital writing, to an extent, different from writing with pen and paper. To better understand students’ digital writing skills, we take advantage of the information provided by computer-based assessments—keyboard and mouse activity data. We examine the relationship between students’ use of the keyboard and mouse during the assessment and students’ writing achievement. Our data comes from the first national computer-based writing assessment in the United States, the 2011 National Assessment of Educational Progress (NAEP) assessment. Using data from over 24,100 eighth-grade students, we found that the number of keypresses had a distinct and direct effect on writing achievement scores, controlling for word count. We also identified several different patterns of keyboard and mouse activity on the computer-based NAEP assessment.

This is a preview of subscription content, log in to check access.

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Notes

  1. 1.

    We note that NAEP reports plausible values to improve the ability to draw inferences about a population’s true proficiency level (Rogers et al. 2013). Our analysis is not intended to be representative of the population’s proficiency level, but instead to understand the relationship of this micro-level keyboard and mouse activity on an individual, human-scored writing test. We acknowledge that the writing tasks were varied across the sample.

References

  1. Allen, N. L., & Donoghue, J. R. (1996). Applying the Mantel–Haenszel procedure to complex samples of items. Journal of Educational Measurement, 33(2), 231–251. https://doi.org/10.1111/j.1745-3984.1996.tb00491.x.

    Article  Google Scholar 

  2. Allen, N. L., Donoghue, J. R., & Schoeps, T. L. (2001). The NAEP 1998 Technical Report. NCES 2001-509. Washington, DC: National Center for Education Statistics.

  3. Almond, R., Deane, P., Quinlan, T., Wagner, M., & Sydorenko, T. (2012). A preliminary analysis of keystroke log data from a timed writing task. ETS Research Report Series, 2012(2), i–61.

  4. Applebee, A. N. (2007). Issues in large-scale writing assessment perspectives from the National Assessment of Educational Progress. Journal of Writing Assessment, 3(2), 81–98. Retrieved from http://www.journalofwritingassessment.org/archives/3-2.2pdf.

  5. Applebee, A. N., & Langer, J. A. (2011). A snapshot of writing instruction in middle schools and high schools. English Journal, 100(6), 14–27.

    Google Scholar 

  6. Bazerman, C., Applebee, A., Berninger, V., Brandt, D., Graham, S., Jeffery, J., et al. (Eds.). (2018). The lifespan development of writing. Urbana, IL: National Council of English.

    Google Scholar 

  7. Beaton, A., Rogers, A., Gonzalez, E., Hanly, M., Kolstad, A., Rust, K., et al. (2011). The NAEP Primer. Washington, DC: National Center for Educational Statistics.

    Google Scholar 

  8. Beaton, A. E., & Zwick, R. (1992). Chapter 1: Overview of the National Assessment of Educational Progress. Journal of Educational Statistics, 17(2), 95–109. https://doi.org/10.3102/10769986017002095.

    Article  Google Scholar 

  9. Benjamini, Y., & Hochberg, Y. (1995). Controlling the false discovery rate: A practical and powerful approach to multiple testing. Journal of the Royal Statistical Society: Series B (Methodological), 57(1), 289–300.

    Google Scholar 

  10. Bergman, L. R., Magnusson, D., & El-Khouri, B. M. (2002). Studying individual development in an interindividual context. A person-oriented approach. Mahwah, New Jersey, London: Lawrence-Erlbaum Associates.

    Google Scholar 

  11. Chen, E., Niemi, D., Wang, J., Wang, H., & Mirocha, J. (2007). Examining the generalizability of direct writing assessment tasks. CSE technical report 718. Los Angeles, CA: National Center for Research on Evaluation, Standards, and Student Testing (CRESST). https://doi.org/10.1037/e643812011-001.

  12. Chun, D., Smith, B., & Kern, R. (2016). Technology in language use, language teaching, and language learning. Modern Language Journal, 100, 64–80. https://doi.org/10.1111/modl.12302.

    Article  Google Scholar 

  13. Cochran-Smith, M. (1991). Word processing and writing in elementary classrooms: A critical review of related literature. Review of Educational Research, 61(1), 107–155. https://doi.org/10.3102/00346543061001107.

    Article  Google Scholar 

  14. Common Core State Standards Initiative (CCSSI). (2017). Language progressive skills. Retrieved June 07, 2017, from http://www.corestandards.org/ELA-Literacy/L/language-progressive-skills/.

  15. Crossley, S. A., Weston, J., McLain Sullivan, S. T., & McNamara, D. S. (2011). The development of writing proficiency as a function of grade level: A linguistic analysis. Written Communication, 28(3), 282–311. https://doi.org/10.1177/0741088311410188.

    Article  Google Scholar 

  16. Daiute, C. (1986). Physical and cognitive factors in revising: Insights from studies with computers. Research in the Teaching of English, 20(2), 141–159.

    Google Scholar 

  17. De La Paz, S., & Graham, S. (2002). Explicitly teaching strategies, skills, and knowledge: Writing instruction in middle school classrooms. Journal of Educational Psychology, 94(4), 687. https://doi.org/10.1037/0022-0663.94.4.687.

    Article  Google Scholar 

  18. Deane, P. (2011). Writing assessment and cognition. ETS Research Report Series, 1, i–60. https://doi.org/10.1002/j.2333-8504.2011.tb02250.x.

    Article  Google Scholar 

  19. DeVoss, D. N., Eidman-Aadahl, E., & Hicks, T. (2010). Because digital writing matters: Improving student writing in online and multimedia environments. Hoboken: Wiley.

    Google Scholar 

  20. Eccles, J., & Midgley, C. (1990). Changes in academic motivation and self-perception during early adolescence. In R. Montemayor (Ed.), Early adolescence as a time of transition (pp. 1–29). Beverly Hills, CA: Sage Publishing.

    Google Scholar 

  21. Engeström, Y. (1987). Learning by expanding: An activity-theoretical approach to developmental research. Helsinki: Orienta-Konsultit Oy.

    Google Scholar 

  22. Fang, Z., Schleppegrell, M. J., & Cox, B. E. (2006). Understanding the language demands of schooling: Nouns in academic registers. Journal of Literacy Research, 38(3), 247–273. https://doi.org/10.1207/s15548430jlr3803_1.

    Article  Google Scholar 

  23. Goldfine, R. (2001). Making word processing more effective in the composition classroom. Teaching English in the Two-Year College, 28(3), 307–315.

    Google Scholar 

  24. Graham, S. (2018). A writer(s) within community model of writing. In C. Bazerman, A. Applebee, V. Berninger, D. Brandt, S. Graham, J. Jeffery, P. Matsuda, S. Murphy, D. Rowe, M. Schleppegrell, & K. Wilcox (Eds.), The lifespan development of writing (pp. 271–325). Urbana, IL: National Council of English. https://doi.org/10.1080/00461520.2018.1481406.

    Google Scholar 

  25. Graham, S., & Perin, D. (2007). A meta-analysis of writing instruction for adolescent students. Journal of Educational Psychology, 99(3), 445–476. https://doi.org/10.1037/0022-0663.99.3.445.

    Article  Google Scholar 

  26. Greenleaf, C. (1994). Technological indeterminacy: The role of classroom writing practices and pedagogy in shaping student use of the computer. Written Communication, 11(1), 85–130. https://doi.org/10.1177/0741088394011001005.

    Article  Google Scholar 

  27. Haertel, E. (1984). An application of latent class models to assessment data. Applied Psychological Measurement, 8(3), 333–346.

    Article  Google Scholar 

  28. Halliday, M. A. K. (1993). Towards a language-based theory of learning. Lingquistics and Education, 5, 93–116. https://doi.org/10.1016/0898-5898(93)90026-7.

    Article  Google Scholar 

  29. Horkay, N., Bennett, R. E., Allen, N., Kaplan, B., Yan, F. (2006). Does it matter if I take my writing test on computer? An empirical study of mode effects in NAEP. Journal of Technology, Learning, and Assessment, 5(2), 4–48.

    Google Scholar 

  30. Johnson, E. G. (1992). The design of the National Assessment of Educational Progress. Journal of Educational Measurement, 29(2), 95–110. https://doi.org/10.1111/j.1745-3984.1992.tb00369.x.

    Article  Google Scholar 

  31. KewalRamani, A., Zhang, J., Wang, X., Rathbun, A., Corcoran, L., Diliberti, M., et al. (2018). Student access to digital learning resources outside of the classroom (NCES 2017-098). U.S. Department of Education. Washington, DC: National Center for Education Statistics. Retrieved April 6, 2018 from https://nces.ed.gov/pubsearch/pubsinfo.asp?pubid=2017098.

  32. Kim, Y. S. G., Schatschneider, C., Wanzek, J., Gatlin, B., & Al Otaiba, S. (2017). Writing evaluation: rater and task effects on the reliability of writing scores for children in Grades 3 and 4. Reading and Writing, 30(6), 1287–1310. https://doi.org/10.1007/s11145-017-9724-6.

    Article  Google Scholar 

  33. Leontyev, A. N. (1978). Activity, consciousness, and personality. Englewood Cliffs, NJ: Prentice-Hall.

    Google Scholar 

  34. Leontyev, A. N. (1981). Problems of the development of mind. Moscow: Progress.

    Google Scholar 

  35. Macarthur, C. A. (1999). Overcoming barriers to writing: Computer support for basic writing skills. Reading & Writing Quarterly, 15(2), 169–192. https://doi.org/10.1080/105735699278251.

    Article  Google Scholar 

  36. MacArthur, C. A. (2009). Reflections on research on writing and technology for struggling writers. Learning Disabilities Research & Practice, 24(2), 93–103.

    Article  Google Scholar 

  37. McCutchen, D. (1996). A capacity theory of writing: Working memory in composition. Educational Psychology Review, 8(3), 299–325. https://doi.org/10.1007/BF01464076.

    Article  Google Scholar 

  38. McNamara, D. S., Crossley, S. A., & Roscoe, R. (2013). Natural language processing in an intelligent writing strategy tutoring system. Behavior Research Methods, 45(2), 499–515. https://doi.org/10.3758/s13428-012-0258-1.

    Article  Google Scholar 

  39. Mo, Y., & Troia, G. (2017). Similarities and differences in constructs represented by U.S. States’ middle school writing tests and the 2007 National Assessment of Educational Progress writing assessment. Assessing Writing, 33(July 17), 48–67. https://doi.org/10.1016/j.asw.2017.06.001.

    Article  Google Scholar 

  40. Morphy, P., & Graham, S. (2012). Word processing programs and weaker writers/readers: A meta-analysis of research findings. Reading and Writing, 25(3), 641–678. https://doi.org/10.1007/s11145-010-9292-5.

    Article  Google Scholar 

  41. Murphy, S., & Smith, M. A. (2018). The faraway stick cannot kill the nearby snake. In C. Bazerman, A. Applebee, V. Berninger, D. Brandt, S. Graham, J. Jeffery, P. Matsuda, S. Murphy, D. Rowe, M. Schleppegrell, & K. Wilcox (Eds.), The lifespan development of writing (pp. 210–243). Urbana, IL: National Council of English.

    Google Scholar 

  42. National Assessment Governing Board (NAGB). (2010). Writing framework for the 2011 National Assessment of Educational Progress. Washington, DC: U.S. Department of Education.

    Google Scholar 

  43. National Center for Education Statistics (NCES). (2012). The Nation’s report card: Writing 2011. Washington, DC: Institute of Education Sciences, U.S. Department of Education.

    Google Scholar 

  44. National Center for Education Statistics (NCES). (2014a). Lessons learned from the 2012 grade 4 writing computer-based assessment study. Washington, DC: Institute of Education Sciences, U.S. Department of Education. Retrieved February 28, 2019 from, http://nces.ed.gov/nationsreportcard/writing/lessons/default.aspx.

  45. National Center for Education Statistics (NCES). (2014b). NAEP 2012 pilot writing computer-based assessment study-grade 4: Technical memorandum: Institute of Education Sciences. Institute of Education Sciences, U.S. Department of Education: Washington, DC.

    Google Scholar 

  46. National Center for Education Statistics (NCES). (n.d.). NAEP technical documentation: Calculation and use of replicate weights. Washington, D.C.: Institute of Education Sciences, U.S. Department of Education. Retrieved April 19, 2018 from https://nces.ed.gov/nationsreportcard/tdw/weighting/2000_2001/2000state_repweights.aspx.

  47. Olinghouse, N., Santangelo, T., & Wilson, J. (2012). Examining the validity of single-occasion, single-genre, holistically scored writing assessments. In E. V. Steendam, M. Tillema, G. Rijlaarsdam, & H. V. D. Bergh (Eds.), Measuring writing: Recent insights into theory, methodology and practices (pp. 55–82). New York: Guilford.

    Google Scholar 

  48. Porter, A., McMaken, J., Hwang, J., & Yang, R. (2011). Common core standards: The new US intended curriculum. Educational Researcher, 40(3), 103–116. https://doi.org/10.3102/0013189X11405038.

    Article  Google Scholar 

  49. Purcell, K., Buchanan, J., & Friedrich, L. (2013a). The impact of digital tools on student writing and how writing is taught in schools. Washington, DC: Pew Research Center’s Interent & American Life Project. Retrieved February 28, 2019 from http://pewinternet.org/Reports/2013/Teachers-technology-and-writing.

  50. Purcell, K., Heaps, A., Buchanan, J., & Friedrich, L. (2013b). How teachers are using technology at home and in their classrooms. Washington, DC: Pew Research Center’s Interent & American Life Project. Retrieved February 28, 2019 from http://pewinternet.org/Reports/2013/Teachers-and-technology.

  51. Rogers, A., Stoeckel, J., & Sikali, E. (2013). National Assessment of Educational Progress (NAEP) 2011 writing assessment restricted-use data files data companion. Washington, DC: National Center for Education Statistics.

    Google Scholar 

  52. Russell, M., & Haney, W. (1997). Testing writing on computers. Education Policy Analysis Archives, 5(3), 1–20. https://doi.org/10.14507/epaa.v5n3.1997.

    Article  Google Scholar 

  53. Russell, M., & Plati, T. (2002). Does it matter with what I write? Comparing performance on paper, computer and portable writing devices. Current Issues in Education, 5(4), 1–15.

    Google Scholar 

  54. Sandene, B., Horkay, N., Bennett, R. E., Allen, N., Braswell, J., Kaplan, B., et al. (2005). Online assessment in mathematics and writing: Reports from the NAEP Technology-Based Assessment Project, Research and Development Series. NCES 2005-457. Washington, DC: Institute of Education Sciences, U.S. Department of Education.

  55. Tate, T., Warschauer, M., & Abedi, J. (2016). The effects of prior computer use on computer-based writing: The 2011 NAEP writing assessment. Computers & Education, 101, 115–131. https://doi.org/10.1016/j.compedu.2016.06.001.

    Article  Google Scholar 

  56. Tate, T., Warschauer, M., & Kim, Y.-S. G. (2019). Learning to compose digitally: The effect of prior computer use and keyboard activity on NAEP writing. Reading & Writing. https://doi.org/10.1007/s11145-019-09940-z.

    Article  Google Scholar 

  57. Vargha, A., Torma, B., & Bergman, L. R. (2015). ROPstat: A general statistical package useful for conducting person-oriented analyses. Journal for Person-Oriented Research, 1(1-2), 87–98. https://doi.org/10.17505/jpor.2015.09.

    Article  Google Scholar 

  58. Vygotsky, L. S. (1962). Thought and language. Cambridge, MA: MIT Press. https://doi.org/10.1037/11193-000.

    Google Scholar 

  59. Vygotsky, L. S. (1978). Mind and society. Cambridge, MA: MIT Press.

    Google Scholar 

  60. Vygotsky, L. S. (1981). The genesis of higher mental functions. In J. V. Wertsch (Ed.), The concept of activity in Soviet psychology (pp. 144–188). Armonk, NY: M.E. Sharpe.

    Google Scholar 

  61. Wang, Q., & Pomerantz, E. M. (2009). The motivational landscape of early adolescence in the United States and China: A longitudinal investigation. Child Development, 80(4), 1272–1287. https://doi.org/10.1111/j.1467-8624.2009.01331.x.

    Article  Google Scholar 

  62. Warschauer, M. (2006). Laptops and literacy: Learning in the wireless classroom. New York: Teachers College Press.

    Google Scholar 

  63. Way, W. D., Davis, L. L., & Strain-Seymour, E. (2008). The validity case for assessing direct writing by computer. A Pearson Assessments & Information White Paper.

  64. Wenglinsky, H. (2005). Using technology wisely: The keys to success in schools. New York, NY: Teachers College Press.

    Google Scholar 

  65. Wertsch, J. V. (1979). From social interaction to higher psychological processes. A clarification and application of Vygotsky’s theory. Human Development, 22(1), 1–22.

    Article  Google Scholar 

  66. Wertsch, J. V. (1991). Voices of the mind: A sociocultural approach to mediated action. Cambridge, MA: Harvard University Press.

    Google Scholar 

  67. White, S., Kim, Y. Y., Chen, J., & Liu, F. (2015). Performance of fourth-grade students in the 2012 NAEP computer-based writing pilot assessment: Scores, text length, and use of editing tools. NCES 2015-119. Washington, DC: Institute of Education Sciences, US Department of Education.

  68. Wigfield, A., & Eccles, J. S. (2000). Expectancy–value theory of achievement motivation. Contemporary Educational Psychology, 25(1), 68–81. https://doi.org/10.1006/ceps.1999.1015.

    Article  Google Scholar 

  69. Williamson, G. L., Fitzgerald, J., & Stenner, A. J. (2013). The Common Core State Standards’ quantitative text complexity trajectory: Figuring out how much complexity is enough. Educational Researcher, 42(2), 59–69. https://doi.org/10.3102/0013189X12466695.

    Article  Google Scholar 

  70. Zheng, B., Lawrence, J., Warschauer, M., & Lin, C. H. (2015). Middle school students’ writing and feedback in a cloud-based classroom environment. Technology, Knowledge and Learning, 20(2), 201–229.

    Article  Google Scholar 

  71. Zwick, R. (1987). Assessing the dimensionality of NAEP reading data. Journal of Educational Measurement, 24(4), 293–308.

    Article  Google Scholar 

Download references

Acknowledgements

This work has been supported by funding from the Spencer Foundation, Grant No. 201500153.

Author information

Affiliations

Authors

Corresponding author

Correspondence to Tamara P. Tate.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Tate, T.P., Warschauer, M. Keypresses and Mouse Clicks: Analysis of the First National Computer-Based Writing Assessment. Tech Know Learn 24, 523–543 (2019). https://doi.org/10.1007/s10758-019-09412-x

Download citation

Keywords

  • NAEP
  • Writing
  • Secondary students
  • Digital writing
  • Log data
  • Clickstream data
  • Trace data