Advertisement

Technology, Knowledge and Learning

, Volume 24, Issue 4, pp 523–543 | Cite as

Keypresses and Mouse Clicks: Analysis of the First National Computer-Based Writing Assessment

  • Tamara P. TateEmail author
  • Mark Warschauer
Original research

Abstract

The quality of students’ writing skills continues to concern educators. Because writing is essential to success in both college and career, poor writing can have lifelong consequences. Writing is now primarily done digitally, but students receive limited explicit instruction in digital writing. This lack of instruction means that students fail to take advantage of the affordances of digital tools. The writing process is shaped by the tools used, which makes digital writing, to an extent, different from writing with pen and paper. To better understand students’ digital writing skills, we take advantage of the information provided by computer-based assessments—keyboard and mouse activity data. We examine the relationship between students’ use of the keyboard and mouse during the assessment and students’ writing achievement. Our data comes from the first national computer-based writing assessment in the United States, the 2011 National Assessment of Educational Progress (NAEP) assessment. Using data from over 24,100 eighth-grade students, we found that the number of keypresses had a distinct and direct effect on writing achievement scores, controlling for word count. We also identified several different patterns of keyboard and mouse activity on the computer-based NAEP assessment.

Keywords

NAEP Writing Secondary students Digital writing Log data Clickstream data Trace data 

Notes

Acknowledgements

This work has been supported by funding from the Spencer Foundation, Grant No. 201500153.

References

  1. Allen, N. L., & Donoghue, J. R. (1996). Applying the Mantel–Haenszel procedure to complex samples of items. Journal of Educational Measurement, 33(2), 231–251.  https://doi.org/10.1111/j.1745-3984.1996.tb00491.x.CrossRefGoogle Scholar
  2. Allen, N. L., Donoghue, J. R., & Schoeps, T. L. (2001). The NAEP 1998 Technical Report. NCES 2001-509. Washington, DC: National Center for Education Statistics.Google Scholar
  3. Almond, R., Deane, P., Quinlan, T., Wagner, M., & Sydorenko, T. (2012). A preliminary analysis of keystroke log data from a timed writing task. ETS Research Report Series, 2012(2), i–61.Google Scholar
  4. Applebee, A. N. (2007). Issues in large-scale writing assessment perspectives from the National Assessment of Educational Progress. Journal of Writing Assessment, 3(2), 81–98. Retrieved from http://www.journalofwritingassessment.org/archives/3-2.2pdf.
  5. Applebee, A. N., & Langer, J. A. (2011). A snapshot of writing instruction in middle schools and high schools. English Journal, 100(6), 14–27.Google Scholar
  6. Bazerman, C., Applebee, A., Berninger, V., Brandt, D., Graham, S., Jeffery, J., et al. (Eds.). (2018). The lifespan development of writing. Urbana, IL: National Council of English.Google Scholar
  7. Beaton, A., Rogers, A., Gonzalez, E., Hanly, M., Kolstad, A., Rust, K., et al. (2011). The NAEP Primer. Washington, DC: National Center for Educational Statistics.Google Scholar
  8. Beaton, A. E., & Zwick, R. (1992). Chapter 1: Overview of the National Assessment of Educational Progress. Journal of Educational Statistics, 17(2), 95–109.  https://doi.org/10.3102/10769986017002095.CrossRefGoogle Scholar
  9. Benjamini, Y., & Hochberg, Y. (1995). Controlling the false discovery rate: A practical and powerful approach to multiple testing. Journal of the Royal Statistical Society: Series B (Methodological), 57(1), 289–300.Google Scholar
  10. Bergman, L. R., Magnusson, D., & El-Khouri, B. M. (2002). Studying individual development in an interindividual context. A person-oriented approach. Mahwah, New Jersey, London: Lawrence-Erlbaum Associates.Google Scholar
  11. Chen, E., Niemi, D., Wang, J., Wang, H., & Mirocha, J. (2007). Examining the generalizability of direct writing assessment tasks. CSE technical report 718. Los Angeles, CA: National Center for Research on Evaluation, Standards, and Student Testing (CRESST).  https://doi.org/10.1037/e643812011-001.
  12. Chun, D., Smith, B., & Kern, R. (2016). Technology in language use, language teaching, and language learning. Modern Language Journal, 100, 64–80.  https://doi.org/10.1111/modl.12302.CrossRefGoogle Scholar
  13. Cochran-Smith, M. (1991). Word processing and writing in elementary classrooms: A critical review of related literature. Review of Educational Research, 61(1), 107–155.  https://doi.org/10.3102/00346543061001107.CrossRefGoogle Scholar
  14. Common Core State Standards Initiative (CCSSI). (2017). Language progressive skills. Retrieved June 07, 2017, from http://www.corestandards.org/ELA-Literacy/L/language-progressive-skills/.
  15. Crossley, S. A., Weston, J., McLain Sullivan, S. T., & McNamara, D. S. (2011). The development of writing proficiency as a function of grade level: A linguistic analysis. Written Communication, 28(3), 282–311.  https://doi.org/10.1177/0741088311410188.CrossRefGoogle Scholar
  16. Daiute, C. (1986). Physical and cognitive factors in revising: Insights from studies with computers. Research in the Teaching of English, 20(2), 141–159.Google Scholar
  17. De La Paz, S., & Graham, S. (2002). Explicitly teaching strategies, skills, and knowledge: Writing instruction in middle school classrooms. Journal of Educational Psychology, 94(4), 687.  https://doi.org/10.1037/0022-0663.94.4.687.CrossRefGoogle Scholar
  18. Deane, P. (2011). Writing assessment and cognition. ETS Research Report Series, 1, i–60.  https://doi.org/10.1002/j.2333-8504.2011.tb02250.x.CrossRefGoogle Scholar
  19. DeVoss, D. N., Eidman-Aadahl, E., & Hicks, T. (2010). Because digital writing matters: Improving student writing in online and multimedia environments. Hoboken: Wiley.Google Scholar
  20. Eccles, J., & Midgley, C. (1990). Changes in academic motivation and self-perception during early adolescence. In R. Montemayor (Ed.), Early adolescence as a time of transition (pp. 1–29). Beverly Hills, CA: Sage Publishing.Google Scholar
  21. Engeström, Y. (1987). Learning by expanding: An activity-theoretical approach to developmental research. Helsinki: Orienta-Konsultit Oy.Google Scholar
  22. Fang, Z., Schleppegrell, M. J., & Cox, B. E. (2006). Understanding the language demands of schooling: Nouns in academic registers. Journal of Literacy Research, 38(3), 247–273.  https://doi.org/10.1207/s15548430jlr3803_1.CrossRefGoogle Scholar
  23. Goldfine, R. (2001). Making word processing more effective in the composition classroom. Teaching English in the Two-Year College, 28(3), 307–315.Google Scholar
  24. Graham, S. (2018). A writer(s) within community model of writing. In C. Bazerman, A. Applebee, V. Berninger, D. Brandt, S. Graham, J. Jeffery, P. Matsuda, S. Murphy, D. Rowe, M. Schleppegrell, & K. Wilcox (Eds.), The lifespan development of writing (pp. 271–325). Urbana, IL: National Council of English.  https://doi.org/10.1080/00461520.2018.1481406.CrossRefGoogle Scholar
  25. Graham, S., & Perin, D. (2007). A meta-analysis of writing instruction for adolescent students. Journal of Educational Psychology, 99(3), 445–476.  https://doi.org/10.1037/0022-0663.99.3.445.CrossRefGoogle Scholar
  26. Greenleaf, C. (1994). Technological indeterminacy: The role of classroom writing practices and pedagogy in shaping student use of the computer. Written Communication, 11(1), 85–130.  https://doi.org/10.1177/0741088394011001005.CrossRefGoogle Scholar
  27. Haertel, E. (1984). An application of latent class models to assessment data. Applied Psychological Measurement, 8(3), 333–346.CrossRefGoogle Scholar
  28. Halliday, M. A. K. (1993). Towards a language-based theory of learning. Lingquistics and Education, 5, 93–116.  https://doi.org/10.1016/0898-5898(93)90026-7.CrossRefGoogle Scholar
  29. Horkay, N., Bennett, R. E., Allen, N., Kaplan, B., Yan, F. (2006). Does it matter if I take my writing test on computer? An empirical study of mode effects in NAEP. Journal of Technology, Learning, and Assessment, 5(2), 4–48.Google Scholar
  30. Johnson, E. G. (1992). The design of the National Assessment of Educational Progress. Journal of Educational Measurement, 29(2), 95–110.  https://doi.org/10.1111/j.1745-3984.1992.tb00369.x.CrossRefGoogle Scholar
  31. KewalRamani, A., Zhang, J., Wang, X., Rathbun, A., Corcoran, L., Diliberti, M., et al. (2018). Student access to digital learning resources outside of the classroom (NCES 2017-098). U.S. Department of Education. Washington, DC: National Center for Education Statistics. Retrieved April 6, 2018 from https://nces.ed.gov/pubsearch/pubsinfo.asp?pubid=2017098.
  32. Kim, Y. S. G., Schatschneider, C., Wanzek, J., Gatlin, B., & Al Otaiba, S. (2017). Writing evaluation: rater and task effects on the reliability of writing scores for children in Grades 3 and 4. Reading and Writing, 30(6), 1287–1310.  https://doi.org/10.1007/s11145-017-9724-6.CrossRefGoogle Scholar
  33. Leontyev, A. N. (1978). Activity, consciousness, and personality. Englewood Cliffs, NJ: Prentice-Hall.Google Scholar
  34. Leontyev, A. N. (1981). Problems of the development of mind. Moscow: Progress.Google Scholar
  35. Macarthur, C. A. (1999). Overcoming barriers to writing: Computer support for basic writing skills. Reading & Writing Quarterly, 15(2), 169–192.  https://doi.org/10.1080/105735699278251.CrossRefGoogle Scholar
  36. MacArthur, C. A. (2009). Reflections on research on writing and technology for struggling writers. Learning Disabilities Research & Practice, 24(2), 93–103.CrossRefGoogle Scholar
  37. McCutchen, D. (1996). A capacity theory of writing: Working memory in composition. Educational Psychology Review, 8(3), 299–325.  https://doi.org/10.1007/BF01464076.CrossRefGoogle Scholar
  38. McNamara, D. S., Crossley, S. A., & Roscoe, R. (2013). Natural language processing in an intelligent writing strategy tutoring system. Behavior Research Methods, 45(2), 499–515.  https://doi.org/10.3758/s13428-012-0258-1.CrossRefGoogle Scholar
  39. Mo, Y., & Troia, G. (2017). Similarities and differences in constructs represented by U.S. States’ middle school writing tests and the 2007 National Assessment of Educational Progress writing assessment. Assessing Writing, 33(July 17), 48–67.  https://doi.org/10.1016/j.asw.2017.06.001.CrossRefGoogle Scholar
  40. Morphy, P., & Graham, S. (2012). Word processing programs and weaker writers/readers: A meta-analysis of research findings. Reading and Writing, 25(3), 641–678.  https://doi.org/10.1007/s11145-010-9292-5.CrossRefGoogle Scholar
  41. Murphy, S., & Smith, M. A. (2018). The faraway stick cannot kill the nearby snake. In C. Bazerman, A. Applebee, V. Berninger, D. Brandt, S. Graham, J. Jeffery, P. Matsuda, S. Murphy, D. Rowe, M. Schleppegrell, & K. Wilcox (Eds.), The lifespan development of writing (pp. 210–243). Urbana, IL: National Council of English.Google Scholar
  42. National Assessment Governing Board (NAGB). (2010). Writing framework for the 2011 National Assessment of Educational Progress. Washington, DC: U.S. Department of Education.Google Scholar
  43. National Center for Education Statistics (NCES). (2012). The Nation’s report card: Writing 2011. Washington, DC: Institute of Education Sciences, U.S. Department of Education.Google Scholar
  44. National Center for Education Statistics (NCES). (2014a). Lessons learned from the 2012 grade 4 writing computer-based assessment study. Washington, DC: Institute of Education Sciences, U.S. Department of Education. Retrieved February 28, 2019 from, http://nces.ed.gov/nationsreportcard/writing/lessons/default.aspx.
  45. National Center for Education Statistics (NCES). (2014b). NAEP 2012 pilot writing computer-based assessment study-grade 4: Technical memorandum: Institute of Education Sciences. Institute of Education Sciences, U.S. Department of Education: Washington, DC.Google Scholar
  46. National Center for Education Statistics (NCES). (n.d.). NAEP technical documentation: Calculation and use of replicate weights. Washington, D.C.: Institute of Education Sciences, U.S. Department of Education. Retrieved April 19, 2018 from https://nces.ed.gov/nationsreportcard/tdw/weighting/2000_2001/2000state_repweights.aspx.
  47. Olinghouse, N., Santangelo, T., & Wilson, J. (2012). Examining the validity of single-occasion, single-genre, holistically scored writing assessments. In E. V. Steendam, M. Tillema, G. Rijlaarsdam, & H. V. D. Bergh (Eds.), Measuring writing: Recent insights into theory, methodology and practices (pp. 55–82). New York: Guilford.Google Scholar
  48. Porter, A., McMaken, J., Hwang, J., & Yang, R. (2011). Common core standards: The new US intended curriculum. Educational Researcher, 40(3), 103–116.  https://doi.org/10.3102/0013189X11405038.CrossRefGoogle Scholar
  49. Purcell, K., Buchanan, J., & Friedrich, L. (2013a). The impact of digital tools on student writing and how writing is taught in schools. Washington, DC: Pew Research Center’s Interent & American Life Project. Retrieved February 28, 2019 from http://pewinternet.org/Reports/2013/Teachers-technology-and-writing.
  50. Purcell, K., Heaps, A., Buchanan, J., & Friedrich, L. (2013b). How teachers are using technology at home and in their classrooms. Washington, DC: Pew Research Center’s Interent & American Life Project. Retrieved February 28, 2019 from http://pewinternet.org/Reports/2013/Teachers-and-technology.
  51. Rogers, A., Stoeckel, J., & Sikali, E. (2013). National Assessment of Educational Progress (NAEP) 2011 writing assessment restricted-use data files data companion. Washington, DC: National Center for Education Statistics.Google Scholar
  52. Russell, M., & Haney, W. (1997). Testing writing on computers. Education Policy Analysis Archives, 5(3), 1–20.  https://doi.org/10.14507/epaa.v5n3.1997.CrossRefGoogle Scholar
  53. Russell, M., & Plati, T. (2002). Does it matter with what I write? Comparing performance on paper, computer and portable writing devices. Current Issues in Education, 5(4), 1–15.Google Scholar
  54. Sandene, B., Horkay, N., Bennett, R. E., Allen, N., Braswell, J., Kaplan, B., et al. (2005). Online assessment in mathematics and writing: Reports from the NAEP Technology-Based Assessment Project, Research and Development Series. NCES 2005-457. Washington, DC: Institute of Education Sciences, U.S. Department of Education.Google Scholar
  55. Tate, T., Warschauer, M., & Abedi, J. (2016). The effects of prior computer use on computer-based writing: The 2011 NAEP writing assessment. Computers & Education, 101, 115–131.  https://doi.org/10.1016/j.compedu.2016.06.001.CrossRefGoogle Scholar
  56. Tate, T., Warschauer, M., & Kim, Y.-S. G. (2019). Learning to compose digitally: The effect of prior computer use and keyboard activity on NAEP writing. Reading & Writing.  https://doi.org/10.1007/s11145-019-09940-z.CrossRefGoogle Scholar
  57. Vargha, A., Torma, B., & Bergman, L. R. (2015). ROPstat: A general statistical package useful for conducting person-oriented analyses. Journal for Person-Oriented Research, 1(1-2), 87–98.  https://doi.org/10.17505/jpor.2015.09.CrossRefGoogle Scholar
  58. Vygotsky, L. S. (1962). Thought and language. Cambridge, MA: MIT Press.  https://doi.org/10.1037/11193-000.CrossRefGoogle Scholar
  59. Vygotsky, L. S. (1978). Mind and society. Cambridge, MA: MIT Press.Google Scholar
  60. Vygotsky, L. S. (1981). The genesis of higher mental functions. In J. V. Wertsch (Ed.), The concept of activity in Soviet psychology (pp. 144–188). Armonk, NY: M.E. Sharpe.Google Scholar
  61. Wang, Q., & Pomerantz, E. M. (2009). The motivational landscape of early adolescence in the United States and China: A longitudinal investigation. Child Development, 80(4), 1272–1287.  https://doi.org/10.1111/j.1467-8624.2009.01331.x.CrossRefGoogle Scholar
  62. Warschauer, M. (2006). Laptops and literacy: Learning in the wireless classroom. New York: Teachers College Press.Google Scholar
  63. Way, W. D., Davis, L. L., & Strain-Seymour, E. (2008). The validity case for assessing direct writing by computer. A Pearson Assessments & Information White Paper.Google Scholar
  64. Wenglinsky, H. (2005). Using technology wisely: The keys to success in schools. New York, NY: Teachers College Press.Google Scholar
  65. Wertsch, J. V. (1979). From social interaction to higher psychological processes. A clarification and application of Vygotsky’s theory. Human Development, 22(1), 1–22.CrossRefGoogle Scholar
  66. Wertsch, J. V. (1991). Voices of the mind: A sociocultural approach to mediated action. Cambridge, MA: Harvard University Press.Google Scholar
  67. White, S., Kim, Y. Y., Chen, J., & Liu, F. (2015). Performance of fourth-grade students in the 2012 NAEP computer-based writing pilot assessment: Scores, text length, and use of editing tools. NCES 2015-119. Washington, DC: Institute of Education Sciences, US Department of Education.Google Scholar
  68. Wigfield, A., & Eccles, J. S. (2000). Expectancy–value theory of achievement motivation. Contemporary Educational Psychology, 25(1), 68–81.  https://doi.org/10.1006/ceps.1999.1015.CrossRefGoogle Scholar
  69. Williamson, G. L., Fitzgerald, J., & Stenner, A. J. (2013). The Common Core State Standards’ quantitative text complexity trajectory: Figuring out how much complexity is enough. Educational Researcher, 42(2), 59–69.  https://doi.org/10.3102/0013189X12466695.CrossRefGoogle Scholar
  70. Zheng, B., Lawrence, J., Warschauer, M., & Lin, C. H. (2015). Middle school students’ writing and feedback in a cloud-based classroom environment. Technology, Knowledge and Learning, 20(2), 201–229.CrossRefGoogle Scholar
  71. Zwick, R. (1987). Assessing the dimensionality of NAEP reading data. Journal of Educational Measurement, 24(4), 293–308.CrossRefGoogle Scholar

Copyright information

© Springer Nature B.V. 2019

Authors and Affiliations

  1. 1.School of EducationUniversity of CaliforniaIrvineUSA

Personalised recommendations