Investigating Writing Processes with Keystroke Logging

Abstract

Already in the 1970s, researchers in linguistics and psychology became interested in understanding how written language production worked, why students’ texts ended up in a specific way, and whether writing instruction could be improved by an increased understanding of students’ actual activities during writing - what happens “behind the scenes”. They observed writing processes through video-recordings and think-aloud protocols, both of which required laborious manual analyses, but with the advent of affordable computers in the 1990s keystroke logging was developed. Keystroke logging records all keystrokes and mouse movements and provide them with a time stamp to allow playback and analyses. The purpose of this chapter is to introduce the reader to the concept of keystroke logging, explain briefly how it works, and give an overview of currently available software. First, we provide a short historical background. We then move into the core idea and functionality of keystroke logging in general before turning to descriptions of specific pieces of software. We summarise similarities and differences, aiming to show that choice of software should be governed by the research question. Finally, we discuss research that uses keystroke logging as a research tool, and provide examples of research about keystroke logging as a pedagogical tool.

Behind each written text is a writer who wants to communicate something. He or she may have spent considerable time thinking about content and structure during the writing process and meanwhile carefully formulated and revised sentences to create, for instance, an exciting story, a good argument, or a comprehensive message. These underlying processes are, however, invisible for the reader, perhaps a teacher who meets the product of those processes—the finally edited text. As pointed out by Chukarev-Hudalainen (2019, p. 126) “nobody stands behinds students’ shoulders and watches how they write their texts”. It has, however, been shown that the type and timing of fine-grained processes that writers engage in during text production, can account for up to 80% of the variance in writing quality (Breetvelt et al., 1994). Understanding students’ writing processes may therefore be of essence for teachers of academic writing.

Already in the end of 1970s and the beginning of the 1980s, researchers in linguistics and psychology became interested in understanding how written language production worked, why students’ texts ended up in a specific way, and whether the teaching of writing could be improved by an increased understanding of the students’ actual activities during writing. Independent of each other, but more or less simultaneously, Hayes and Flower (1980) in the US and Matsuhashi (1982) in the UK developed different approaches to achieve this.

Hayes and Flower, with a background in cognitive psychology, let writers think aloud about their processes while writing a text. Their main interest was to understand the major components of the writing process and the interaction between them, and the analyses of these self-reported data resulted in their seminal model of writing (Hayes & Flower, 1980). The original model, with planning, translation, and revision as the main components, has since been updated several times, and in 2014 Hayes & Berninger proposed a three-level framework consisting of a control level, a process level, and a resource level (Hayes & Berninger, 2014). The process level includes components of the writing processes and the task environment. The process components are the proposer (pre-linguistic idea generation), the translator (formulation), the transcriber (graphomotor planning and execution of handwriting or key-pressing), and the evaluator (checks whether ideas and produced texts will meet the goals of the topic, make the right impact on the audience etc., or whether this information could be improved to better meet those goals). The task environment includes collaborators and critics, task materials, and physical transcribing technology. The underlying assumption is that cognitive resources are limited—if one process requires more resources, there will be less available for others (McCutchen, 1996; Olive et al., 2002). Think-aloud protocols have been, and are still, used fairly extensively (Wengelin et al., 2019), because they are capable of providing rich information about the higher-level processes in text production. They have, however, been criticized both for relying on the writers’ subjective experiences and for being reactive.

The method used by Matsuhashi (1982), who had a linguistic background and was more interested in the lower-level linguistic processes, was less intrusive, and was inspired by spoken-language researchers such as Fromkin (1973), Goldman-Eisler (1968), and Hockett (1967). They viewed pauses in speech as a window to the linguistic and cognitive processes underlying language production. To capture pauses in writing, Matsuhashi video-recorded writers, zooming in on their hands, to register when they lifted and lowered their pens to think or to revise. This method offered no verbalised insights to the writers’ minds, but provided more fine-grained and objective information, and was less reactive than think-aloud protocols.

Both these methods generated data which required laborious work and extensive interpretation by the analysts. With the advent of affordable and available personal computers, keystroke logging software, and later automatic handwriting capturing, were developed, both of which can be viewed as a modern and more automatized version of Matsuhashi’s video recordings. Only keystroke logging is covered in this chapter. This technology provides (a) possibilities to capture the temporal dynamics of typing and revision with higher resolution than video recordings and (b) automatic analyses of these patterns. During the 1990s four programs became known among writing researchers—interestingly, once again produced more or less independent of each other: FAU-word (Levy & Ransdell, 1994) which was developed at Florida Atlantic University, USA; TraceIt (Severinson Eklundh & Kollberg, 1992), which was developed at the Royal Institute of Technology, Stockholm, Sweden; ScriptLog (Strömqvist & Malmsten, 1997) which was developed at University of Gothenburg, Sweden, and TransLog (Lykke-Jakobsen, 1999) which was developed at Copenhagen Business School, Denmark specifically for translation studies. Of these four, ScriptLog and TransLog are still being developed, in parallel with a few more recent pieces of software. We outline currently available programs, their main functionality, and similarities and differences between them in the tools list at the end of the chapter.

1 Core Idea of the Technology

The core idea of keystroke logging is to record all the events of the writing process and thereby enable researchers to observe writing processes and thus increase their understanding of what happens “behind the scenes” during text production. Each event—a keystroke or a mouse movement—is provided with a time stamp to allow playback and analyses. The example below shows the first two sentences of a text produced by a Swedish university student who participated in an experimental writing research project at her university (rough English translation is provided below).

Fusk är ett av de vanligare problemen i skolor, och förmodligen långt vanligare än de flesta lärare vet om. I regel är det nog vanligare att pojkar fuskar än att flickor gör det, och om vi tittar närmre på könsrollerna så kan vi kanske hitta ledtrådar till hur barn (jag kommer utgå från barn i skolålder i denna text) förhåller sig till fusk.

[Cheating is one of the most common problems in schools, and probably much more common than most teachers know. As a rule, it is probably more common that boys cheat than that girls do it, and if we look closer at the gender roles we may perhaps find clues to how children (I will children of school age as my point of departure) relate to cheating.]

After the writing session, the full text encompassed 16 orthographic sentences, 377 words or 2289 characters including spaces, punctuation marks, and line feeds. To produce her 2289 characters, the writer conducted 3504 key “events” (key presses including arrow keys and backspace but also mouse movements). Thus, her total her writing process included more than 1200 events of which the readers are not aware—probably mainly revisions but she could also have just moved around in the text, or even written and deleted gibberish as a thinking strategy. Possibly (hopefully) one reason for her thinking and revising was to adapt the finally edited text to the intended receiver.

The basic type of events generated by keystroke logging programs are keystrokes that change the text somehow—letters, numbers, punctuation, spaces, line feeds etc., but also keypresses/events, such as delete, backspace, arrow keys, cut and paste shortcuts etc. Further, most programs available today record mouse movements and, more recently, some have incorporated the possibility to synchronize the recordings with eye trackers. This enables investigations of writers’ reading of their own emerging texts during writing. In common for all programs is that the events they record are saved in a log, which can be used both for generating various types of analysis files and for playback of the writing process. The table below shows a part of a simplified log file corresponding to the text fragment in 1a. Each line corresponds to a recorded event during the writing session. The left column shows the time stamp, and the right column the content of the event. We enter the scene just as the writer is about to finish the first sentence. The first line of the log file shows the full stop ending the first sentence. This is followed by a ‘space’. Then–as the time stamp indicates–there is a pause of approximately 9 seconds after which the writer starts the second sentence with the words Fusk har nog förekommit (‘cheating has probably occurred’). However, she changes her mind and deletes 26 characters by means of 26 consecutive presses of the BACKSPACE key. Instead of these, she writes the words: I regel (‘As a rule’) which are the two first words of the second sentence in the finally edited text.

Time stamp (event ended)	Event output
00:01:16.019
00:01:16.219	SPACE
00:01:25.731	F
00:01:25.930	u
00:01:26.091	s
00:01:26.211	k
00:01:26.379	SPACE
00:01:26.563	h
00:01:26.747	a
00:01:26.923	r
00:01:27.019	SPACE
00:01:27.363	a
00:01:28.179	l
00:01:28.347	l
00:01:28.523	t
00:01:29.058	BACK
00:01:29.250	BACK
00:01:29.434	BACK
00:01:29.650	BACK
00:01:30.811	n
00:01:31.139	å
00:01:31.978	BACK
00:01:32.443	o
00:01:32.578	g
00:01:32.803	SPACE
00:01:33.027	b
00:01:33.394	BACK
00:01:33.546	f
00:01:33.706	ö
00:01:33.866	r
00:01:34.211	e
00:01:34.499	k
00:01:34.618	o
00:01:34.882	m
00:01:35.018	m
00:01:35.164	i
00:01:35.298	t
00:01:35.434	SPACE
00:01:35.634	s
00:01:35.763	å
00:01:36.098	BACK
00:01:36.593	BACK
00:01:36.639	BACK
00:01:36.686	BACK
00:01:36.733	BACK
00:01:36.780	BACK
00:01:36.826	BACK
00:01:36.873	BACK
00:01:36.920	BACK
00:01:36.967	BACK
00:01:37.014	BACK
00:01:37.061	BACK
00:01:37.108	BACK
00:01:37.155	BACK
00:01:37.201	BACK
00:01:37.248	BACK
00:01:37.570	BACK
00:01:37.810	BACK
00:01:38.034	BACK
00:01:38.226	BACK
00:01:38.418	BACK
00:01:38.578	BACK
00:01:38.730	BACK
00:01:38.898	BACK
00:01:39.082	BACK
00:01:39.258	BACK
00:01:50.658	I
00:01:50.826	SPACE
00:01:51.074	r
00:01:51.314	e
00:01:51.522	g
00:01:51.706	e
00:01:51.794	l
00:01:51.930	SPACE

Keystroke logging tends to generate massive amounts of data, and the readability of log files, even such a simple one as in the example above, is low. To increase readability, most program also generate a linear file which shows what is typically considered a static replacement of the playback. It shows the linear representation of how the text was produced:

<START><34.119>Fusk <4.463>är <10.543><BACKSPACE1>, <BACKSPACE2> ett s<BACKSPACE1>av de vanligare problemen i skolor, och förmodligen långt vanligara än de flesta vä<BACKSPACE2>lärare vet om. <9.511>Fusk har allt<BACKSPACE4>nå<BACKSPACE1>og b<BACKSPACE1>förekommit så<BACKSPACE26><11.399>I regel är det npog<BACKSPACE2>o<BACKSPACE2>og vanligare att pojkar fuskar<BACKSPACE2>ar än att flickor gör det, och om vi tt<BACKSPACE1>ittar närmre på könsro<BACKSPACE24> tittar närmre på ko<BACKSPACE1>önstr<BACKSPACE2>rollerna så ä<BACKSPACE1>kan vi npg<BACKSPACE2>og hitta en <4.423>ledtråf<BACKSPACE1>d till <3.391>hur <4.327>barn (jag kommer utgå från barn i skolålder i denna tes<BACKSPACE1>z<BACKSPACE1>xt) <2.879>förhåller sig till fusk.

Numbers in angle brackets show pauses in the writing process, here defined as keyboard inactivity longer than 2 seconds (but see for example Wengelin (2006) for a discussion about the complexity of pause thresholds). Angle brackets with BACKSPACE and a number (X) in them shows how the backspace key was pressed X times, for example 26 presses to delete the beginning of the second sentence.

As example of how the log file can be visualised and used for further analysis, Fig. 1 demonstrates a process graph. This illustrates the whole 29-min writing process that led up to the text which began with the sentences shown in the example above. The graph incorporates several aspects that could be of interest to a writing process researcher: pauses, revisions, and the dynamics of the process. The X-axis displays the temporal progression of the writing process, and the right Y-axis shows number of written characters. The blue line represents the total number of written characters, and the green line the number of characters left in the text at any given moment. Thus, the space between them represents number of deleted characters. Approximately 3 min into the writing session, we find the deletion of 26 letters as shown in the linear representation above, here represented by the “dip” in the green line. Pauses are shown as yellow dots and the left Y-axis shows their durations. The longest pause of this writer took place after about ten minutes and was almost 50 seconds long.

Fig. 1

A process graph

2 Main Products and Their Functional Specifications

There are two main types of keystroke logging programs: stand-alone programs with specialized editors for experimental research, and programs which can be used “behind” existing commercial word processers, such as MS Word. The latter type offers a more authentic setting than the first mentioned, but both require special software to be installed in the computer. These programs have to a large extent, but not solely, been restricted to laboratory-based research. Educational institutions, commercial companies etc. may not always be comfortable with—or have IT policies allowing—unfamiliar logging software installed in their computers. However, recently, web-based systems that could easily be deployed without having to install specialized software have started to emerge. See the descriptions of ScriptLog and CyWrite below.

To our knowledge, there are currently five programs available that have been developed for academic use and which are free of charge. While InputLog, ScriptLog and TransLog have been around for long, GenographicX and Cywrite can to a certain extent be viewed as “the new kids on the block” and they do indeed add useful functionality.

InputLog (Leijten & Van Waes, 2013) is probably the most widely known and used keystroke logging program at the moment. It was developed at the University of Antwerp, Belgium by Luuk van Waes and Mariëlle Leijten. InputLog logs data in all Windows applications but is particularly well equipped for analyses of text production in MS Word, which allows writers to format their texts as they would do in an authentic setting. It logs keystrokes, mouse activity, and Windows events, and offers a large range of process analyses, visualisations, and linguistic analysis (for Dutch and English). InputLog can be used with eye tracking but does not synchronize the output of the keystroke logging and the eye tracking automatically. The program also includes a report module specifically aiming to facilitate writing tutors in providing process feedback to their students (Vandermeulen et al., 2020). The analysis module of InputLog can import files from ScriptLog (see below). Researchers who want a more experimental and controlled recording environment, but still wish to have access to the rich variety of pre-programmed analyses offered by InputLog, can combine the two.

ScriptLog (Wengelin et al., 2019) was originally developed as a Mac program at the University of Gothenburg, Sweden, by Sven Strömqvist and colleagues (Strömqvist & Malmsten, 1997). The current version is platform independent (Mac, Windows, Linux) and was developed at Lund University by Johan Frid, Roger Johansson, Victoria Johansson, and Åsa Wengelin. The program offers several experimental settings, such as various elicitation formats (pictures, sounds, texts, the triple task paradigm, and opportunities to manipulate the text already produced during the writing session (e.g. Meulemans et al., 2022). To offer a controlled and replicable writing environment, the program has a simple built-in editor which logs keystrokes and mouse activity and offers full synchronisation with SMI eye trackers. A simple web-based version (without eye tracking synchronisation) is currently being developed. The analysis functionality is more limited in the current ScriptLog version than in previous ones. The program offers real-time playback, step by step playback and fast forward playback. In addition, it produces basic summary statistics, temporal data, and a linear file, but no revision analysis. The developers recommend exporting keystroke data to InputLog (and offers an export function to do so).

TransLog II (Carl, 2012) is a Windows-based program which was originally developed, by Arnt Lykke Jakobsen (Lykke Jakobsen, 1999) at the Copenhagen Business School, Denmark, for translation process research. The current version, which was developed by Arnt Lykke Jakobsen and Michael Carl, can, however, be used for any study of human reading and writing processes. It consists of a supervisor module in which experiments can be designed, and recordings replayed, and a user module in which the experiments are run. Translog records keystrokes and mouse activity and offers synchronisation with both EyeLink and SMI eye trackers. The program has no built-in analysis module but exports several tab-separated summary tables that can easily be processed by various visualization and (statistical) analysis tools.

GenoGraphiX-Log (Usoof et al., 2020) was developed by Gilles Caporossi, HEC Montréal, Canada, Christophe Leblay, University of Turku, Finland, and Hakim Usoof, University of Peradeniya, Sri Lanka. The program functions on both Mac and Windows computers and the idea behind it is to combine text genetics and graph-theory with keystroke logging. The authors state that “[t]he software is intended to be used by teachers, students, researchers and writing professionals.” The recording module uses its own built-in editor and offers a choice of free writing, translation and editing translations. Both keyboard and mouse activity are recorded, but no synchronization with eye tracking is available. The analysis module offers a wide range of visualizations and the possibility to export log data in the form of (.tsv) files into spreadsheet or statistical software for ease of use.

CyWrite (Chukharev-Hudalainen, 2019) was developed by Evgeny Chukharev-Hudilainen, at Iowa State University, US, and differs from all the afore-mentioned tools in two ways. Firstly, the recording module is web-based, and secondly, it aims to include automated writing evaluation (AWE)—although the AWE modules have not yet (spring 2022) been publicly released. Having a web-based recording module is useful, in that no specialized software that could possibly be suspected to be a spy-ware needs to be installed in the computer. Moreover, a web application simplifies preparations for researchers who want to collect larger numbers of data. Like several of the other programs, CyWrite features a simple text editor that captures composition with the possibility to combine keystroke logging and eye tracking. Keystrokes, text changes, and eye fixations are recorded. Keystroke timings are obtained programmatically via event handlers in the JavaScript code running in the user’s web browser. CyWrite’s editor interfaces with a wide range of the eye trackers. The keystroke, text-change, and eye-tracking logs are streamed live to a server-based analysis module where they are analysed and persistently stored. The logged events are then rendered in a post-session viewer offering playback and visualization of the data.

In addition to the above-mentioned programs, all of which are more or less available to the research community—either via the web or by contacting the persons/groups behind them, there are also examples of “in house” software developed by various research groups to solve specific research problems. Perhaps the most well-known example is what in an early stage was called EyeWrite (Simpson & Torrance, 2007). It was originally developed by Mark Torrance at Nottingham Trent University, and Sol Simpson at SR Research. This is not a stand-alone program but rather code that provides a logging/eye tracking editor object within Experiment Builder and PsychoPy. When needed, the researchers/developers adapt the code to their current research questions. Because of this constantly changing functionality, the current Nottingham Trent team have stopped referring to it with a specific name, and just as in TransLog, there is no built-in analysis module. Instead, tab-separated data is generated for analysis in any spreadsheet software or through code written by researchers in Python or R on a needs-basis.

3 Research

As already mentioned, keystroke logging was developed as a research tool to enhance the understanding of the cognitive processes of written language production. It has mainly (but not solely) been used in experimental settings by psycholinguists, and cognitive scientists. Examples of research questions are: What can fluency and pausing in writing tell us about cognitive processes in writing, what are the challenges of writing for writes of different proficiency levels and what strategies do they use to meet those challenges, how does writing processes develop across ages, and how can awareness about writing processes be used to promote writing development. Many of these studies have had university students as their participants, because they are relatively easy to recruit, and usually have above average typing skills. However, such studies have rarely focussed on academic writing per se, but rather recorded short writing sessions (≈30 min) with specific experimental aims. Bowen (2019) is an interesting exception. He used keystroke logging to collect all writing episodes of four academic essays that were part of undergraduate assessment in the English language and literature subject and analyzed the revisions within the framework of Systemic Functional Linguistics (SFL). He could thus demonstrate how lexicogrammatical choice unfolds as text is being written.

There is no prescribed pedagogical use or single answer to what keystroke logging adds to academic writing, which is the focus of this book, but logging of writing can, undoubtedly, be useful in instructional settings, apart from just investigating various aspects of language development—which of course are of indirect pedagogical relevance. Direct pedagogical applications have been developed—first and foremost for observing, understanding, and reflecting upon writing processes—but there are also attempts of using keystroke logging for evaluation and intervention.

As regards observation and reflection, Rijlaarsdam and Couzijn (2000, p. 176) argued that “If students are not even aware of their writing strategies and their results, they can hardly be expected to evaluate—and thus deliberately change, maintain or abandon—them”. Based on the idea that writers need to become aware of various processes of their own writing, but that it wouldn’t be realistic to assume that they will be able to juggle their own writing processes—integrating language, topic, genre, and audience—and learn from them simultaneously, observational learning in writing typically involves the observation of another person (teacher or peer) carrying out a certain writing task (Rijlaarsdam et al., 2008). Keystroke logging can facilitate this by means of its playback function, where a keystroke log can be replayed an infinite number of times, which allows observation not only of other writers but also of one’s own writing processes. Using this to direct students’ attention to their processes, rather than the finally edited text, will enhance their awareness of their own writing (eg. Lindgren et al., 2011). As reflection and verbalization are crucial aspects of observational learning, pedagogical usage of keystroke logging typically involves commenting on and dialogue about the playback as it goes along. The writer is either asked to comment on everything that happens on the screen, or on more specific aspects such as long pauses or revisions. Van Waes, Leijten, Lindgren & Wengelin (2016) suggest that writers’ playback of their own text production output transforms into new input, in terms of Swain’s (2000) output hypothesis. Studies have shown that using keystroke logging as an awareness raiser have proven effective in writing classrooms of both adolescents (Lindgren, 2005; Lindgren & Sullivan, 2003) and adults (Sullivan & Lindgren, 2002), in both first and second language writing (Lindgren et al., 2017), in translation classes (Hansen, 2006; Schrijver et al., 2012), and in professional development (Ehrensberger-Dow & Perrin, 2013; Eherensberger-Dow & Massey, 2013; Perrin, 2013).

For writing teachers, the playback function of keystroke logging can also provide indepth knowledge about their students’ writing strategies, what they struggle with, how they overcome challenges etc., and this may be a key to understanding how to support their writing development. Studies of writers with dyslexia have, for example, shown how their struggling with spelling and avoiding of difficult words influence their writing processes, “stealing” cognitive capacity from higher-level processes, resulting in lower-quality texts (Sumner & Connelly, 2019; Sumner et al., 2013; Wengelin, 2007). And in L2 research Smith (2012) combined keystroke logging with eye tracking to investigate what learners attended to during chat sessions with instructors who used corrective recast. They concluded that playback from these technologies can be valuable in helping to determine which features of the input are likely to be noticed by learners.

Another aspect of observational learning is modelling. Several studies have shown that observing models, either a teacher (Harris et al., 2006) or a peer (Fidalgo et al., 2008; Hillocks, 1986) is a successful strategy in writing instruction. In the classroom a teacher typically models a certain behaviour but using peers has also shown results in academic settings. For instance, Van de Weijer et al. (2019) carried out a short intervention study with small groups of university students with and without hearing impairment, to train their skills in argumentative writing. They found no consistent improvements, but the variation indicated that further research with longer intervention and larger groups would be of great interest.

Recently, researchers have also attempted to bridge the gap between writing process research and pedagogical applications by proposing that keystroke logging could be used for automatic evaluation of students’ engagements in writing processes as well as for writing process interventions. For example, Dux Speltz and Chukharev-Hudilainen (2021) carried out an experimental study with the purpose of increasing fluency in students’ text production by providing automated real-time feedback during the writing process and comparing that to a control condition with no feedback. Participants wrote more text, expressed more ideas, and produced higher-quality texts in the fluency-focused intervention condition. However, the cost of this was—both as shown in their text products, and according to their own statements in a follow-up survey—that they sacrificed accuracy.

4 Implications for Writing Theory and Practice

Keystroke logging was developed as an unintrusive approach to investigating how the writing process unfolds in real time. By analysing typing and its temporal characteristics, such as fluent and less fluent phases, pauses and revisions we can gain some insights into how writers orchestrate the different subprocesses of writing. This is useful not only for cognitive researchers but for writing instructors who are interested in understanding the challenges and strategies of their students. However, to interpret these activities in the light of more specific cognitive processes, additional information is needed, for example from eye tracking or stimulated recall interviews. While this may be viewed as a drawback in experimental research, it could be turned into a useful feature for writing instruction. By letting writing students reflect upon and comment on their own, or their peers’, composition processes, their awareness about writing will increase, and they may possibly be able to improve their strategies. However, more research is needed on this, and on academic writing processes in general. Furthermore, researchers/writing instructors who want to try this should be aware that the available programs have been developed for slightly different purposes, and thus work on different platforms, have different recording functionality, and offer different analyses/playback functions.

5 Tools Table

Software	Original Purpose	Editor type	Access	Platform	License
CyWrite https://github.com/chukharev/cywrite	Writing processes	Built-in NotePad-like	Web-based	Any with a web browser	Free
GenoGraphiX-Log https://www.ggxlog.net/	Writing processes, and translation processes in combination with text genetics	Built-in NotePad-like	Local installation	MacOs, Windows	Free
InputLog https://www.inputlog.net/	Writing processes	MS Word, Google Docs	Local installation	Windows	Free, but you need to obtain a user code from the developers
ScriptLog Contact Johan.Frid@humlab.lu.se	Writing processes	Built-in NotePad-like	Local installation	MacOs, Windows, Linux	Free
TransLog-II https://sites.google.com/site/centretranslationinnovation/translog-ii	Translation Processes	Built-in NotePad-like	Local installation	Windows	Free