Keywords

1 Overview

Student peer review of drafts in progress had its genesis at the start of the process movement in writing instruction (see Anson, 2013; Crowley, 1998), although the method was used occasionally many decades earlier (Walker, 1917). Broad shifts in theories of learning positioned students not as competitors but potential collaborators, and fears that students would “steal” each other’s ideas or writing were replaced with theories of intertextuality in which texts are inevitably influenced by other texts and ideas (Bazerman, 2004). Research on the cognitive processes of composing pointed to the role of feedback in supporting revision (see Becker, 2006). Early advocacy for peer review argued that the method helps students to improve both their specific projects and also their overall writing ability (Brooke et al., 1994; Nystrand, 2002; Nystrand & Brandt, 1989). In addition, the peers themselves gain competence through the process of reading, commenting on, and, in the group meetings, grappling with textual decisions on behalf of the writer (Bruffee, 1973, 1978, 1984; Elbow & Belanoff, 1989; Flanigan & Menendez, 1980; Hardaway, 1975; Hawkins, 1976; Palincsar & Brown, 1984; Spear, 1988; Topping, 2008; Van Den Berg et al., 2006). Anecdotally, the effectiveness of peer review has met with a range of instructional opinions, but some research has shown that its success depends on careful preparation and orientation of students beforehand (Brammer & Rees, 2007; McGroarty & Zhu, 1997; Min, 2006).

Student peer review began predominantly with the exchange of papers followed by small-group, face-to-face meetings in class (or sometimes in out-of-class meetings) where students discussed their drafts (see Vatalaro, 1990, for several methods). In some cases, especially in content-focused courses across the curriculum, students simply exchanged their papers in class, commented at home, and handed their comments to their peers during the next class. Conferences were supported by peer-review guides and sometimes teacher-led orientations to help students understand what they were supposed to do (Flanigan & Menendez, 1980). Follow-up often included student reflections on the feedback and plans for revision. Early digital technologies such as word processing still required printed copies because there was no other practical way to exchange work. It was not until the arrival of the internet that students could begin to exchange writing and comments online.

Before the development of specific platforms, the earliest uses of technology for peer review involved the digital exchange of papers and comments. Word processing facilitated the provision of intertextual or end comments by peer reviewers, and eventually the exchange of documents was managed online (Dickenson, 1986; MacArthur, 1988; Owston et al., 1992). Since then, cloud-based servers, feedback mechanisms based on machine learning, and markup programs have all enhanced both the practicalities and the learning features of peer review. However, some of the most theoretically important dimensions of peer review, especially the way face-to-face meetings engage students in extensive and helpful discussions and negotiations that lead to fortuitous revision and deeper learning, can be reduced or bypassed online.

2 Core Idea of the Technology

Digital peer review programs have been designed to simplify peer response to students’ work in progress and its transmission. Typically, students upload drafts of papers to the program where they are available for other students, usually in pre-determined pairs or small groups, to read and comment on, sometimes anonymously and sometimes not. The basic principle under which such systems operate is the exchange of drafts and comments to facilitate productive authorial revision. However, because novice writers often lack the ability to diagnose problems or make critically useful suggestions on either their own or others’ work, the systems are usually accompanied by features designed to help students with these processes in order to improve their own learning about text and to assist their peers in revision. In addition to the goal of improving writing, many systems also are geared toward improving the experience of the peer review process itself, especially because this process is fundamental to much writing in collaborative work settings. Digitizing the processes that ordinarily take place physically in classrooms offers opportunities for the inclusion of many such features to be described under functional specifications.

3 Functional Specifications

The digital peer review systems discussed here are those used for educational purposes rather than for professional peer review such as anonymous journal manuscript review.

The most basic digital peer review processes utilize existing programs and platforms to enable students to comment on their peers’ drafts in progress. Most word processing programs offer tools to comment on drafts in the margins, interpose comments using features like MS Word’s Track Changes, and write comments at the end. However, it can be cumbersome for students to work on one draft in a group using word-processing programs because drafts need to be exchanged one student at a time. Different versions of the draft can become confused in transit, and files can be lost. Students can each share their comments on a separate draft, but the author must then consider each in turn, and students cannot comment on each other’s feedback to agree, disagree, or add material to the feedback.

Google Docs, One Drive, and other cloud-based applications allow students to write a draft online and share it (with editing privileges) with their peers, who can then insert comments at various points in the text. Collaboration is one advantage of this method because students can all comment on a draft (even simultaneously in real time), and each peer’s comments are differentiated. The author can then easily accommodate the comments during revision and remove them, yielding a clean text. However, unlike face-to-face peer review, it is difficult to have a conversation about the draft or the suggestions. Students must be shown how to avoid privacy breaches and also how to work with the files when they want to pull them off the platform.

Some more sophisticated peer review systems are built into existing LMSs. These systems have additional functionalities to augment the simple exchange of commentary. Teachers can include rubrics to guide students’ analysis and commentary and create small groups for the review, and students can make both marginal and summative comments and use markup tools to highlight words, paragraphs, or sections of a draft. Because the systems are built into their LMS, students often find it convenient to access and use them. Some LMSs such as Moodle and Google Classroom offer ways for instructors to acknowledge students’ peer reviews and build them into a grading scheme based on posted criteria.

Freestanding peer review systems can be adopted to facilitate a number of interactions around drafts in progress. Some systems include features that allow students to create drafts of papers based on smaller writing tasks and upload material in several formats. Peers can comment on these progressively drafted pre-writing materials along the way, helping the writer to shape a text even before it has been developed into a full rough draft. Instructors can include checklists or prompts for students to consider. As students submit peer comments, instructors can also intervene and help to shape the responses effectively. Authors can also give feedback to the peer reviewers, assessing whether the feedback was helpful, well presented, etc., in a process called “back evaluation.” More robust systems also offer analytics that show, for example, the extent to which students’ drafts improve, how effective the feedback was, and what sorts of plans students made for revision.

Training-based peer review systems are designed to help students to provide effective comments on peers’ papers by including a review feedback component. Typically, students read and comment on preexisting papers using guided questions tied to evaluation criteria, and then score the papers on a scale. Automated feedback lets the students know how well they have done with the trial peer reviews. This prepares the students for the review of their peers’ submissions (often but not always anonymously). If the same rubric is used, the quality of students’ reviews can also be determined, and grades can be assigned based on the system’s determination. Some systems allow students to later compare their own reviews with those of the other group members to see what they missed (or what their peers missed). The educational goal behind such systems aims to improve not only the individual peers’ papers but the reviewer’s ability to critically read work in progress and provide insightful and helpful feedback. Instructors can manipulate the system through the provision of assignments, questions, and evaluation rubrics.

Currently, training-based peer review systems are being augmented with machine-learning capabilities (see Lin et al., 2018). For example, specific kinds of comments can be assigned labels based on whether the comment makes a suggestion, identifies an error, or points to a specific place in the text. Labels can be determined by trained raters until agreement is reached, and scores can be assigned to labels based on the accuracy of their characteristics. This label information is used as inputs to create an algorithm that can determine scores for subsequent student reviews. Feedback to reviewers from the system can prompt them to improve their comments. The system can also assign grades for the quality of peer review comments at the end of the cycle. Continuous data input improves the accuracy of the system; as reviewers are assigned scores, the system can “learn” who the most competent reviewers are and add features of their reviews to its database (see Leijen, 2014).

Some platforms enable peer review of multimodal texts such as video. PlayPosit, for example, has functions that allow a student to upload or connect to a video. Students review the uploaded material and then are prompted to offer comments and suggestions. The system is interactive so that multiple reviewers can see and respond to each other’s comments as if in conversation. Rubrics and criteria can be included.

4 Main Products

URLs and other information about the systems are located in the table at the end of this chapter.

Among the simpler forms of digital support for peer review are Google Docs or word processing programs that facilitate extensive commenting, such as MS Word. Google Classroom primarily facilitates the sharing of files and comments, allowing for the distribution of assignments, which invites students to engage in peer review. Students can insert comments into text boxes. Most LMSs include some form of peer review support. Moodle, for example, has an add-on to its Assignments function that organizes peer reviews and provides various metrics. Canvas includes a peer review option in which students open a peer’s paper within the LMS and provide comments. Students can access both peer and instructor comments. Instructors can include rubrics for students to rate certain aspects of their peers’ papers. Comment buttons allow students to add marginal feedback at certain points in the paper. Other tools facilitate drawing, highlighting, or striking out text. General comments (not tied to specific parts of the text) can also be added, and files can be attached.

Freestanding peer review systems available for classroom use include Peer Studio, a free cloud-based tool developed at Stanford University and the University of California at San Diego that relies on instructor rubrics. The system also allows students to consider a peer review against a comparison submission to make judgments for revision. The comparison feature is generated by AI-based, machine-learning technology that analyzes each student’s reviewing history and that of their classmates to identify optimal comparison submissions. PeerGrade, a subscription-based program, works in a similar way: the instructor sets up an assignment, the students upload papers, each peer can write comments in the margins and also communicate with each other about the comments, and the teacher gets a complete overview at the end of the process. iPeer is an open-source tool that manages the peer review process. Among its affordances are the provision of rubrics, a way to review student comments before releasing them, a progress reporting form, and export functions in different formats.

Among the peer review systems that include training, analytics components, and machine learning, Kritik offers instructors a way to customize rubrics and provides a feedback mechanism through which students rate the effectiveness of their peers’ comments. A gamified reward system provides incentive. SWoRD (which stands for Scaffolded Writing and Rewriting in the Discipline) was developed by researchers at the University of Pittsburgh and has been used not only to support student peer review but also to research the effects of peer review on students’ learning and writing processes. When a student uploads a paper to the system, it assigns four to six peer reviewers automatically. Instructors can include evaluation prompts or rubrics designed themselves or pulled from a shared library. Instructors can also share their prompts and rubrics with specific collaborators. The student revises the paper based on the peers’ comments and resubmits it, after which the same reviewers examine the revised version. The algorithm in the system analyzes the peers’ ratings for agreement and bias, and the author rates the reviews for their helpfulness. Reviewers are assigned grades based partly on how accurate they are and partly on how helpful they are. Because the system relies on machine learning, instructors don’t need to determine the effectiveness of the reviews. Students are therefore trained in peer review while authors receive feedback for revision. Analytics built into the system provide information for teachers to improve their assignments and instruction. SWoRD is now licensed by Panther Learning and is called Peerceptiv.

MyReviewers (recently renamed USF Writes), developed and housed at the University of South Florida, includes the usual document management tools in most peer-review software but also offers workflow management for students, instructors, and administrators and a number of resources. Drafts are uploaded to a cloud-based storage space. Teachers and/or students can then use various markup tools to highlight text, add sticky notes, insert text boxes, or enter links; preloaded rubrics automatically calculate weighted scores on the draft. The system has built-in learning analytics, such as inter-rater agreement, that can assess student performance.

Expertiza, developed and housed at North Carolina State University, is a peer review system incorporating machine-learning features for review assessment. Students are assigned rubrics to review the work of peers who have uploaded files; both author and peers are anonymous. Questions on the rubric (which guides the students’ evaluations) can be assigned scores and students can add comments. Student authors can consider aggregated advice and scores as they rethink their drafts; they also use another rubric to rate the quality of their peers’ feedback (such as how helpful, accurate, and respectful the advice is). Instructors can include the scores from this feedback in their assessment of students’ work. The system allows more than one iteration of feedback and response. Currently Expertiza is incorporating machine learning to collect data on the quality of peer reviews by assigning “reputations” to certain reviewers and using the reputations to estimate reviewer reliability. The algorithms in the system compare a student’s reviews with the scores of other reviewers of the same text, as well as distributions of scores and how lenient or strict a particular reviewer is. Another feature of Expertiza is the peer review of learning objects in different disciplines. Students produce reusable materials such as discussions of difficult concepts or animated lecture slides. These objects are submitted for peer review, resulting in the highest-rated objects being incorporated into instructional materials for the class or in presentations.

Eli Review, developed at Michigan State University, is an online peer review platform that allows instructors to create prompts, organize reviews, and design checklists in order to realize specific learning goals. They can also watch reviews in real time as students provide each other with comments, then make decisions about further coaching. Students can nominate specific peer reviews as models for other students to follow. Iterative stages allow students to plan revisions in their work based on the reviews. Analytics show improvements in drafts, quality of feedback, and both plans for revisions and the revisions themselves.

Calibrated Peer Review (CPR) was developed at UCLA by Dr. Orvill Chapman with the assistance of funding from the (U.S.) National Science Foundation and the Howard Hughes Medical Institute. Like other training-based peer review systems, CPR involves anonymous peer review of student work and provides feedback to reviewers about their reviews. In the pre-review training component, students write brief essays on provided prompts with supporting questions. After submitting their essays, the students then score three “calibration” essays in order to practice peer review, followed by scoring three of their peers’ essays. In the final step, students return to their own essays, which they re-read and score. Instructors can create their own assignments or use one in the system’s databank. A full description of the program, applied to an introductory biology class, can be found in Robinson (2001).

5 Research

Research on peer review, in both L1 and L2 contexts, is voluminous. It is not the purpose of this chapter to review research on the more general use of peer review but to cite studies that focus specifically on the use of digital technologies to facilitate the student peer review process.

Many studies have been conducted by the developers of specific peer-review platforms. Previous research on SWoRD is summarized in Schunn (2016) and individual studies mentioned there will not be cited here. Some studies showed agreement between student ratings and instructor ratings; in one study, student ratings in Advanced Placement classes using SWoRD were similar to the ratings of experts trained by the College Board to evaluate essays. Other studies showed improvements in student writing as a result of peer evaluation. In studies that masked the origin of feedback, students’ ratings of helpfulness of peer feedback were similar to their ratings of instructor feedback. Other studies found that students improved their own writing as a result of providing feedback to their peers.

Research was conducted at various stages in the development of then-named MyReviewers. Warnsby et al. (2018) examined the role of praise, criticism, authority, and power relations in a corpus analysis of 50,000 peer reviews curated through MyReviewers across multiple institutional contexts. Results revealed a mix of functions, and also that students used more positively glossed feedback than negatively glossed. Results also varied by institutional context and other factors such as writer experience. In a comparison study of 46,689 peer and 30,377 instructor reviews submitted to MyReviewers, Moxley and Eubanks (2015) found that student ratings were higher than those of instructors but that this difference declined over time. In addition, higher-scoring students on instructor ratings gave scores on peers’ paper that were closer to those of instructors. The most recent studies of MyReviewers emerged from a multi-institutional grant from the (U.S.) National Science Foundation. In one study (Anson et al., 2021), corpus analysis was employed to compare key concepts used in peer reviews in foundational writing courses with courses in the disciplines, in part to see whether these terms transferred across contexts. Although some transfer occurred, it did so less effectively when instructors in the disciplines did not use the rubric function of MyReviewers to reactivate the knowledge gained in the foundational course.

Ramachandran and Gehringer (2011) demonstrated that automated classification of peer reviews in Expertiza based on quality and tone is possible when applied to student review data. Metareviewing (reviewing reviews) is tedious for instructors and problematic for students (who may not have been trained to evaluate the quality of reviews), but machine learning techniques such as latent semantic analysis can automate the process. The researchers report on experiments showing that certain programming steps can predict metareview scores of students’ reviews, thus providing valuable automated feedback.

A study of Calibrated Peer Review used in a lecture-based zoology course (Walvoord et al., 2008) found that the system assigned higher scores to student essays than did instructors. In addition, students’ abilities to write technical material or summarize scientific articles with full understanding of the material improved over the course as a result of peer review. In addition, the system reduced the time instructors spent grading student work. However, an interview-based article published in UCLA’s Daily Bruin (Rosenbluth & Lewis-Simó, 2018) reports that students using the version at the time did not find it to be very effective.

6 Implications

Compared with conventional student peer review, digital peer review offers several advantages, the first being speed and convenience of exchange. Students can usually manipulate text for font size or other aspects of readability, which is impossible on paper, to accommodate learning differences and visual impairment. There is usually unlimited room for feedback which is not the case with paper copies, and from a psychological perspective, comments can be provided in different ways (such as through collapsed boxes or stickies) to avoid overwhelming a student writer with heavily marked up text. Built-in rubrics, feedback, and training systems can, unlike conventional review, compel students to consider certain questions or to learn how successful their commentary is based on machine-learning feedback algorithms. From an instructional perspective, the management of peer review can be greatly facilitated by providing access from a single online portal, by auto-generating quality of feedback scores, and by tying feedback to grading systems. Because texts and comments typically reside in safe spaces for backup and retrieval, material is not lost.

Digital peer review also has the advantage of generating data that can be useful to individual instructors as well as in classroom research, action research, or formal studies. Curation of student peer reviews and revisions can, over time, yield enormous data sets that can be digitally analyzed for more robust findings than the mostly classroom-limited and small-group studies of peer review in the past. In addition to the reports that some of the systems can generate, raw data in the form of student reviews, use of stickies or prepared comment banks, and drafts and revisions can be analyzed for many features including the relationship between student reviews and specific kinds of revision, the affective dimensions of peer feedback, and comparisons of student and teacher feedback.

As stated previously, existing digital peer review systems do not allow for the kind of deep discussion of drafts, with live consideration and negotiation of possible revision, that are the hallmark of conventional face-to-face meetings. In addition, many peer review systems push students toward an evaluative stance (e.g., by asking them to make grade-based or other decisions about the quality of a draft), rather than encouraging a more formative, advisory stance even with the provision of rubrics. The use of rubrics with a few questions can direct students’ attention to salient issues but also reduce more impressionist and holistic responses to student texts, or create a “checklist” mentality to what is usually a complex response involving interpretation of multiple textual dimensions. Finally, depending on the system, confidentiality and privacy can be a concern when files are transmitted, shared, stored, and returned.

Because peer review involves complex interpersonal and affective dimensions of collaboration and response in both face-to-face and online settings, more research is needed to explore which aspects of peer review are effective relative to students’ perceptions. For example, research has shown that when students become preoccupied with the interpersonal nature of response to their work, they can be diverted from a focus on their texts and how to improve them (see Anson, forthcoming). These affective and interpersonal responses can be influenced by perceptions of difference, including students’ identity constructs, racial and ethnic characteristics, and other factors.

7 List of Tools

Software

Description

URL

Google Docs

Peer review function; free; Web-based; share function for others’ feedback; specifies reviewer; can reveal changes

https://www.google.com/docs

MS Word

Peer review function; proprietary; Web and download; individual feedback using tools such as Insert Comments

Microsoft.com

Google Classroom

Peer review function; proprietary; Web-based; full suite of functions including peer review

edu.google.com

Moodle

Peer review function; freemium; Web-based; LMS includes peer review function in Assignments; review metrics & analysis; reusable comments; can flag poor reviews

Moodle.org

Canvas

Peer review function; proprietary; Web-based; LMS includes peer review function; can assign or randomize groups; guides process; can include rubrics and feedback tools

https://www.instructure.com/canvas

Peer Studio

Freestanding peer review platform; free; Web-based; automates many functions; uses instructor-provided rubrics; AI-based review analysis; guides process

https://www.peerstudio.org/

Peer Grade

Freestanding peer review platform; free trial then student fee; Web-based; teacher provides assignment and rubric; students provide review & engage in discussions of it; provides teachers with overviews

https://www.peergrade.io/

iPeer

Freestanding peer review platform; open source; Web-based; assignment & rubric creation; reminders & scheduling; student feedback system; can be team-based

https://ipeer.ctlt.ubc.ca/

Kritik

Freestanding peer review platform; proprietary; Web-based; LMS integration; includes training, machine learning, & analytics; students rate reviews; gamified reward system

https://www.kritik.io/

SWoRD (Peerceptiv)

Freestanding peer review platform; proprietary; Web-based; coordinates anonymous peer review; offers rubric options; students provide back evaluations; includes analytic features

https://peerceptiv.com/

MyReviewers (USF Writes)

Freestanding peer review platform; student subscription; Web-based; Workflow management, markup tools, rubrics, automated score calculation and learning analytics

myreviewers.usf.edu

Expertiza

Freestanding peer review platform; open source (request account); Web-based; rubrics; automated scoring & aggregated advice; machine learning & reputation ranking

https://expertiza.ncsu.edu/

Eli Review

Freestanding peer review platform; free trial then student subscription; Web-based; create reviews, checklists, & prompts; manages review process; allows revision plans; provides some analytics

https://elireview.com/

Calibrated Peer Review

Freestanding peer review platform; proprietary; Web-based; assignment library; training calibration; anonymous review; scores produced for writer & reviewers

http://cpr.molsci.ucla.edu/Home