In the era of digital open science, OA journals have mushroomed on the Web. Do these journals provide access to quality research? Does this openness extend to peer review and, if so, how is peer review conducted by these OA journals? In a sting-operation experiment, Science correspondent John Bohannon (2013) found that of the 304 versions of a fabricated paper with flawed research submitted to 304 OA journals, 255 submissions received a decision (the mean for acceptance was 40 days; the mean for rejection was 24 days). Surprisingly, 157 journals accepted a version of the paper. Was this reflected in the peer reviews? Only 36 reviews recognized the paper’s scientific problems whereas “about 60% of the final decisions occurred with no sign of peer review” (p 64). Rupp et al. (2019) concluded “although predatory publishing did not exist ten years ago, today, it represents a major problem in academic publishing” (p 516). There is an “apparent rise in scientific fraud” (Naik 2011) as well as peer review fraud. A “peer review ring” scandal resulted in the retraction of 60 articles at once by a prestigious journal (Barbash 2014). BioMed Central discovered fake peer reviewers involved in 50 manuscripts and took actions to investigate and retract 43 papers (Lawrence 2015). Haven et al. (2019) report from their survey and focus group that “Biomedical researchers and social science researchers were primarily concerned with sloppy science and insufficient supervision. Natural sciences and humanities researchers discussed sloppy reviewing and theft of ideas by reviewers, a form of plagiarism” (Abstract, Results).
The mainstream peer review systems in scientific and scholarly communication typically operate anonymously (Kriegeskorte 2012). This established, blind peer review model for journals has been criticized as being a flawed process (Smith 2006) or a broken system (Belluz et al. 2016). Peer review bias and unfairness exist to varying degrees in different disciplines (Lee et al. 2013; Rath and Wang 2017). Is there a way to restore the trust in peer review for scientific and scholarly publishing? Pioneers and innovators believe that transparency is the key (Fennell et al. 2017).
OPR initiatives and practices
A small number of pioneering journals have been offering forms of OPR since the turn of the century. Launched in 2001, the journal Atmospheric Chemistry and Physics, was among the first OA OPR journals (Pöschl and Koop 2008), along with 36 journals published by BioMed Central (https://www.biomedcentral.com/journals-a-z).
More than 10 years ago, Nature conducted a four-month trial of a hybrid model in which the manuscripts underwent formal closed review by referees and were posted to a preprint site for open review by community readers. The exploratory results showed limited use in improving the process. (Opening up peer review 2007). In January 2016, Nature Communications started a new OPR trial where the authors could decide on a blind or open review model at submission time and have their review reports published upon the acceptance of the manuscript while the reviewers could decide if they would remain anonymous or sign the review reports (Nature 2015). One year into the trial, 60% of the 787 published papers had open reports (Nature 2016). Four years later, Nature announced that it would add eight Nature Research journals to the trial project beginning in February 2020. The announcement reports that in 2018, 70% of the trial journal articles published open reports; 98% of the authors who published their reviewer reports responded they would do so again. Over the four years, 80% of papers had at least one referee named, which seemed to corroborate the results of a 2017 survey of Nature referees: the majority favored experimenting with alternative and more transparent models (Nature 2020).
F1000 beta-tested an open research platform as F1000Research in 2012. Articles submitted to F1000Research are published within 6–14 days and followed by a totally transparent peer review process during which a reviewer’s recommendation and report are published alongside the article. The process was not moderated by an editor. A key difference between post-publication OPR is that F1000Research does not make decisions on acceptance or rejection. Instead, it adopts the algorithm for indexing based on the review results: a minimum of 2 approved or 1 approved plus 2 approved with reservations by reviewers. Another distinct feature is that the review process is totally transparent and open in real-time with both open identities and open reports (https://f1000research.com/for-referees/guidelines).
Choosing a middle ground, PeerJ launched a new optional OPR journal in 2013; as of this writing, 80% of authors have chosen open reports, and 40% of reviewers have signed review reports (https://peerj.com/benefits/review-history-and-peer-review/). Adopting a similar model, the publisher MDPI first announced optional post-publication OPR in 2014 by the journal Life and by 2018 all journals adopted optional OPR. Rittman (2018) reports that 23% of MDPI journal papers published at least one review with open identities. The percentage of the 14 early OPR MDPI journals with open reports include Publications (60%), Dentistry (52%), Medical Sciences (51%), Quantum Beam Science (48%), Life (46%), Brain Sciences (44%), J (43%), Behavioral Sciences (41%), Economies (40%), Cosmetics (39%), Administrative Sciences (38%), Condensed Matter (37%), Animals (34%) and Atoms (33%). EMBO Press reports that currently, 95% of their authors chose to publish review reports alongside their papers (EMBO Press 2020).
Another option for open reports, in addition to appearing alongside the article (e.g., PeerJ) or in a stand-alone volume (e.g., Elsevier), is for reviewers to deposit their review reports to a research partnership service such as Publons.com. Here the decision to publish reports is made by the reviewers rather than the authors or publishers, given that Publons was created to credit reviewers and authenticate their claims. Recently, Wiley partnered with Publons for their OPR initiatives with 40 participating journals (Wiley2018). Wiley’s prestigious journal Clinical Genetics was the pioneering journal for this initiative (Graf 2019). As of March 2020, Wiley added 10 titles in early 2020 to expand this initiative (Moylan 2020).
OPR research
As an innovation in peer review, OPR pursues transparency and openness to improve the process (Wang et al. 2016a, b). Transparency in peer review was rigorously studied by researchers for the journal BMJ in the 1990s before the first journals implemented OPR. These early research examples that studied the effect of making reviewer identities known to authors or posting reviewer names with the paper concluded that these practices had no effect on the quality of the reviews (Godlee et al. 1998; van Rooyen et al. 1999). Walsh et al. (2000) conducted a controlled trial in British Journal of Psychiatry to investigate whether open peer review was feasible. Of the 322 reviewers, 245 (76%) agreed to sign their reviews. A total of 408 unsolicited manuscripts of original research were randomly assigned to the two groups of reviewers. To evaluate the reviews, a seven-item instrument was used to compare the quality of the reviews: importance of research question, originality, methodology, presentation, constructiveness of comments, substantiation of comments, and interpretation of results; in addition, the tone of the review was rated. With cautious notes, the researchers reported that the signed reviews were more courteous and of higher quality than unsigned reviews. Bornmann et al. (2012) compared the reviewer comments of a closed peer review journal and an open peer review journal. They found that the reviewer comments in the open review journal were significantly longer than the reviewer comments in the closed review journal.
Since then, a few studies have investigated author and reviewer attitudes towards OPR, characteristics of open reviews and methods of OPR adoption by existing and new journals. In 2012, Elsevier began a pilot OPR project of selected trial journals (Mehmani and van Rossum 2015). A survey of editors, authors, and reviewers of the five participating trial journals was conducted in 2015 to assess the impact of open review (Mehmani 2016). Forty-five percent of the reviewers revealed their identities. The majority of the reviewers (95%) commented that publishing review reports had no influence on their recommendations. Furthermore, 33% of the editors identified overall improvement in the review quality, and 70% of these editors said that the open review reports were more in-depth and constructive. Only a small proportion of the authors indicated that they would prefer not to publish in open review journals. Mehmani reported high usage of review reports by counting the clicks to the review reports, which indicated the value of open review to the readers.
At a webinar sponsored by Elsevier to discuss how to improve transparency in peer review, Agha (2017) reported on the experience of two Elsevier pilot OPR journals (International Journal of Surgery and Annals of Medicine and Surgery) that published peer reviewer reports as supplemental volumes. He concluded: “60% of the authors like it or like it a lot and 35% are more likely to publish because of it.” Bravo et al. (2019) observed and analyzed Elsevier’s pilot project of five OPR journals from 2015 to 2017. In order to compare referee behavior before and after OPR, the dataset included 9220 submissions and 18,525 reviews from 2010 to 2017. They found “that publishing reviewer reports did not significantly compromise referees’ willingness to review, recommendations, or turn-around time. Younger and non-academic scholars were more willing to accept invitations to review and provided more positive and objective recommendations. Male referees tended to write more constructive reports during the pilot. Only 8.1% of referees agreed to reveal their identity in the published report.” (Abstract). The authors also published review reports alongside their paper. Wang et al. (2016a, b) analyzed the optional OPR journal PeerJ’s publicly available reports for the first three years of the journal (2013–2016). They found that the majority of the papers (74%) published during this time period had open reports; 43% of which had open identities.
If transparency in peer review is the key to tackling the various issues facing the current peer review system, will authors and reviewers embrace OPR? Several large-scale surveys have collected data on attitudes towards OPR with diverse findings. Mulligan et al. (2013) found that only 20% of respondents were in favor of making the identity of the reviewers known to authors of the reviewed manuscripts; 25% of respondents were in favor of publishing signed review reports. In 2016, the OpenAIRE consortium conducted a survey of OPR perceptions and attitudes by inviting respondent participation through social media, distribution lists and publishers’ newsletters. Of the valid 3062 responses, 76% of respondents reported having taken part in an OPR process as an author, reviewer or editor. The survey results show that the respondents are more willing to support open reports (59%) than open identities (31%). The majority of the respondents (74%) believe that reviewers should be given the option to make their identities open. (Ross-Hellauer et al. 2017) Another survey of European researchers conducted by the European Union’s OpenUP Project in 2017 received 976 valid responses. The results of this survey also show that respondents support open reports (39%) more than open identities (29%). This survey also reports a gender difference in supporting open identities (i.e., 35% of female researchers versus 26% of male researchers) (Görögh et al. 2019).
A recent survey by ASAPbio (2018) asked authors and reviewers in the life sciences about their perspectives on OPR. Of the 358 authors, the majority were comfortable (20.67%) or very comfortable (51.96%) with publishing their recent paper’s peer reviews with referees’ names; when asked about the same reviews to be published without referees’ names, the number dropped but still represented the majority: 19.56% were comfortable and 37.71% were very comfortable. Of the 291 reviewers, the majority would be comfortable (32.30%) or very comfortable (40.21%) with posting their last peer review anonymously given the opportunity to remove or redact appraisals or judgments of importance; regarding signing the same review, 28.15% of respondents were comfortable and 32.30% were very comfortable. These results suggest that the majority of the authors are willing to publish their papers’ review reports, with a preference for signed reviews; the majority of the reviewers are willing to have their review reports published without sensitive information, with a preference for anonymity.
The analysis of nearly 2600 responses to Wiley’s 2019 Open Research Survey indicates that the respondents’ preferred peer review models are double-blind (79%), transparent (44%), and single-blind (34%). Twenty-eight percent of the respondents were not aware of the transparent review model (Moylan 2019).
OPR conceptualization and implementation
Despite the growing interest in OPR, there still is no uniform definition of OPR or generally agreed upon best implementation model. Ford (2013) reviewed the literature on the topic to define and characterize OPR. Acknowledging the diverse views of OPR, she states “the process incorporates disclosure of authors’ and reviewers’ identities at some point during an article’s review and publication” (p. 314). She further characterized OPR by openness (i.e., signed review, disclosed review, editor-mediated review, transparent review, and crowd-sourced/public review), and timing (pre-publication, synchronous, and post-publication).
Ross-Hellauer (2017) conducted a systematic literature review and identified seven elements based on 22 definitions of OPR. Of the seven elements, open identities and open reports are considered core elements to recognize OPR journals. The other five elements in the order of frequency of occurrences include open participation, open interaction, open pre-review manuscripts, open final-version commenting, and open platforms/decoupled review. These elements formed a framework for two surveys conducted by OpenAIRE (Ross-Hellauer et al. 2017) and OpenUP (Görögh et al. 2019). Similarly, Tennant et al. (2017) provided a comprehensive review of journals’ peer review practices from the past to the present, which they published in the OPR journal F1000Research. Taking a much broader perspective, they examined the pros and cons of open reviews, including public commentary and staged publishing.
Fresco-Santalla and Hernandez-Perez (2014) illustrated how OPR has been manifested by different journals: open reviews (for all or specific papers), signed reviews (obligatory, pre- or post-publication), readership access to review reports (required or optional) and readership comments (pre- or post- publication). Wang and Tahamtan (2017) identified 155 OPR journals, of which the majority were in medicine and related fields. They also found the various characteristics in the implementations by the OPR journals. According to Tattersall (2015), there were ten leading OPR platforms.