Background

Evidence indicates that there is a need to improve the quality of peer reviewer reports in biomedical journals [1, 2]. Published biomedical papers may have a direct impact on clinical practice and inform policy. Therefore, it is crucial that peer reviewer reports, a screen before the diffusion of new knowledge, are of the highest quality possible to inform editors’ decision on the fate of the manuscript [3, 4].

Unlike other professional groups, many editors and peer reviewers of biomedical journals operate largely without formal training. It is assumed that having expertise as an author provides, by default, the skills necessary to be a scientific editor and/or peer reviewer. However, this assumption is problematic, potentially having a number of negative implications for the overall quality of biomedical publishing [5].

Alongside the lack of standardised training, the lack of a clear, accepted definition of the roles and tasks of peer reviewers has also been highlighted [6]. A systematic review evaluated the impact of interventions aimed at improving the quality of peer review of randomised controlled trials (RCTs) for biomedical publications. The authors concluded that clarification of the roles and tasks of peer reviewers would be a step forward in quality improvement of peer reviewing [2]. In fact, a recent study showed that the most important tasks in peer review, as perceived by peer reviewers evaluating RCTs, were not congruent with the tasks most often requested by journal editors in their guidelines to reviewers [6].

Organisations such as the Council of Science Editors provide a general overview of reviewer roles and responsibilities [7]. However, within the biomedical field, the roles and tasks of peer reviewers are often closely related to the structural characteristics of the editorial process itself. For example, some (but not all) journals require peer reviewers to assess novelty and/or clinical relevance of articles in addition to assessing scientific rigour. Journal expectations of how a reviewer report should be written may vary. Some journals encourage reviewers to follow a specific structure in their reporting, whereas other journals prefer free text. Additionally, there may be differences between journals’ requests for peer reviewer recommendations regarding whether an article should be accepted for publication or not. Furthermore, differences in roles and tasks between journals may also be linked to the organisational set-up and resources of the journals and publishers. Given these differences, we believe that it is important to distil the core roles and tasks to enable peer reviewers to meet basic, global standards. In order to do this, we first need to compile a comprehensive list of the different roles and tasks described in the literature.

While core competencies for biomedical journal editors have already been systematically identified [8] and agreed upon [9], we are unaware of any body of literature looking into peer reviewers’ roles and tasks.

The aim of this scoping review is to determine the roles and tasks of peer reviewers as depicted in biomedical literature. For the purposes of this research, we consider ‘roles’ to refer to the overarching nature of peer reviewers’ function, whereas ‘tasks’ refer more specifically to actions that fulfil these roles.

Our specific objectives were to answer the following two research questions while summarising the existing literature:

  1. 1.

    What are the expected roles of peer reviewers in the editorial peer review process in biomedical journals?

  2. 2.

    What are the range of tasks that peer reviewers are expected to perform for biomedical journals?

Methods

This scoping review was guided by the methodological framework proposed by Arksey and O’Malley [10], as well as the amendments made to this framework by Levac et al. [11] and by the Joanna Briggs Institute [12]. The framework consists of six consecutive stages: (1) identifying the research question, (2) identifying relevant studies; (3) study selection; (4) charting the data; (5) collating, summarising and reporting results; and (6) consultation. We performed the last stage through qualitative interviews, with results to be reported separately [13]. A study protocol containing all methodological details was published before conducting this scoping review [14]. Although initially specified in the protocol, we did not carry out the review of journal guidelines to peer reviewers. Due to the extensive volume of the initially proposed work, this aspect of the research will be carried out and published separately.

We used the PRISMA-ScR (Preferred Reporting Items for Systematic reviews and Meta-Analyses extension for Scoping Reviews) checklist to report our results (Additional file 6) [15].

Study selection: inclusion and exclusion criteria

Any article with a specific focus and/or statements mentioning roles, tasks and competencies pertaining to the contribution of peer reviewers to the journal editorial process was included. Articles referring solely to roles and tasks that were not related to manuscript peer reviewing in biomedical journals (e.g. grant peer review, professional performance review and peer review of teaching) were excluded. There were no date and language restrictions.

Disciplines

We adopted MEDLINE’s journal selection criteria for our definition of health. This definition includes journals that are ‘predominantly devoted to reporting original investigations in the biomedical and health sciences, including research in the basic sciences; clinical trials of therapeutic agents; effectiveness of diagnostic or therapeutic techniques; or studies relating to the behavioural, epidemiological, or educational aspects of medicine’. In order to ensure feasibility of the study, we did not include journals from the disciplines of psychology, education, physical or natural sciences.

Study designs

The review considered all study designs to be eligible. Based on findings from a preceding scoping review of competencies for scientific editors of biomedical journals [8], it was anticipated that a substantial proportion of relevant statements would be identified in publications that are not only presenting the results of research (subsequently termed ‘research-based publications’) but also in non-research-based publications including book chapters, commentaries and editorials, as well as grey literature. Therefore, we also searched for non-peer-reviewed resources on websites.

Search strategy for peer-reviewed literature

The Peer Review of Electronic Search Strategies (PRESS) 2015 Guideline statement was used to guide the electronic literature search strategies [16]. These were further refined in collaboration with a Health Sciences Librarian. Subsequently, the following databases were searched: Cochrane Library, Cumulative Index to Nursing and Allied Health Literature (CINAHL), Educational Resources Information Centre (ERIC), EMBASE (via Ovid), PsycINFO (via Ovid), MEDLINE (via Ovid), Scopus and Web of Science. There were no date or language restrictions. The search strategy for MEDLINE can be found in the online Additional file 1. In addition, we hand-searched websites of JAMA, Nature and Science using keywords related to peer review to identify any additional literature that was not detected by the search strategy.

Grey literature search

We searched for grey literature on websites of existing networks (e.g. EQUATOR Network, New Frontiers of Peer Review (PEERE)), biomedical journal publishers (e.g. BMJ Publishing Group, Elsevier, Springer Nature, Taylor & Francis, Wiley) and organisations that offer resources for reviewers (including educational courses, for example those provided by Cochrane and Publons). Relevant blogs, newsletters (e.g. The METRICS Research Digest), surveys and reports of authors/reviewer workshops were also searched. We further hand-searched available abstracts from the various International Congresses on Peer Review and Scientific Publication [17, 18].

Screening

Following the execution of the search strategy, the identified records (titles and abstracts) were collated in a reference manager (Endnote) for de-duplication. The final unique set of records was imported into a systematic review paper manager (Covidence) that facilitated independent screening by two reviewers. The screening of titles and abstracts and subsequent full-text screening was performed independently by two reviewers (KG and DC). Disagreements between reviewers were resolved by consensus.

Charting the data

A data extraction form was developed a priori to capture information on each eligible document included in the review. General study characteristics extracted were as follows: first author name, year of publication, country of first author, language of publication and study design. For grey literature, we extracted the URL, title of the document, language of publication and who produced the document.

In addition, for all documents, we collected descriptions of any statements potentially relating to the roles and tasks of peer reviewers. Two people (KG, DC) carried out the data extraction. In the first step, data were extracted from eligible full texts into Microsoft Excel by KG. Subsequently, DC compared the full text of each eligible document with the extracted data on Microsoft Excel to ensure that all relevant information had been included.

Collating, summarising and reporting the results

Initially, all relevant statements (full sentences) related to roles from all data sources were extracted into a Microsoft Excel sheet by KG. Subsequently, each sentence was coded into smaller text units semantically as close as possible to the original, full sentence. Overlapping or duplicate text units were collated following discussion and agreement with DC, resulting in a list of unique statements for roles. Finally, we grouped these statements into emergent overarching themes to provide a better overview of results. All relevant statements (full sentences) related to tasks from all data sources were also extracted into a Microsoft Excel sheet by KG and mapped using pre-defined categories adapted from work carried out by Hirst and Altman [19]. In order to produce a meaningful list, we only included tasks that would apply to all types of studies. Tasks that are not common to all types of studies, for example, those related specifically to RCTs and systematic reviews, were not extracted (Additional files 7 and 8).

Results

Literature search

A total of 23,176 records were returned by the search strategy which included a substantial number of records related to ‘hospital peer review’. In the first step, one researcher (KG) screened all irrelevant records out by title and abstract, leaving 2763 possibly relevant articles which were then screened by two reviewers by title and abstract (KG and DC). Six hundred records were eligible for full-text screening. Disagreements regarding eligibility were resolved through discussion and achieving consensus between the two reviewers. Subsequently, 391 biomedical publications were excluded, leaving 209 publications that met the inclusion criteria (Fig. 1). From these 209 publications, there were 24 original research articles, 45 review articles and 140 book chapters, editorials, commentaries, letters and tutorials. We also included 13 grey literature sources.

Fig. 1
figure 1

Study flow diagram

Full size image

Research-based publications

A total of 24 publications from the database search were considered relevant to the roles and tasks of peer reviewers in biomedical journals (Additional file 2).

Only one of these articles was primarily focused on roles and tasks. Seven studies reported on surveys, eight were descriptive studies and two were randomised controlled trials (RCT). The remaining five articles comprised a randomised trial, two intervention studies and two systematic reviews. Publication dates ranged from 1991 to 2016. Most of the studies were published in 2013 (n = 4) and 2005 (n = 3).

All articles were written in English. Eleven of the studies’ first authors were based in the USA, three in the UK, two in France and one each in Australia, Canada, Hong Kong, India, Italy and Japan. The remaining two studies did not include details on the first authors’ country affiliation.

Non-research-based publications

A total of 185 non-research-based publications were considered relevant.

Overall, 45 publications were review articles (Additional file 3). For the review articles, the date of publication ranged from 1974 to 2016; 2008 (n = 6), 2014 (n = 5) and 2015 (n = 5) were the three years with the most studies. Two review articles were in Spanish and 41 in English. This sample included 23 studies with first authors originating from the USA, five from the UK, four from Australia and Spain and one from Canada, Greece, India, Japan, Korea and Palestine.

The remaining 140 publications consisted of 122 editorials, two book chapters, 12 commentaries, two letters and two tutorials (Additional file 4). The publication dates ranged from 1983 to 2017; 2016 (n = 15) and 2014 (n = 15) were the two years with the most studies. All articles were written in English except for two articles that were written in Portuguese. This sample included 48 studies with first authors based in the USA; 10 in the UK; nine in Canada; seven in Australia; four in Brazil; three in Germany; two in Denmark, India, Portugal and Spain; and one each in Austria, Croatia, The Netherlands and Singapore.

Search of networks and publishers

The search of networks and publishers resulted in 13 additional documents from websites. Among the sample, four documents were blogs/column, five were training/webinar documents and two were guidelines from professional associations, societies and organisations. Lastly, two were guidance documents by publishers (Additional file 5).

Collating and summarising the data

In an effort to create a useful summary of the data, we collated and combined the statements retrieved from all included sources in Table 1 and Table 2, where we present a detailed breakdown of all themes and related statements. Each statement is linked back to the specific papers in the Additional files 2, 3, 4 and 5.

Table 1 Role-related statements (‘roles’ refer to the overarching nature of peer reviewers’ function. The statements are ranked by numerical frequency. Each statement is linked back to the specific papers in the Additional files 2, 3, 4 and 5)
Full size table
Table 2 Task-related statements (‘tasks’ refer to specific actions that fulfil ‘roles’ that refer to the overarching nature of peer reviewers’ function. The statements are ranked by numerical frequency)
Full size table

A total of 1462 statements related to roles were extracted, resulting in 76 unique statements. These were grouped into 13 emergent themes where peer reviewers were considered to be proficient experts in their field (3 items), dutiful/altruistic towards scientific community (7 items), familiar with journal (2 items), unbiased and ethical professionals (18 items), self-critical professionals (4 items), reliable professionals (7 items), skilled critics (15 items), respectful communicators (6 items), gatekeepers (2 items), educators (2 items), advocates for author/editor/reader (3 items) and advisors to editors (2 items). Roles that do not fall within the remit of peer reviewers were also identified (5 items).

The ‘skilled critics’ and ‘unbiased and ethical professionals’ themes appeared most frequently. Figure 2 shows the identified themes according to the number of associated statements, with larger circles denoting a higher number.

Fig. 2
figure 2

Themes related to roles of peer reviewers

Full size image

We also extracted 2026 statements related to peer reviewers’ tasks, resulting in 73 unique statements. These were grouped under six themes: organisation and approach to reviewing (10 items), make general comments (10 items), assess and address content for each section of the manuscript (36 items), address ethical aspects (5 items), assess manuscript presentation (8 items) and provide recommendations (4 items). The themes ‘assess and address content for each section of the manuscript’ had the highest number of statements while the theme related to ethical aspects had the lowest number (Fig. 3).

Fig. 3
figure 3

Themes related to tasks of peer reviewers

Full size image

Discussion

This scoping review produced a comprehensive list of roles and tasks of peer reviewers, derived from a wide range of sources. We sought to complement an existing scoping review on competencies for scientific editors of biomedical journals [8]. While the focus of the scoping review is biomedicine, it is possible that many of the roles and tasks identified could apply more broadly to the discipline of science (e.g. Science, Technology, Engineering and Mathematics (STEM)).

From our analysis, we found incongruities between the position of the peer reviewer and the position of the editor as reported in the literature. For example, the link between peer reviewers’ recommendations and editorial decision-making—where the former typically informs the latter—is often unclear. One of the roles identified in this article is for peer reviewers to keep in mind that they are not decision-makers regarding the ultimate fate of the manuscript. Such decision making is typically made by the editors who ‘Synthesize reviews and make ultimate editorial decisions in light of peer reviewers’ comments and other editors’ comments’ [9], to take this decision. At the same time, a key reviewer task emerging from this scoping review relates to the provision of a recommendation regarding the manuscript (decision-making: reject, accept, etc.). While peer reviewers should be expected to offer advice to editors on the merits of a manuscript, reviewer recommendations around whether or not to publish the manuscript might actually have a more substantial impact on final editorial decision-making than intended, thereby endangering any aspirations of editorial independence. Research indicates that peer reviewer recommendations have a direct influence on editorial decisions [20]. This becomes a problem when the quality of peer reviewer reports is questionable or when one of the many forms of bias that reviewers may display is present. Lee et al. describe bias as a ‘function of author characteristics’ which includes prestige bias, affiliation bias, nationality bias, language bias and gender bias. There might also be ‘bias as a function of reviewer characteristics’, such as when peer reviewers display content-based bias, confirmation bias, conservatism, bias against interdisciplinary research and publication bias [21].

Evidence from the field of meta-research—the study of science itself—indicates that the biomedical literature is replete with low-quality publications [20, 21] which have evidently successfully passed peer review. This in turn suggests that the filtering function of journal editors is not being properly fulfilled. Thus, the question is whether the responsibility for publication of low-quality manuscripts is shared by both peer reviewers and editors or whether it should be borne solely by editors.

Another example of tension between these two stakeholders is the overlap that exists across certain tasks. For example, a key competency of journal editors should be to ‘evaluate the scientific rigor and integrity of manuscripts’ [9]. A study that analysed editorial discourse in a high-impact journal to provide insight into editorial decision-making found that factors related to science—such as research design and methods—were most often addressed during the internal discussion among the editors [22] . Concurrently, peer reviewers are expected to dedicate time to the evaluation and scoring of manuscripts’ scientific rigour and integrity, in what seems to be a duplication of effort for potentially limited impact. Another perspective on this could be that the editor has made an evaluation and is asking the peer reviewers to do likewise, so it could be perceived as a validation of the editor’s views concerning the scientific validity of the submission. It could also be seen as a way of using collective intelligence during the review process. However, peer reviewers are primarily consulted as experts in their field whose knowledge and expertise can at times be broader than the editors’. Thus, the pertinent question here revolves around their authority as experts and whether they are simply rubber-stamping the editors’ decision.

Furthermore, the advocacy role of peer reviewers appeared several times in the included literature. According to some included articles, peer reviewers are variously expected to be advocates for authors, editors and/or readers. Whether this is a justifiable (and feasible) role is open to debate. Is it possible to be an advocate to all stakeholders simultaneously? If not, which stakeholders should take precedence and what would the order of priority be? The term ‘advocates’ needs to be unpacked and clarified in order for peer reviewers to understand what is expected from them in this particular area.

This overlap of roles, and the existence of apparently malleable boundaries between editors and peer reviewers, may have significant implications for the overall peer review process because there is the potential for misunderstandings to occur, as shown by Chauvin et al. [6]. Given the three problematic discrepancies described above, questions remain around their implications for reviewers’ roles and tasks in terms of authority and responsibility.

The 13 themes identified in the list of reviewers’ roles (Table 1) provided a clear construct of the ideal peer reviewer. Box 1 summarises these attributes, based on the most frequent statements identified in our review and the incongruities between the position of the peer reviewer and the position of the editor. Although this is a useful summary of the literature included, these roles need to be discussed critically beyond merely listing them for academic purposes. Instead, a more holistic approach, where these roles are critically discussed within the context of the broader scientific publication system in a way that acknowledges and recognises the complex and dynamic social relations that characterise the peer review process, should be adopted. Box 2 poses questions about the incongruities between the position of the peer reviewer and editor that need to be critically examined.

Complementary to the roles above, we identified 73 unique tasks that peer reviewers may variously be expected to perform. The large number of potential tasks identified is arguably excessive, especially since the majority of peer reviewers are not paid to perform these tasks and often receive little recognition for their work [23]. However, not all journals share all of these expectations. For example, a recent study on content of grading forms across a range of surgical journals found considerable variation in content, with relatively few journals requiring reviewers to address specific components of a manuscript. The study suggests that substantial variation exists in the grading criteria used to evaluate manuscripts submitted for peer review in this field, with a different emphasis placed on certain criteria correlated to journal impact factors. Grading forms of higher impact factor surgical journals more frequently addressed statistical analysis, ethical considerations and conflict of interest, whereas lower impact factor journals more commonly requested qualitative assessments of novelty or originality, scientific validity and scientific importance [24].

We also observed several contradictions. For example, there was some discrepancy as to whether detection of misconduct and fraud should fall within the remit of peer reviewers. However, in order to be able to detect fraud, it is likely that reviewers would need to check and verify the raw data of a study. Besides being impractical, this would almost certainly discourage prospective reviewers from participating in the already time-consuming peer review process. Research suggests that a small portion of the scientific community is already carrying out a disproportionate amount of peer reviewing [25], with the potential of contributing to downgraded peer review standards. Furthermore, journals often have more opportunities to check certain aspects related to misconduct, for example by using software to detect plagiarism. A study that identified ‘highly rated’ competency-related statements for biomedical editors found widespread agreement among editors that identifying and addressing allegations of fraud or plagiarism was a key competency [26] that should be performed by the editor, not by the peer reviewer.

There was also discrepancy regarding whether peer reviewers should engage in copy editing. Although the majority of included articles stated that copy editing does not fall within the duty of peer reviewers, several articles specifically mentioned that reviewers should offer grammatical and linguistic improvements. One could argue that this is the role of the copy editing team members at the journal, who are specifically trained to identify and address such aspects of the manuscript, whereas peer reviewers might not necessarily be sufficiently familiar with linguistic nuances to do so. The time potentially taken up by copy editing is also worth considering. A study found that lack of time is the principal factor for peer reviewers of biomedical journal in their decision to decline a peer review [27]. The time of peer reviewers is precious; therefore, their primary focus should be on the improvement of scientific content rather than the linguistic fine-tuning.

We found variation in the level of detail provided. Certain tasks were vaguely described. For example, statements such as ‘check adequacy of abstract’ or ‘assess manuscript presentation’ were not specific enough in terms of what exactly is required. Such generic statements are not helpful in explaining what editors expect, particularly to new or inexperienced peer reviewers. Vague guidance may result in vague peer review. One simple but straightforward way of addressing this would be to thoroughly review and revise guidance provided to peer reviewers.

The term ‘advocate’ appeared several times in the included literature. According to some included articles, peer reviewers are variously expected to be advocates for authors, editors and/or readers. Whether this is a justifiable role is open to debate. Is it possible to be an advocate to all stakeholders? If not, which stakeholders should take precedence and what would the order of priority be? The term ‘advocates’ needs to be unpacked and clarified in order for peer reviewers to understand what is expected from them in this particular area.

Based on findings from a preceding scoping review of competencies for scientific editors of biomedical journals [8], it was anticipated that a substantial proportion of relevant statements would be identified in grey literature, rather than in peer-reviewed literature. Therefore, sources of grey literature were searched to supplement the database search strategy in the identification of task and role-related statements. Due to the sheer quantity of potentially relevant grey literature available on the web (e.g. websites of publishers), we have taken a pragmatic approach and focused on selected sources from official organisations that deal with peer review and which were also identified in the scoping review of editor competencies [8]. We also included some popular training courses for peer reviewers. However, we recognise that this is by no means comprehensive and we may have missed some potentially useful documents.

We were able to extract data from articles written in English, German, Spanish and Portuguese, but we are aware that the database and grey literature searches may not have included all available relevant literature due to language restrictions. Additionally, despite our best efforts, it is possible that we may have missed some aspects of peer reviewers’ roles and tasks in our search. We preserved the wording used by authors to describe roles and tasks wherever possible and tried to ensure that any changes to wording reflected the spirit of what was being said when editing was necessary. However, it is possible that at times the subjective and selective nature of data extraction may have resulted in occasional misinterpretation of authors’ intended statements. For example, some streamlining was necessary to ensure that the final list of roles and tasks was both manageable and useful; hence, it is possible that subtle differences between tasks or roles might have been smoothened out in an effort to remove redundant or overlapping items. We expect that some missing items will appear in the next stage of our research, where statements will be refined and expanded during a qualitative study with journal editors.

Conclusion

To our knowledge, this scoping review is the first attempt to systematically identify possible roles and tasks of peer reviewers in biomedical journals. This is the counter piece of the existing scoping review on competencies for scientific editors of biomedical journals [8].

As a standalone research piece, this study will primarily be helpful in demonstrating the extent and nature of existing literature on this topic, as well as displaying the type of roles and tasks requested (Additional files 7 and 8). As such, this will be relevant to a variety of audiences, including publishers, editors, peer reviewers and authors. For example, journal editors may be inspired to review their instructions to peer reviewers, whereas course developers might opt to update the content of training courses for peer reviewers.

In addition, a possible training initiative could include the use of ‘open reports’ (i.e. peer review reports and the authors’ responses that are published alongside the relevant articles) which, according to a systematic review (2017) of the definitions of ‘open peer review’, is one of the seven main characteristics of open peer review [28]. These can be used as an educational tool for authors, editors and peer reviewers alike to unpick the different roles and tasks and to encourage a discussion on this subject. The reports can be prepared in such a way that they would reflect the emergent themes that we identified within our scoping review. Potential settings for such an educational intervention could be events such as faculty development meetings at Universities, where authors, editors and peer reviewers often mingle. The different themes could be presented using the concept of snippets [29] which are short, generally limited to 20–30 min. The focus of a snippet is a single overriding communication objective (SOCO). Our identified themes are well suited to be transformed into snippets and can be taught in the allotted time using carefully curated open reports. This review will also inform a subsequent qualitative study with journal editors, with the aim of gaining further insight into their understanding of peer reviewers’ roles and tasks [13] and eventually laying the groundwork for the development of a set of core competencies for peer reviewers of biomedical journals that could then be facilitated through a consensus exercise.