Background

Peer review is considered fundamental to the integration of new research findings into the scientific community [1]. Journal editors rely on the views of independent experts (peer reviewers) in making decisions on the publication of submitted manuscripts [2] and peer review is widely considered necessary for evaluating the credibility of published research. Worldwide, peer review costs an estimated £1.9 billion annually and accounts for around one quarter of the overall costs of scholarly publishing and distribution [3].

Despite this huge investment and widespread acceptance of the peer review process by the scientific community, little is known about its impact on the quality of reporting of published research [4, 5]. Studies have shown that peer reviewers are not able to appropriately detect errors [6,7,8], improve the completeness of reporting [9], or decrease the distortion of the study results [10]. One such study evaluating the impact of peer review found that although peer reviewers often detect important deficiencies in the reporting of the methods and results of randomised trials they miss more omissions than they detect [9]. That study showed that on average peer reviewers requested few changes. Most changes were deemed by the authors to have had a positive impact on reporting of the final publication; for example, clarification of the primary and secondary outcomes or the toning down of conclusions to reflect the results. However, some changes requested by peer reviewers were deemed inappropriate and could have a negative impact on reporting of the final publication, such as the adding of unplanned post hoc additional analyses.

In this study, we investigated the influence of the peer review process within biomedical journals on the reporting of primary outcome(s) and statistical analyses within published reports of randomised trials. In particular, we examined how often relevant post hoc changes of the primary outcome(s) and/or the primary statistical analysis were requested, how often the authors followed these requests, and whether the frequency of these suggestions was related to specific journal or trial characteristics.

Methods

Sample selection

We searched the US National Library of Medicine’s PubMed database over a 1 year period to identify all primary reports of randomised trials published between May 2014 to April 2015 in the six general medical journals and 12 specialty journals with the highest ISI impact factor in 2012. The six general medical journals were the New England Journal of Medicine, Lancet, JAMA, BMJ, PLoS Medicine and Annals of Internal Medicine. The top 12 specialty journals (Table 1) were identified from the major ISI Web of Knowledge journal citation reports medical subject categories. We restricted inclusion to journals that had published at least 50 articles in 2013 with the Publication Type term ‘Randomized Controlled Trial’ (based on a PubMed search on 7 April 2014) (see Additional file 1).

Table 1 Identification of primary reports of randomised controlled trials (RCTs) (May 2014 to April 2015) and number of authors responding to the online survey
Full size table

We developed a search strategy based on a modified version of the Cochrane highly sensitive search strategy (see Additional file 2). The search was run monthly to identify reports of randomised trials published in the previous month (e.g. the search was performed in the third week of June 2014 for all articles published in the print version of each journal in May 2014). One reviewer (SH) screened the titles and abstracts of all retrieved reports to exclude any obvious reports of ineligible trials. A copy of the full article was then obtained for all non-excluded reports, and assessed by two reviewers to determine if it met the inclusion criteria.

Eligibility criteria

We included all primary reports (that is, those reporting the main study outcome) of randomised trials, defined as a prospective study assessing healthcare interventions in human participants who were randomly allocated to study groups. We included all studies of parallel-group, crossover, cluster, factorial and split-body design. We excluded protocols of randomised trials, secondary analyses, systematic reviews, methodological studies, pilot studies and early phase (phase 1) trials.

Online survey

We sent an invitation email (see Additional file 3) to the corresponding author of each eligible report of a randomised trial identified from our search. These authors were asked to participate in an online survey investigating the type of changes made to manuscripts of randomised trials as part of the peer review process. If they agreed to participate, an Internet link included in the invitation email gave them access to the online survey. Participants were asked to confirm that they agreed to participate before being able to complete the online survey. The online survey was tested by members of the study team to ensure comprehension of the survey questions.

The survey consisted of short questions, which asked them to rate on a 1–10 Likert scale the overall handling of their manuscript by the journal, the quality of the peer review process, and whether the editor or peer reviewers had asked them to change any aspects of their study in a way that deviated from what was planned in their trial protocol. They were also asked specific questions about changes to the primary outcome measure(s) and/or the analysis of the primary outcome, additional analyses not included in the original manuscript and whether they had been asked to modify or change their overall conclusions. Authors were asked to comment on the nature of the changes requested in free-text boxes (see Additional file 4).

The invitation emails were sent out in a batch each month, approximately 6 weeks after publication of the journal articles. We inserted the reference to their trial publication in the invitation email to ensure that we could link author responses to individual journal articles. Participants were informed that all responses would be treated in the strictest confidence and that we would not identify any individual responses. A reminder email was sent 2 weeks after the original email was sent, to those who did not respond to the original request. A second reminder was sent 2 weeks after sending the first reminder.

Data extraction and analysis

Key information was extracted from each eligible report of a randomised trial for which the corresponding author had completed the online survey. After piloting of the data extraction form data extraction was carried out by one reviewer (OA); any uncertainties were resolved by discussion with a second reviewer (SH). We extracted information on the trial design, single or multicentre, number of study arms, number of participants, disease area, type of intervention, specification of the primary outcome and direction of the observed effect, trial registration and source of funding. We summarised the characteristics of the primary reports of randomised trials using proportions, median and interquartile range (IQR). Similarly, authors’ responses to the online survey were summarised using proportions, number, mean, standard deviation, median and interquartile range.

The primary analysis focussed on the responses to the online survey and the nature and extent of changes made to manuscripts by authors as part of the peer review process; in particular, in relation to the reporting of the primary outcome(s) and statistical analyses, how often the authors followed peer reviewer requests, and their main motivation for doing so. We assessed whether requested changes or modifications to the primary outcome(s) and statistical analysis were related to the type of journal (general/specialty), type of intervention (drug/non-drug), source of funding (industry/non-industry), or result of the primary outcome (significant/non-significant) using the chi-squared test or Fisher’s exact test as appropriate.

We also carried out a semi-qualitative analysis of free-text information provided by authors on changes and clarifications requested by peer reviewers and the extent to which authors responded to these requests. Two reviewers (CW, KL) first developed preliminary codes based on the questions posed and a screening of the free-text material. The free-text material was then systematically assigned to codes; if necessary codes were revised or new codes added.

Results

The search identified 2106 possible reports of randomised trials published in the top six general medical journals and top 12 specialty journals between May 2014 and April 2015. After screening the full-text articles we identified 893 primary reports of randomised trials, for which 258 (29%) authors completed the online survey (Table 1). The response rate by journal ranged from 11% (Journal of the American College of Cardiology) to 56% (PLoS Medicine).

Characteristics of primary reports of randomised trials

Table 2 shows the general characteristics of the 258 reports of randomised trials whose authors completed the online survey. The majority of trials were multicentre (n = 191; 74%), parallel group (n = 225; 87%), with two study groups (n = 202; 78%). The median sample size was 325 participants (IQR 138 to 1010). About half of the trials assessed drug interventions (n = 127; 49%), 23% (n = 59) surgical or procedural interventions and 16% (n = 40) behavioural or educational interventions. The primary outcome was clearly defined in 92% (n = 238) of trial reports, of which the estimated treatment effect was reported as statistically significant (P < 0.05) in 49% (n = 116). Most trials reported the trial registration number (n = 235; 91%) and around half (n = 141; 55%) gave a journal reference for the trial protocol; more than half (n = 159; 62%) were non-industry funded.

Table 2 Characteristics of primary reports of randomised trials (n = 258)
Full size table

Response to online survey by authors of primary reports of randomised trials

Tables 3 and 4 summarise the authors’ responses to the online survey, and the nature and extent of changes made to manuscripts by authors as part of the peer review process in relation to the reporting of the primary outcome(s) and statistical analyses. The majority of authors responded that they were satisfied with the handling of their manuscript by the journal (mean 8.6, SD 1.5) and the quality of peer review (mean 8.5, SD 1.5). Fourteen percent (n = 36) of authors responded that the editor or peer reviewers asked them to change an aspect of their study in way that deviated from what was planned in the trial protocol.

Table 3 Authors’ responses to online survey (n = 258)
Full size table
Table 4 Type of change or clarification requested to primary outcome and/or statistical analysis
Full size table

Only eight authors (3%) said that the editor or peer reviewers asked them to change or clarify the trial’s primary outcome, but about a quarter (n = 69; 27%) reported that they had been asked to change or clarify the statistical analysis of the primary outcome. Most of those who were asked to change or clarify the trial’s primary outcome or statistical analysis responded that they had fulfilled the request. The main motivation for making the change to the statistical analysis was either to improve the statistical methods (n = 38; 55%) or to avoid rejection of paper (n = 30; 44%). Overall, there was no evidence of association between authors being asked to change or clarify the trial’s primary outcome and/or statistical analysis and whether the trial was published in a general or specialty journal, investigated a drug or was a non-drug intervention, or was solely/partly industry funded versus non-industry funded (Table 5). The primary outcome was more likely to be statistically significant where authors responded that they had been asked to change or clarify the trial’s primary outcome; however, the number of responses was very small (six out of eight responses; 86%). Conversely, requests for changes or clarification of the statistical analysis of the primary outcome were more common (40 out of 69 responses; 60%) for trials with non-significant primary outcomes.

Table 5 Association between whether editors or peer reviewers asked for changes or clarifications to the primary outcome(s) and/or statistical analysis and specific journal and trial characteristics
Full size table

One third (n = 94; 36%) of authors responded that they were asked to include additional analyses. Again, most authors (n = 83; 88%) had complied with the request with the main motivation being to avoid rejection of paper (n = 49; 52%). Most of the published articles (n = 72; 77%) did not indicate that the additional analyses had not been specified in the protocol. Finally, around a quarter (n = 60; 23%) of authors were asked to modify their overall conclusion, in most cases (n = 53; 88%) to provide a more cautious conclusion. Again, the majority fulfilled the request, the main motivations being to improve reporting of the trial (n = 32; 53%) or to avoid rejection of paper (n = 29; 48%).

Textual analysis of author comments to the online survey questions

Examination of the free-text information provided by the eight authors who answered they were asked ‘to change or clarify the trials primary outcome measure(s)’ showed that only two referred to true ‘changes’. One author reported that editors and a peer reviewer “asked us to present just one primary outcome, despite our protocol specifying two” (trial 434); another that a reviewer asked that “we change the primary outcome from average depression severity during time in trial to depression severity at the end of treatment” (trial 451). In both cases authors successfully argued against these requests. Two further authors reported issues which are actually changes to the statistical analysis of the primary outcome (methods for imputation of missing data and adjustment; trials 478 and 584), two were requested to provide non-pre-specified sensitivity or additional analyses (trials 599 and 328). In the two remaining cases the short information provided suggested that the submitted manuscript had insufficiently defined the primary outcome (trials 396 and 513). True changes requested in the statistical analysis of the primary outcome most often concerned addition to or changes of methods to impute missing data (n = 14), or of the statistical model or test (n = 11), and less frequently to analysis populations or adjustment issues. More often, however, editors and reviewers requested clarification of statistical methods or presentations (see Table 6 for examples of free-text responses).

Table 6 Example of free-text answers provided by authors regarding changes or clarifications of the statistical analysis of the primary outcome
Full size table

Free-text responses to the question on additional analyses requested were typically short. The request most often mentioned explicitly (n = 29) was for additional subgroup analyses. A further 20 authors simply said they were asked for additional analyses. A smaller number of requests can be summarised as for sensitivity analyses to check the robustness of findings, or for presentation of additional data. Twenty authors had been asked to draw more cautious conclusions; a further 20 reported a variety of specific changes which did not make conclusions more positive or negative. Four others were asked to draw stronger negative conclusions, one to draw more positive conclusions in a trial not finding expected differences; the remainder did not give a reason.

Discussion

Summary of main findings

This study provides a unique opportunity to investigate the influence of the peer review process within high-impact medical and specialty journals on the reporting of primary outcome(s) and statistical analyses in reports of randomised trials. Textual analysis of author comments from the online survey provides insight into the types of changes or modifications made by authors as part of the peer review process and the rationale behind these changes.

In our study, most authors who took part in the online survey responded positively regarding the overall handling and overall quality of peer review of their manuscript. We found evidence of journal editors or peer reviewers asking authors to change or clarify the trial’s primary outcome, in only eight out of the 258 (3%) published reports where authors responded to the online survey, which is encouraging given concerns regarding bias associated with selective reporting of primary outcomes in favour of significant outcomes [11]. We found some evidence of authors being asked to make changes to the statistical analysis of the primary outcome; however, in most cases these requests were in fact for clarifications of the existing methods, the main motivation being to either improve reporting of the statistical methods or to avoid rejection by the journal. Overall, we found no association between authors being asked to make these changes and the type of journal, intervention, significance of the primary outcome, or funding source. Some authors reported being asked to modify their overall conclusion, in most cases to provide a more cautious conclusion avoiding spin in interpretation of the trial results [10]. However, some changes requested as part of the peer review process were more concerning, whereby authors were asked to include additional analyses, such as subgroup analyses, that had not been included in the original manuscript, of which the majority were not pre-specified in the trial protocol. This is concerning as these additional analyses could be driven by an existing knowledge of the data, or the interests of the reader, rather than the primary focus of the study [12]. While the Consolidated Standards of Reporting Trials (CONSORT) Statement does not preclude additional analyses being performed it does stress the importance of distinguishing those which were pre-specified in the trial protocol and those which are exploratory [13]. It is, therefore, concerning that most of the reported changes to analyses were not reported as post hoc.

Comparison with other studies

We are not aware of other studies specifically looking at the impact of the peer review process on the reporting of the primary outcome and statistical analyses of randomised trials. Other studies on the impact of peer review have predominantly focussed on the editorial process, such as the use of reporting checklists, blinding of peer reviewers [14, 15], or the implementation of training strategies for peer reviewers [16]. A number of studies have looked at the selective reporting of primary outcomes [11] or switching of outcomes from either the trial registry [17,18,19], or trial protocol [20] and the published manuscript. The COMPare study [21] systematically tracked the switching of outcomes in 67 clinical trials published in the top five medical journals (October 2015 to January 2016) to compare trial outcomes reported in the trial registry or protocol with those reported in the published article. Of the 67 trials, they identified 354 pre-specified outcomes that were not reported in the published manuscript and 357 new non-pre-specified outcomes that were silently added. The COMPare study looked at all trial outcomes and did not differentiate between primary and secondary outcomes.

Limitations

Our study has several limitations. First, despite sending two reminder emails our response rate to the online survey was still relatively low at 29% and, therefore, we do not know about those who did not respond. One might expect both that those with more negative experiences might have been more likely to respond to the survey; or that people who experienced a positive spin after peer review were less likely to respond. Yet, our survey showed that most authors were happy with overall quality and handling of their manuscript as part of the peer review progress. Conversely, it is possible that our findings might present an overly positive picture as we focussed on authors whose manuscript had recently been accepted by a high-impact journal. Open peer review, whereby the peer reviewer’s comments are included alongside the published article would allow a more complete assessment of this problem, without the challenge of low response rates.

Due to the nature of our study, we do not have information on the influence of peer review on the reporting of primary outcome(s) and statistical analyses for those manuscripts which were subsequently rejected by the journal. Finally, our study focussed specifically on high-impact journals where one might expect the journal editors and peer reviewers to be more experienced and more likely to identify potential problems. It is unclear the extent to which these findings are generalizable to other journals with potentially less experienced editors and peer reviewers.

Conclusion

Overall, we found little evidence of a negative impact of the peer review process in terms of selective reporting of primary outcome(s). Most changes requested as part of the peer review process resulted in improvements to the trial manuscript, such as improving clarity of the statistical methods and providing more cautious conclusions. However, some requested changes could be deemed inappropriate and could have a negative impact on reporting in the final publication, such as the request to add non-pre-specified additional analyses.