Background

The scientific community has witnessed a significant shift in the way research findings are disseminated due to the COVID-19 pandemic and the subsequent rise of preprints [1, 2]. Preprints are early versions of scientific research papers that are made publicly available before they have undergone formal peer review and publication. By circumventing the lengthy peer review process, preprints allow for rapid communication on new evidence to inform public health responses. This is particularly crucial during pandemics. Notably, results of the world’s largest COVID-19 platform trial, RECOVERY [3], were first reported as preprints, enabling swift, real-time evaluation of the interventions and potential harms. While discussing the benefits of preprints in patient care, lead RECOVERY author, Peter Horby, emphasized that peer review delays could have life-threatening consequences [4].

Without formal peer review and rigorous quality control, preprints can amplify misleading information stemming from biases, methodological limitations, incomplete analyses, and even fraud [5]. Preprint use has been scrutinized both from a public understanding perspective and in regards to scientific principles. Firstly, there is a concerning lack of understanding of preprint data among the general public. For example, widespread media attention given to two small, biased preprints that erroneously claimed smoking to be protective against COVID-19 impacted public health as it resulted in a surge in nicotine purchases and smoking uptake in certain countries [6].

Secondly, it is reasonable to expect some discrepancies between the content of various documents and sources for the same randomized controlled trial (RCT), particularly between the preprint and the subsequent journal article, as peer review often impacts the content of a manuscript before it is published. A meta-research study of 139 studies reported in preprint and subsequent journal article or in different versions of the preprint found a change in the abstract’s conclusion in 24% of studies [7]. In contrast, a study of 78 preprint–article pairs of RCTs showed consistency in terms of the completeness of reporting [8]. Another analysis of 67 interventional and observational studies found that preprints and their subsequent journal articles were similar in terms of reporting and spin (i.e., distorted interpretation of results) [9]. Similarly, a study of 74 preprint–article pairs of RCTs showed few important differences in treatment effect estimates between the two documents [10].

To further explore the consistency between various documents reporting the results of trials, we assessed the consistency in effect estimates between preprints and subsequent journal articles of COVID-19 RCTs included in a large living systematic review of COVID-19 pharmacological treatments.

Methods

The protocol is available on Open Science Framework (https://osf.io/hfrp4/?view_only=b06282a8429e4ae1af458f4e372576f7). Here, we report the results of objective one - to assess the consistency in the estimates of treatment effects between preprints and the subsequently published articles. We expanded our sample size by including RCTs assessing all pharmacological treatments instead of limiting our analysis to specific treatment types as planned in the protocol. Additionally, we updated the final search to July 20, 2022.

Data source and search

Our study used the data and methods of the COVID-NMA living systematic review (covid-nma.com) [11] [see Methods S1 in the Additional file]. Briefly, COVID-NMA is a living evidence synthesis and living mapping of RCTs on interventions for the prevention and treatment of COVID-19. The search strategy was modified over time to involve searching only two bibliographic databases: the Epistemonikos L-OVE COVID-19 platform [12] and Cochrane COVID-19 Study Register [13]. The Retraction Watch database [14] was also searched to identify retracted trials and directly remove them from the COVID-NMA review (Additional file Table S1). Screening and data extraction were performed by pairs of researchers, independently and in duplicate, with disagreements resolved by consensus and a third researcher, when necessary.

Eligibility criteria

We selected eligible RCTs in the COVID-NMA living systematic review that evaluated pharmacological treatments for patients with COVID-19 and that were originally posted as preprints and subsequently published in a peer-reviewed journal. The last search date was July 20, 2022. We considered the following COVID-NMA-defined critical outcomes:

  • Clinical improvement at day 28 (D28) defined as a hospital discharge or improvement on the scale used by trialists to evaluate clinical progression and recovery.

  • WHO Clinical Progression Score of level 7 or above (i.e., mechanical ventilation +/– additional organ support or death) (D28).

  • All-cause mortality (D28).

  • Incidence of any adverse events.

  • Incidence of serious adverse events.

We excluded RCTs evaluating preventive interventions (e.g., use of personal protective equipment, movement control strategies), vaccines, non-pharmacological treatments, and supportive treatments for patients admitted to the intensive care unit. We also excluded RCTs that did not report any critical outcome and that reported different analyses in both documents (e.g., interim analysis reported in the preprint and final analysis reported in the journal article).

Linking preprint and subsequent journal article

The linkage between the preprint and journal article was performed as part of the COVID-NMA living systematic review. The preprint–article linker was developed in collaboration with a research team from the French National Centre for Scientific Research. The tool automatically generated an alert when a preprint was updated or published as a journal article. Pairs of researchers used the tool to identify these subsequent reports and then extracted any additional and/or updated data independently, meeting for consensus to reconcile any disagreements. Consequently, an accurate record of the corresponding preprint and journal publication reports in the COVID-NMA database is available for download as a preprint-publication pair. To identify eligible RCTs, one researcher (MD) retrieved this record from the COVID-NMA database and selected the first preprint posted on a preprint server and the subsequent journal article. When available, we used the online publication date in order to calculate the delay between preprint post and journal article publication. Otherwise, we used the print publication date.

Data extraction

We retrieved data that were previously extracted in duplicate independently by pairs of researchers, with consensus to resolve disagreements for the COVID-NMA living systematic review: publication type (preprint, journal article), publication date (date that the report was published online, when available), trial registration (prospective, retrospective relative to the start date of the trial), funding type (industry, mixed, public, none, not reported/unclear), study centers (single, multicentric), setting (hospital, outpatient clinic), geographical RCT location according to the World Bank Country Income Classification [15], and intervention details.

For the critical outcome measures under consideration, the number of participants with outcome events and the number of participants analyzed were retrieved. Risk of bias was assessed according to the Cochrane Risk of Bias 2 tool [16] and each outcome result was rated as ‘Low’, or ‘Some concerns’, or ‘High’ risk of bias. Particularly, we considered the overall risk of bias assessments i.e., the highest risk of bias found in any domain for any critical outcome in the trial. The previously extracted data were split into two parts and two researchers (MD, CG) verified these data, meeting for consensus if a discrepancy was found.

Data synthesis

For the descriptive analysis, frequencies and percentages were calculated for categorical variables, while medians with interquartile ranges (IQRs) were calculated for continuous variables.

We systematically explored whether the number of participants with outcome events, number of participants analyzed, and treatment effect estimates were consistent between preprints and subsequent journal articles for all critical outcomes. The discrepancies between results reported in a preprint and subsequent journal article were classified as (1) change in the estimate of the effect of at least one outcome, (2) change in the direction of the effect, (3) change in statistical significance, and (4) change in the overall conclusion. We also investigated whether the outcomes were deleted or added in the journal articles compared to the preprints. We used R software, [17] with the metafor [18] and forestplot [19] packages, for all analyses.

Results

Of the 49,651 records screened, 1230 were assessed for eligibility and we identified 135 treatment RCTs that were originally posted as a preprint and subsequently published as a journal article. We excluded 26 RCTs because they did not conform to eligibility criteria; one preprint was removed from the preprint server, three RCTs were excluded because there was an error in data retrieval (i.e., they were incorrectly labelled in the COVID-NMA database as a preprint but the data were from trial registry results (n = 2) and from the journal article (n = 1)), three RCTs evaluated non-pharmacological treatments, six RCTs did not report any critical outcomes and 13 RCTs reported interim analysis in the preprint and final analysis in the journal article. Increased sample sizes and longer follow-up and enrolment periods were observed in the final analyses of the subsequent journal articles compared to the interim analyses of the preprints. Overall, 109 RCTs were included in the analysis (Fig. 1).

Fig. 1
figure 1

Flowchart of included randomized controlled trials (last search date July 20, 2022)

Full size image

Characteristics of preprints that were subsequently published in a journal article are presented in Table 1. The median delay between preprint and peer-reviewed journal article was 121 (IQR, 73–187) days. The median sample size was 150 (IQR, 71–464) participants, 76% of RCTs had been prospectively registered, 60% received industry or mixed funding, 72% were multicentric trials. The overall risk of bias assessed was rated as ‘some concern’ for 80% of RCTs.

Of the 109 preprint–article pairs of RCTs, 81 were consistent for all outcomes. We found six RCTs with at least one outcome missing in the journal article, and 14 RCTs with at least one outcome added in the journal article compared to the preprint. There were nine RCTs that had at least one outcome with a change in the number of participants with outcome events or the number of participants analyzed, which yielded a minor change in the estimate of the effect (Fig. 2) [20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37]. There was one RCT with a change in the direction of the effect. No changes in the statistical significance or overall conclusions between preprint and journal article were observed for any RCT.

Fig. 2
figure 2

Discrepancy in effect estimates between preprint and subsequent journal article of COVID-19 RCTs. RCT, randomized controlled trial; n, number of participants with outcome events; N, number of participants analyzed; CI, confidence interval; D28, day 28; article, peer-reviewed journal; WHO-CPS, World Health Organization Clinical Progression Score

Full size image

Characteristics of the preprints that were never published in a peer-reviewed journal are compared to those that were published (Additional file Table S2). Generally, we found that basic characteristics of RCTs initially posted as preprints were similar between those that were subsequently published and those that were not.

Table 1 Characteristics of preprint–article RCTs
Full size table

Discussion

In this study, we analyzed the consistency in treatment effect estimates between RCTs first available as a preprint and subsequently published in a peer-reviewed journal. We found only trivial discrepancies between COVID-19 preprints and subsequent journal articles in most pharmacological treatment RCTs. Nevertheless, some outcomes were added and deleted in the journal articles compared with the preprints and one trial showed a change in the direction of effect between preprint and subsequent journal article.

Our study findings demonstrate substantial agreement with the conclusions of other COVID-19 studies. In a retrospective review of 74 RCTs included in a living network meta-analysis [38, 39] up to August 2021, Zeraatkar et al. did not observe important discordance between the first preprint and subsequent journal article [10]. The cross-sectional study by Bero et al. found only marginal changes to outcomes reporting and spin between 67 preprint–article pairs of studies published between March and October 2020 [9]. In contrast, in a meta-research study of preventive, therapeutic, or post-acute care interventions for COVID-19, Oikonomidi et al. found significant changes in results and abstract’s conclusions in 55% of the sample of 66 preprint–article studies published up to August 2020 [7].

While over half (58%) of preprints are subsequently published in a peer-reviewed journal [40], the fact is that some will remain unpublished, due to journal rejection because of poor methodological and statistical quality or, in rare cases, lack of submission. Based on this, some suggest that preprints should be excluded from meta-analyses [41]. Thus, as part of objective two of our protocol, we conducted a meta-epidemiological study, selecting 37 meta-analyses at different timepoints that included both preprint and journal article RCTs [42].

Strengths and limitations

We assessed the consistency of results between preprint and journal article pairs of RCTs, as significant changes found in the subsequent journal article bring the reliability of preprint data into question. Furthermore, our data were retrieved from a large living systematic review (COVID-NMA). COVID-NMA employed a validated, comprehensive search strategy to identify all relevant evidence.

There are some limitations of our assessment. Firstly, this research was conducted on COVID-19 RCTs, so results may not be generalizable to other fields and study types. In non-COVID-19-related studies, Carneiro et al. [43] determined that preprints were lacking in reporting quality but, on average, the quality of reporting between preprints and subsequent journal articles was comparable. Another study found small differences in journal article conclusions of 7.2% of non-COVID-19–related and 17.2% of COVID-19–related abstracts compared to the preprint [44]. Secondly, for those preprints that were never published in a journal, we could not evaluate whether peer review prevented journal publication due to unsupported conclusions. Nevertheless, we found that trial characteristics were generally similar between preprints that were subsequently published in peer-reviewed journals and those that remained unpublished. Finally, our study is limited to the decisions of the living review. For example, protocol revisions could affect the sample composition.

Conclusion

We identified changes in effect estimates in 8% of COVID-19 randomized controlled trials between preprint and subsequent journal article. Some outcomes were deleted or added in the journal articles; therefore, it is important to retrieve both documents and explore reasons for discrepancies. Certainly, a critical approach should be adopted when using results from preprints due to the lack of peer review.