Advertisement

BMC Medicine

, 16:87 | Cite as

Avoidable waste of research related to outcome planning and reporting in clinical trials

  • Youri Yordanov
  • Agnes Dechartres
  • Ignacio Atal
  • Viet-Thi Tran
  • Isabelle Boutron
  • Perrine Crequit
  • Philippe Ravaud
Open Access
Research article

Abstract

Background

Inadequate planning, selective reporting, and incomplete reporting of outcomes in randomized controlled trials (RCTs) contribute to the problem of waste of research. We aimed to describe such a waste and to examine to what extent this waste could be avoided.

Methods

This research-on-research study was based on RCTs included in Cochrane reviews with a summary of findings (SoF) table. We considered the outcomes reported in the SoF tables as surrogates for important outcomes for patients and other decision makers. We used a three-step approach. (1) First, in each review, we identified, for each important outcome, RCTs that were excluded from the corresponding meta-analysis. (2) Then, for these RCTs, we systematically searched for registrations and protocols to distinguish between inadequate planning (an important outcome was not reported in registries or protocols), selective reporting (an important outcome was reported in registries or protocols but not in publications), and incomplete reporting (an important outcome was incompletely reported in publications). (3) Finally, we assessed, with the consensus of five experts, the feasibility and cost of measuring the important outcomes that were not planned. We considered inadequately planned or selectively or incompletely reported important outcomes as avoidable waste if the outcome could have been easily measured at no additional cost based on expert evaluation.

Results

Of the 2711 RCTs included in the main comparison of 290 reviews, 2115 (78%) were excluded from at least one meta-analysis of important outcomes. Every trial contributed to 55%, on average, of the meta-analyses of important outcomes. Of the 310 RCTs published in 2010 or later, 156 were registered. Inadequate planning affected 79% of these RCTs, whereas incomplete and selective reporting affected 41% and 15%, respectively. For 63% of RCTs, we found at least one missing important outcome for which the waste was avoidable and for 30%, the waste was avoidable for all important outcomes.

Conclusions

Most of the RCTs included in our sample did not contribute to all the important outcomes in meta-analyses, mostly because of inadequate planning or incomplete reporting. A large part of this waste of research seemed to be avoidable.

Keywords

Randomized controlled trial Waste of research Outcome Selective reporting Core outcome set 

Abbreviations

CI

Confidence interval

COMET

Core Outcome Measures in Effectiveness Trials

COS

Core outcome sets

ICTRP

International Clinical Trials Registry Platform

RCT

Randomized controlled trial

SoF

Summary of findings

WHO

World Health Organization

Background

Clinical trials are only as credible as their outcomes, so to inform decision-making appropriately, randomized controlled trials (RCTs) must evaluate the outcomes that are most important to patients and their caregivers [1, 2, 3, 4, 5, 6, 7, 8]. Failure to do so could contribute to the overwhelming problem of waste in research [9, 10, 11, 12, 13, 14, 15, 16]. Waste of research related to inadequate outcome planning, selective reporting, and incomplete reporting of outcomes in RCTs prevents patients and their physicians from making well-informed decisions, with potential serious consequences if ineffective or harmful treatments are promoted [9, 12, 13, 14, 17, 18].

Therefore, when planning an RCT, researchers are expected to measure all important outcomes [9]. However, previous studies found that less than one-fifth of diabetes RCTs and less than one-quarter of cardiovascular trials considered patient-important outcomes as their primary outcomes [19, 20]. Instead, researchers frequently rely on surrogates as a proxy for final patient-important outcomes because these outcomes allow for smaller, faster, and thus, cheaper clinical trials [21, 22, 23, 24]. However, surrogates can be misleading, because they may show exaggerated treatment effect sizes or even an apparent benefit of harmful treatments, as was the case for the use of antiarrhythmic drugs after myocardial infarction, which led to the deaths of several thousand patients decades ago [25, 26, 27, 28].

Selective reporting has been repeatedly described as another important issue affecting RCT outcomes [29, 30, 31, 32]. Outcome reporting bias arises when outcomes are selectively reported based on the nature and direction of the results [33]. In a recent study, the median proportion of RCTs with discrepancies between registered and published primary outcomes was 31% [34]. Statistically significant outcomes were 2 to 4 times more likely to be reported in publications than non-significant ones, which biases the available body of evidence toward more positive results [30].

Similarly, a median of 31% to 50% of efficacy outcomes were found to be incompletely reported in RCT articles [29, 35, 36]. With incomplete reporting of outcomes, outcomes cannot be included in meta-analyses, which poses a serious threat to the usability of trial results: not including all available results in meta-analyses can lead to a truncated vision of the overall body of evidence.

The purpose of this study was to describe the waste of research related to inadequate planning, selective reporting, or incomplete reporting of outcomes in RCTs and to examine to what extent this waste could be avoided. For this, we addressed the following specific questions: (1) What proportion of RCTs are excluded from meta-analyses due to outcome reasons? (2) Were the exclusions related to inadequate planning, selective reporting, or incomplete reporting? (3) Was it feasible to measure the missing outcomes at the planning stage, and at what cost?

Methods

We performed a research-on-research study based on RCTs included in Cochrane systematic reviews. We used the outcomes reported in the summary of findings (SoF) tables of reviews as surrogates for important outcomes, because Cochrane systematic review SoF tables should include the most important outcomes for patients and other decision makers, whether they are available in RCTs or not [7]. This study used a three-step approach. (1) First, for all important outcomes, we identified the RCTs that were included in the Cochrane reviews but excluded from the corresponding meta-analyses because the important outcome was missing. (2) Then, we systematically searched for trial registrations and protocols to distinguish between the outcomes that were not planned (not included in registries or protocols, i.e., inadequate planning) and outcomes that were planned. For the planned outcomes, we distinguished between those that were adequately reported and those that were incompletely reported (poor reporting) or not reported (selective reporting). (3) Finally, we assessed, via expert consensus, the feasibility and cost of measuring the outcomes that were not planned.

The eligibility criteria at each phase of the study are summarized in Additional file 1.

Identification of RCTs excluded from meta-analyses

Data sources

We obtained data from all Cochrane systematic reviews published between March 2011 and September 2014. They were provided by the Cochrane Collaboration editorial unit as XML files and contained all information reported by the review authors in RevMan, the software developed by the Cochrane Collaboration for preparing and maintaining systematic reviews [37]. Cochrane systematic reviews of interventions are organized by comparisons of two treatment groups. Meta-analyses are then performed for one or more outcomes within each comparison. Cochrane reviewers are encouraged to present an SoF table summarizing information on the quality of evidence and treatment effect magnitude (from the meta-analysis result) for the most important outcomes (see example in the review by Mocellin and colleagues [38]) [7]. According to the Cochrane collaboration, these outcomes should be important to all research end users, that is, patients and other decision makers [7]. These outcomes include a “wide variety of events such as mortality and major morbidity (such as stroke and myocardial infarction); however, they may also represent frequent minor and rare major side effects, symptoms and quality of life, burdens associated with treatment, and resource issues (costs)” [7].

Selection of relevant Cochrane systematic reviews

Using R 3.1.1 and the XML package, one of us (IA) removed from the set of reviews provided by the Cochrane Collaboration withdrawn Cochrane reviews and then identified all reviews of RCTs with an SoF table. We excluded reviews that included observational studies and those including only RCTs published before 2007 because we focused on recent RCTs. From all eligible Cochrane reviews, using a random number generator, we drew a random sample of 300 for an in-depth evaluation but excluded a further 10 reviews because their SoF tables mixed various interventions, which resulted in a final sample of 290 Cochrane reviews.

Identification of important outcomes

We considered outcomes reported in the SoF table as a surrogate for important outcomes. Therefore, we manually identified and extracted all outcomes reported in the SoF tables. Most reviews had a single SoF table, but some had several tables, corresponding to different comparisons. In this case, we focused on the main comparison as acknowledged by the authors. If the main comparison was not reported by the review authors or if various comparisons were reported in the same SoF table, we selected the comparison with the most outcomes and included the largest number of RCTs. If the SoF table reported various interventions (e.g., presented three meta-analyses of different outcomes for three different interventions), the review was excluded.

We classified each outcome as follows: mortality, other clinical event (e.g., myocardial infarction or stroke), therapeutic decision (e.g., transfusion), function (e.g., disability), pain, quality of life, adverse events or side effects (identified as such by the review authors), physiological variable (e.g., blood pressure or weight), biological variable (e.g., cholesterol levels), radiological variable (e.g., measure of joint space), compliance (e.g., discontinuation for any reason), process (e.g., duration of surgical procedure), resource use (hospitalization), cost-effectiveness, and satisfaction with care [39].

Identification of RCTs excluded from meta-analyses

For each review and for each important outcome, we identified all RCTs excluded from the outcome in the corresponding meta-analysis. To do so, by using the reference list of all studies included in the Cochrane review, we first identified all RCTs available for the selected comparison by extracting all RCTs included in any meta-analysis reported for this comparison. Then, for each of these RCTs, we manually evaluated whether they contributed to the meta-analysis of each important outcome by screening RCTs included in the corresponding meta-analysis.

Evaluation of the reason for a missing outcome

We a priori hypothesized that the exclusion of an RCT from a meta-analysis for outcome reasons could be related to inadequate planning (i.e., the outcome was not planned to be measured), selective reporting (i.e., the outcome was planned but not reported in the publication), or incomplete reporting (i.e., the outcome was reported in the publication but not in a way that allowed for pooling of data).

To distinguish among inadequate planning, selective reporting, or incomplete reporting, for each RCT excluded from at least one meta-analysis, we screened the data available in Cochrane reviews and systematically searched for trial registration and/or protocols.

We focused on RCTs published in 2010 or later to maximize the chance of identifying trial registration or protocols.

Search for RCT registration and protocols

For each RCT, we individually assessed all available reports and information. To do that, we extracted the references identified from the “References to studies included in this review” section of the Cochrane reviews. One of us (YY) retrieved all the articles, conference abstracts, reports etc. related to the identified RCT. We screened all articles for any information regarding registration (name of trial registry and/or registration number) and/or protocol availability. When no reference to a trial registration was found, we used the following approach:
  1. 1.

    We searched the full text of the Cochrane review for any information regarding trial registration.

     
  2. 2.

    If no information was found, we searched the World Health Organization (WHO) International Clinical Trials Registry Platform (ICTRP). Also, according to the author’s affiliation, we searched the local registry if not part of the WHO ICTRP. We used the article title, the first and last author’s name as author or as investigator, or the author’s affiliation as search keywords.

     
  3. 3.

    If no information was found, we searched Google with the publication title and keywords regarding registration (e.g., registry, registration, NCT etc.).

     
  4. 4.

    If no information was found, we contacted the corresponding author to ask whether the trial was registered and whether a protocol was available.

     

Evaluation of the reason for the missing outcome

For each identified RCT, one of us (YY) classified each missing outcome into one of the following five categories:
  1. 1.
    Inadequate planning
    • The outcome was not planned according to the protocol or the registry entry. It was not reported in the available trial reports.

     
  2. 2.
    Selective reporting
    • The outcome was planned according to the protocol or the registry entry but was not reported in the available trial reports.

     
  3. 3.
    Incomplete reporting
    • The outcome was planned according to the protocol or the registry entry and was reported in the available trial reports but incompletely or not in a way that allowed for meta-analysis (e.g., mixed model analyses reported, no results per group, missing control group results, difference in means without the standard error, etc.).

    • No protocol or registry entry was found. The outcome was reported in the available trial reports but not in a way that allowed for meta-analysis.

     
  4. 4.
    Unable to distinguish between selective reporting and inadequate planning
    • No protocol or registry entry was found and the outcome was not reported in the available trial reports.

     
  5. 5.
    Other situations
    • The outcome was listed in the trial reports, but there was no event (e.g., the outcome was death but no death occurred).

    • The outcome concerned adverse events, but there was no event (e.g., the outcome was “Major Complications—Visceral injury,” but no complications were reported).

    • The outcome was reported in the available trial reports in a way that could allow for inclusion in a meta-analysis but was not included in the meta-analysis.

     

As a quality assessment measure, 10% of the RCTs were classified independently by two reviewers (YY and AD), with no disagreements.

Evaluation of research waste

Feasibility and cost of measuring the missing outcomes that were not planned

To evaluate whether the missing outcomes that were not planned could have been easily measured, we used an expert consensus approach. One of us (YY) presented to an expert panel of five methodologists and trialists (AD, IB, PC, PR, and VT) each RCT for which at least one missing outcome was classified as not planned. After evaluating standardized information on the population, interventions in the experimental and control group, and other outcomes evaluated, the panel of experts were asked to answer the following questions regarding the missing outcomes:
  • “According to you, given the other outcomes measured in the trial, and based on your experience, would you consider that measuring the presented outcome would be easy, moderately easy, difficult or impossible in most cases from a trialist’s perspective?”

  • “According to you, given the other outcomes measured in the trial, and based on your experience, would you consider that measuring the presented outcome would be easy, moderately easy, difficult or impossible in most cases from a patient’s perspective?”

  • “According to you, given the other outcomes measured in the trial, and based on your experience, what would be the approximate cost of measuring this outcome: no cost defined as ≤ 1% of the total cost of the trial; minor cost, defined as ≤ 5%; moderate cost, 5% to 15%; or major cost, 15% or more”. These percentages were indicative.

  • “According to you, how important is the missing outcome: major importance; non-major importance?”

One of us (YY) ensured that all experts first gave their opinion and then discussed together, to avoid the opinion of one leading person influencing the others. The final feasibility and cost evaluation of every proposed adjustment was based on the group consensus.

The qualifications and areas of expertise of the experts involved are summarized in Additional file 2.

Avoidable waste of research related to missing outcomes

We defined avoidable waste related to missing outcomes as missing important outcomes related to:
  • Selective reporting

  • Incomplete reporting

  • Inadequate planning, if the outcome was judged by the expert panel as easy to collect from both the trialist and patient perspective, not costly (i.e., no or minor additional cost), and of critical importance

At the trial level, such waste could have been partially avoided if the trial could have been included in the meta-analysis of at least one missing outcome and totally if it could have been included in all meta-analyses of missing outcomes.

Statistical analysis

The analysis was mainly descriptive. Continuous data are presented as median (Q1–Q3) and categorical data as frequencies (%, 95% confidence interval [CI]). All analyses involved use of R 3.0.2 (2013–09-25) (R Foundation for Statistical Computing, Vienna, Austria. http://www.r-project.org/).

Results

Identification of RCTs excluded from meta-analyses

Selection and characteristics of the Cochrane systematic reviews

The complete selection process is described in Fig. 1. Briefly, from the 2796 Cochrane systematic reviews published between March 2011 and September 2014, 820 reviews corresponded to our eligibility criteria. Our selection process resulted in a sample of 290 reviews (5047 RCTs), with a median of 11 RCTs per review (Q1–Q3: 5–21) (Fig. 1). The subset of included reviews appeared comparable to the eligible Cochrane Reviews (Additional file 3). The reviews investigated 47 different health research topics, the most common being airways (7%, n = 21/290), menstrual disorders and subfertility (7%, n = 20/290), and anesthesia (6%, n = 18/290) (Table 1). Experimental interventions were pharmacological in 60% of reviews (n = 175/290).
Fig. 1

Flow chart of the selection process. This figure summarizes the selection process for Cochrane reviews, RCTs, and outcomes. The analyzed sample involved 290 Cochrane reviews, which included 2711 RCTs in the SoF table of the main comparison. The SoF tables reported 1414 important outcomes. RCT randomized controlled trial, SoF summary of findings

Table 1

Characteristics of selected systematic reviews and outcomes

Review characteristics

n = 290

 

Median (Q1–Q3)

 No. of trials per review

11 (5–21)

 No. of trials in the main comparison

5 (3–12)

 No. of comparisons per review

2 (1–5)

Cochrane review groups

n = 290

 

No. (%)

 Airways

21 (7)

 Menstrual disorders and subfertility

20 (7)

 Anesthesia

18 (6)

 Oral health

15 (5)

 Schizophrenia

15 (5)

 Hepato-biliary

12 (4)

 Neuromuscular disease

12 (4)

 Musculoskeletal

11 (4)

 Infectious diseases

10 (3)

 Other

156 (54)

Type of assessed intervention

n = 290

 

No. (%)

 Pharmacological

175 (60)

 Non-pharmacological

115 (40)

Outcomes characteristics

n = 141

 No. of outcomes per SoF table, median (Q1–Q3)

5 (3–7)

 No. of trials per meta-analysis, median (Q1–Q3)

3 (2–7)

Outcome categories

n = 1414

 

No. (%)*

 Function

384 (27)

 Other clinical events

198 (14)

 Adverse events; side effects

174 (12)

 Mortality

138 (10)

 Quality of life

98 (7)

 Biological variables

89 (6)

 Process, resource use

74 (5)

 Pain

71 (5)

 Physiological variables

56 (4)

 Compliance

45 (3)

 Therapeutic decision

33 (2)

 Satisfaction with care

24 (2)

 Radiological variables

23 (2)

 Cost-effectiveness

16 (1)

*The total exceeds 100% because some outcomes were included in more than one category

The reviews included a median of 5 outcomes per SoF table for the main comparison (Q1–Q3: 3–7, Min–Max: 1–12), for a total of 1414 outcomes. Mortality represented 10% (n = 138/1414) of the outcomes, and other clinical events, 14% (n = 198/1414); quality of life, 7% (n = 98/1414); function, 27% (n = 384/1414); and adverse events, 12% (n = 174/1414) (Table 1). Biological variables, process and resource use, and physiological variables accounted for 6% (n = 89/1414), 5% (n = 74/1414), and 4% (n = 56/1414), respectively. The corresponding meta-analyses included a median of 3 RCTs (Q1–Q3: 2–7), maximum 76.

Proportion of RCTs excluded from the meta-analyses due to outcome reasons

The 290 reviews included a total of 2711 RCTs in the main comparison; 596 (22%; 95% CI: 20–24) were included in all meta-analyses of important outcomes, and 2115 (78%, 95% CI: 76–80) were excluded from at least one meta-analysis. Every RCT contributed to 55%, on average, of the meta-analyses for important outcomes.

Evaluation of the reason for missing outcomes

Among the 2115 RCTs excluded from at least one meta-analysis of important outcomes, 310 were published in 2010 or later. We further excluded 19 RCTs, because their reports were not accessible (e.g., retracted paper) or were not in English or French (e.g., Chinese), which left 291 RCTs for further evaluation, with a total of 971 missing outcomes. A registration number or a protocol was retrieved for 54% (n = 156/291) of these RCTs (corresponding to 461 missing outcomes).

Reasons for missing outcomes were incomplete reporting for 21% of missing outcomes (n = 204/971) in 40% of RCTs (n = 117/291), and inadequate planning for 29% of missing outcomes (n = 282/971) in 42% of RCTs (n = 123/291) (Table 2). Confirmed selective reporting represented 4% of missing outcomes (n = 36/971) in 9% of RCTs (n = 25/291). Nevertheless, 42% of RCTs (n = 122/291) had no protocol or registration, so for these, we could not distinguish between selective reporting and inadequate planning.
Table 2

Classification of reasons for missing outcomes

 

All trials published in 2010 or later

Reasons for missing outcome

All trials (registered and unregistered)

Registered trials

No. of affected outcomes (%)

N = 971

No. of affected trials (%)* N = 291

No. of affected outcomes (%)

N = 461

No. of affected trials (%)*

N = 156

Inadequate planning

282 (29)

123 (42)

282 (61)

123 (79)

Selective reporting

36 (4)

25 (9)

34 (7)

23 (15)

Incomplete reporting

204 (21)

117 (40)

98 (21)

64 (41)

Unable to distinguish between selective reporting and lack of planning

363 (39)

122 (42)

  

Other situations

86 (9)

63 (24)

47 (10)

41 (26)

*The total exceeds 100% because some outcomes were included in more than one category

When restricting our description to registered RCTs (n = 156), inadequate planning concerned 61% of missing outcomes (n = 282/461) in 79% of RCTs (n = 123/156), whereas incomplete reporting accounted for 21% of missing outcomes (n = 98/461) in 41% of RCTs (n = 64/156), and selective reporting, 7% of missing outcomes (n = 34/461) in 15% of RCTs (n = 23/156) (Table 2).

Evaluation of research waste

Feasibility of measuring the missing outcomes that were not planned

We submitted the 282 outcomes missing due to inadequate planning from 123 RCTs to our panel of experts. For 78% of outcomes (n = 221/282), the experts judged the outcome of critical importance given the context. For these 221 critically important outcomes, they considered that 82% (n = 182) could have been easily measured from both the trialist and patient perspective at no cost (Additional file 4).

Avoidable waste of research due to missing important outcomes

For the 291 RCTs published in 2010 or later, taking into account selective or incomplete outcome reporting, waste of research could have been partially avoided for 43% (n = 126) and totally (i.e., the trial could have been included in all meta-analyses of important outcomes) for 12% (n = 34) (Fig. 2). If we also consider missing outcomes that could have been easily measured from the planning stage at no additional cost as judged by our experts, waste of research could have been partially avoided for 63% of RCTs (n = 183) and totally for 30% (n = 86) (Fig. 2).
Fig. 2

Avoidable waste of research related to missing outcomes in RCTs published in 2010 or later and not contributing to all meta-analyses of important outcomes (n = 291). This figure summarizes the evolution of the outcomes status of SoF tables (O1 to O12 because SoF tables included from 1 to 12 outcomes per review) for each RCT at the different steps of our study: step I (i.e., information as it was extracted), step II (i.e., after evaluating the reason for the missing outcome) and after considering if there was no selective reporting or incomplete reporting, and finally step III (i.e., evaluation of waste of research) and after accounting for inadequate planning and selective reporting or incomplete reporting (i.e., adequate planning and no selective reporting or incomplete reporting). On the y-axis, each line represents one of the 291 RCTs published in 2010 or later and not contributing to all meta-analyses of the outcomes reported in the SoF table. The x-axis represents the outcomes reported in the SoF table. Each brick represents a single outcome of the SoF table. The color is a visual representation of the presence or absence of the outcome in the RCT. Light blue means that the outcome was present in the corresponding RCT and dark blue that the outcome was absent in the RCT. RCT randomized controlled trial, SoF summary of findings

Discussion

We evaluated the avoidable waste of research due to outcome-related reasons across a large number of RCTs included in recent Cochrane systematic reviews in a variety of medical fields. Our analysis revealed that 78% of the RCTs were not included in all meta-analyses of important outcomes and that every RCT contributed to 55%, on average, of the meta-analyses of these important outcomes. Among registered RCTs, inadequate planning was the most common reason affecting 79% of RCTs with missing outcomes. Such waste could have been partially avoided for 63% of the RCTs and totally for 30%.

Our findings suggest that many RCTs were not included in all meta-analyses of important outcomes and that every RCT contributed to only half of the meta-analyses, on average. These results seem consistent with the literature. In a recent study of patient-important outcomes, the median proportion of trials included in a meta-analysis for such outcomes was about 60% [39]. Approximately half of the RCTs included in another sample of Cochrane reviews were included in the pooled effect size estimates in the meta-analyses of patient-important outcomes [40], whereas more than one-third of the outcomes pre-specified in the review were not reported in the results sections [18, 41].

Although possible reasons for excluding RCTS from meta-analyses have been hypothesized, the respective contribution of inadequate planning and selective reporting or incomplete reporting was unknown. Our results provide major insights into this question, showing that many RCTs do not plan the most important outcomes to evaluate. Inadequate planning represents a missed opportunity within a trial because equipoise remains unchanged after this trial. However, a missed opportunity at the trial level will result in waste of research at the meta-analysis level. Actually, clinical research should be considered a sequential process, with each new trial contributing to the existing body of evidence. If a trial fails to do so because of a missed opportunity, it becomes a source of waste of research and resources at the meta-analysis level because this trial will not contribute to the overall evidence. Cochrane reviewers may select outcomes that differ in importance from those selected by trialists (typically those defined as primary outcomes and used for sample size calculation). Trialists rarely consider adverse events (particularly severe adverse events) as primary outcomes and for sample size calculation. However, these outcomes are crucial to assess and report because they could be included in meta-analyses to increase power and inform decision-making. Identifying important outcomes is challenging, and without a consensus on outcomes, heterogeneity in outcomes evaluated will remain [41].

To overcome these issues, several initiatives have emerged to improve the relevance and consistency of outcomes used in clinical research [4, 42, 43, 44, 45]; one is the Core Outcome Measures in Effectiveness Trials (COMET) initiative promoting the development and use of standardized core outcome sets (COSs). COMET offers extremely helpful tools for researchers aiming to develop COSs in their fields. However, although the number of COSs is rapidly increasing, it remains limited [46, 47, 48], and searching for specific existing COSs for a given condition is difficult. The COMET website may evolve to present available COSs in a tabular fashion to help trialists identify the outcomes to use and researchers the conditions for which COSs are needed. Another perspective would be to use the outcomes reported in the SoF table of Cochrane reviews to develop COSs, because most of these outcomes seem important to patients.

Incomplete reporting was another common reason for outcomes excluded from meta-analyses. Two main situations can be distinguished. First, the failure to report results adequately (e.g., reporting means without standard deviations) and second, the reporting of an analysis that cannot be included as such in a meta-analysis (e.g., reporting of repeated data analysis). The first situation is clearly related to poor reporting. We previously showed that results were more completely reported at ClinicalTrials.gov than in publications, probably because of the standardized template used to report results at ClinicalTrials.gov [49]. Such standardized templates may help trialists determine which data should be reported and should be included in reporting guidelines and required by journals to improve the presentation of results. The other situation is more complex because it does not reflect poor reporting, just reporting in a different way. Studies with repeated data are meant to record measurements at numerous time points to inform researchers of changes over time. However, several approaches for meta-analyzing these types of data exist, and they can differ in terms of the data needed for analysis [50]. Therefore, reporting trial results not just for an immediate use but also considering a later inclusion in meta-analyses can be challenging. The Instrument for reporting of Planned Endpoints in Clinical Trials (InsPECT) reporting guidelines (currently under development, https://www.inspect-statement.org/) might help trial authors mitigate the effects of incomplete reporting in RCTs.

Finally, selective reporting of outcomes seems to affect a few trials. It remains an important issue in clinical trials, with, on average, one-third of discrepancies in primary outcomes between protocols and publications [29] or between registry information and publications [31]. In our situation, this concerns only outcomes that were planned and not reported at all in publications. Actually, reviewers consider a given outcome whether this outcome is reported as a primary or a secondary outcome.

Strengths and limitations of the study

In this study, we used an original approach to evaluate the main reasons why a trial was not included in a meta-analysis and the part of this waste that could be avoided. We analyzed a large set of trials included in a vast, unselected, sample of recent Cochrane reviews exploring various health-care research topics. Assembling numerous experts in such a variety of fields to identify the most important outcomes to evaluate would have been challenging. As a proxy, we used the outcomes reported in the SoF tables that were considered by the review authors as the most important to measure for a given comparison in a particular health condition [7]. We used the consensus of an expert panel of recognized methodologists and trialists to assess the feasibility and costs of measuring the missing outcomes. To avoid overestimating waste of research, we also asked the panel to evaluate the importance of outcomes and considered that research was wasted for only those outcomes the panel confirmed to be of critical importance.

Our study has several limitations. First, we focused only on trials included in Cochrane reviews published between 2010 and 2014. However, it has been reported that Cochrane and non-Cochrane reviews have systemic differences, likely reflective of different methodology [51]. Then, we focused on trials included in the reviews but excluded from at least one meta-analysis. We did not consider the number of trials excluded from the reviews for outcome-related reasons. Therefore, we probably underestimated the proportion of trials excluded from meta-analyses for outcome-related reasons. Despite the recommendation of the Cochrane Handbook to determine trial eligibility independently of outcomes measured (studies should not be excluded just because they provide no usable data), some trials may be excluded for this reason [7]. Thus, although, our sample of RCTs was from a large variety of medical fields, the database used (Cochrane reviews published between 2010 and 2014 and with an SoF table), the limited sample size, the language restrictions, or the expertise of our panel of experts may limit the generalizability of our results. Finally, as a quality measure, 10% of the RCTs were classified independently by two reviewers, since this assessment is challenging and implies some subjectivity.

Conclusions

Our study shows that most RCTs included in our sample of Cochrane reviews did not contribute to all meta-analyses of the most important outcomes mainly because of inadequate planning or incomplete reporting. Such waste could have been partially avoided for 63% of the trials and totally for 30%. We need to accelerate the development and dissemination of COSs and reduce poor outcome reporting to avoid this waste.

Notes

Acknowledgments

We thank Elise Diard for help in designing Fig. 2.

Funding

This study did not receive any specific funding.

Availability of data and materials

Data available upon request by academic researchers.

Authors’ contributions

YY was involved in study conception, selection of trials, data extraction, data analysis, interpretation of results, and drafting the manuscript. AD was involved in study conception, selection of trials, data extraction, data analysis, interpretation of results, and drafting the manuscript. IA was involved in initial data management, interpretation of results, and drafting the manuscript. VT, IB, and PC were involved in the interpretation of results and drafting the manuscript. PR was involved in study conception, interpretation of results, and drafting the manuscript. All authors read and approved the final manuscript.

Ethics approval and consent to participate

Ethics approval was not applicable. This was a research-on-research study of pre-existing published research; therefore, there was no patient recruitment.

Competing interests

The authors declare that they have no competing interests. All authors have completed the International Committee of Medical Journal Editors uniform disclosure form at www.icmje.org/coi_disclosure.pdf (available on request from the corresponding author) and declare that they received no support from any organization for the submitted work, they have no financial relationships with any organizations that might have an interest in the submitted work in the previous 3 years, and they have no other relationships or activities that could appear to have influenced the submitted work.

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary material

12916_2018_1083_MOESM1_ESM.docx (19 kb)
Additional file 1: Summary of the inclusion and exclusion criteria for the different phases of the study. (DOCX 18 kb)
12916_2018_1083_MOESM2_ESM.docx (15 kb)
Additional file 2: Qualifications and areas of expertise of the experts involved in the final step of the study. (DOCX 15 kb)
12916_2018_1083_MOESM3_ESM.docx (16 kb)
Additional file 3: Characteristics of the 820 Cochrane systematic reviews and the analyzed subset of 290 reviews. (DOCX 15 kb)
12916_2018_1083_MOESM4_ESM.doc (56 kb)
Additional file 4: Experts’ opinion of the feasibility and costs of measuring the missing outcomes. (DOC 56 kb)

References

  1. 1.
    Tugwell P, Boers M. OMERACT conference on outcome measures in rheumatoid arthritis clinical trials: introduction. J Rheumatol. 1993;20(3):528–30.PubMedGoogle Scholar
  2. 2.
    Frank L, Basch E, Selby JV. Patient-centered outcomes research I: the PCORI perspective on patient-centered outcomes research. Jama. 2014;312(15):1513–4.CrossRefPubMedGoogle Scholar
  3. 3.
    Koroshetz W. A core set of trial outcomes for every medical discipline? BMJ. 2015;350:h85.CrossRefPubMedGoogle Scholar
  4. 4.
    Williamson P, Altman D, Blazeby J, Clarke M, Gargon E. Driving up the quality and relevance of research through the use of agreed core outcomes. J Health Serv Res Policy. 2012;17(1):1–2.CrossRefPubMedGoogle Scholar
  5. 5.
    Tunis SR, Clarke M, Gorst SL, Gargon E, Blazeby JM, Altman DG, Williamson PR. Improving the relevance and consistency of outcomes in comparative effectiveness research. J Comp Eff Res. 2016;5(2):193-205.Google Scholar
  6. 6.
    Murad MH, Shah ND, Van Houten HK, Ziegenfuss JY, Deming JR, Beebe TJ, Smith SA, Guyatt GH, Montori VM. Individuals with diabetes preferred that future trials use patient-important outcomes and provide pragmatic inferences. J Clin Epidemiol. 2011;64(7):743–8.CrossRefPubMedGoogle Scholar
  7. 7.
    Higgins JPT, Green S (editors). Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 [updated March 2011]. The Cochrane Collaboration, 2011. Available from http://handbook.cochrane.org.
  8. 8.
    Ioannidis JPA. Why most clinical research is not useful. PLoS Med. 2016;13(6):e1002049.CrossRefPubMedPubMedCentralGoogle Scholar
  9. 9.
    Chalmers I, Glasziou P. Avoidable waste in the production and reporting of research evidence. Lancet. 2009;374(9683):86–9.CrossRefPubMedGoogle Scholar
  10. 10.
    Al-Shahi Salman R, Beller E, Kagan J, Hemminki E, Phillips RS, Savulescu J, Macleod M, Wisely J, Chalmers I. Increasing value and reducing waste in biomedical research regulation and management. Lancet. 2014;383(9912):176–85.CrossRefPubMedGoogle Scholar
  11. 11.
    Chalmers I, Bracken MB, Djulbegovic B, Garattini S, Grant J, Gulmezoglu AM, Howells DW, Ioannidis JP, Oliver S. How to increase value and reduce waste when research priorities are set. Lancet. 2014;383(9912):156–65.CrossRefPubMedGoogle Scholar
  12. 12.
    Chan AW, Song F, Vickers A, Jefferson T, Dickersin K, Gotzsche PC, Krumholz HM, Ghersi D, van der Worp HB. Increasing value and reducing waste: addressing inaccessible research. Lancet. 2014;383(9913):257–66.CrossRefPubMedPubMedCentralGoogle Scholar
  13. 13.
    Glasziou P, Altman DG, Bossuyt P, Boutron I, Clarke M, Julious S, Michie S, Moher D, Wager E. Reducing waste from incomplete or unusable reports of biomedical research. Lancet. 2014;383(9913):267–76.CrossRefPubMedGoogle Scholar
  14. 14.
    Ioannidis JP, Greenland S, Hlatky MA, Khoury MJ, Macleod MR, Moher D, Schulz KF, Tibshirani R. Increasing value and reducing waste in research design, conduct, and analysis. Lancet. 2014;383(9912):166–75.CrossRefPubMedPubMedCentralGoogle Scholar
  15. 15.
    Macleod MR, Michie S, Roberts I, Dirnagl U, Chalmers I, Ioannidis JP, Al-Shahi Salman R, Chan AW, Glasziou P. Biomedical research: increasing value, reducing waste. Lancet. 2014;383(9912):101–4.CrossRefPubMedGoogle Scholar
  16. 16.
    Moher D, Glasziou P, Chalmers I, Nasser M, Bossuyt PMM, Korevaar DA, Graham ID, Ravaud P, Boutron I. Increasing value and reducing waste in biomedical research: who's listening? Lancet. 2016;387(10027):1573–86.CrossRefPubMedGoogle Scholar
  17. 17.
    Yordanov Y, Dechartres A, Porcher R, Boutron I, Altman DG, Ravaud P. Avoidable waste of research related to inadequate methods in clinical trials. BMJ. 2015;350:h809.CrossRefPubMedPubMedCentralGoogle Scholar
  18. 18.
    Smith V, Clarke M, Williamson P, Gargon E. Survey of new 2007 and 2011 Cochrane reviews found 37% of prespecified outcomes not reported. J Clin Epidemiol. 2014;68(3):237–45.Google Scholar
  19. 19.
    Gandhi GY, Murad MH, Fujiyoshi A, Mullan RJ, Flynn DN, Elamin MB, Swiglo BA, Isley WL, Guyatt GH, Montori VM. Patient-important outcomes in registered diabetes trials. JAMA. 2008;299(21):2543–9.CrossRefPubMedGoogle Scholar
  20. 20.
    Rahimi K, Malhotra A, Banning AP, Jenkinson C. Outcome selection and role of patient reported outcomes in contemporary cardiovascular trials: systematic review. BMJ. 2010;341:c5707.CrossRefPubMedPubMedCentralGoogle Scholar
  21. 21.
    Svensson S, Menkes DB, Lexchin J. Surrogate outcomes in clinical trials: a cautionary tale. JAMA Intern Med. 2013;173(8):611–2.CrossRefPubMedGoogle Scholar
  22. 22.
    Naci H, Ioannidis JP. How good is “evidence” from clinical studies of drug effects and why might such evidence fail in the prediction of the clinical utility of drugs? Annu Rev Pharmacol Toxicol. 2015;55:169–89.CrossRefPubMedGoogle Scholar
  23. 23.
    Ocana A, Tannock IF. When are “positive” clinical trials in oncology truly positive? J Natl Cancer Inst. 2010;103(1):16–20.CrossRefPubMedGoogle Scholar
  24. 24.
    Gluud C, Krogsgaard K. Would you trust a surrogate respondent? Lancet. 1997;349(9053):665–6.CrossRefPubMedGoogle Scholar
  25. 25.
    The Cardiac Arrhythmia Suppression Trial (CAST) Investigators. Preliminary report: effect of encainide and flecainide on mortality in a randomized trial of arrhythmia suppression after myocardial infarction. N Engl J Med. 1989;321(6):406–12.CrossRefGoogle Scholar
  26. 26.
    Ciani O, Buyse M, Garside R, Pavey T, Stein K, Sterne JA, Taylor RS. Comparison of treatment effect sizes associated with surrogate and final patient relevant outcomes in randomised controlled trials: meta-epidemiological study. BMJ. 2013;346:f457.CrossRefPubMedPubMedCentralGoogle Scholar
  27. 27.
    Moynihan R. Surrogates under scrutiny: fallible correlations, fatal consequences. BMJ. 2011;343:d5160.CrossRefPubMedGoogle Scholar
  28. 28.
    Walter SD, Sun X, Heels-Ansdell D, Guyatt G. Treatment effects on patient-important outcomes can be small, even with large effects on surrogate markers. J Clin Epidemiol. 2012;65(9):940–5.CrossRefPubMedGoogle Scholar
  29. 29.
    Chan AW, Hrobjartsson A, Haahr MT, Gotzsche PC, Altman DG. Empirical evidence for selective reporting of outcomes in randomized trials: comparison of protocols to published articles. JAMA. 2004;291(20):2457–65.CrossRefPubMedGoogle Scholar
  30. 30.
    Dwan K, Altman DG, Cresswell L, Blundell M, Gamble CL, Williamson PR. Comparison of protocols and registry entries to published reports for randomised controlled trials. Cochrane Database Syst Rev. 2011;1:MR000031.Google Scholar
  31. 31.
    Mathieu S, Boutron I, Moher D, Altman DG, Ravaud P. Comparison of registered and published primary outcomes in randomized controlled trials. JAMA. 2009;302(9):977–84.CrossRefPubMedGoogle Scholar
  32. 32.
    Williamson PR, Gamble C. Identification and impact of outcome selection bias in meta-analysis. Stat Med. 2005;24(10):1547–61.CrossRefPubMedGoogle Scholar
  33. 33.
    Kirkham JJ, Dwan KM, Altman DG, Gamble C, Dodd S, Smyth R, Williamson PR. The impact of outcome reporting bias in randomised controlled trials on a cohort of systematic reviews. BMJ. 2010;340:c365.CrossRefPubMedGoogle Scholar
  34. 34.
    Jones CW, Keil LG, Holland WC, Caughey MC, Platts-Mills TF. Comparison of registered and published outcomes in randomized controlled trials: a systematic review. BMC Med. 2015;13:282.CrossRefPubMedPubMedCentralGoogle Scholar
  35. 35.
    Chan AW, Krleza-Jeric K, Schmid I, Altman DG. Outcome reporting bias in randomized trials funded by the Canadian Institutes of Health Research. CMAJ. 2004;171(7):735–40.CrossRefPubMedPubMedCentralGoogle Scholar
  36. 36.
    Chan A-W, Altman DG. Identifying outcome reporting bias in randomised trials on PubMed: review of publications and survey of authors. BMJ. 2005;330(7494):753.CrossRefPubMedPubMedCentralGoogle Scholar
  37. 37.
    Dechartres A, Ravaud P, Atal I, Riveros C, Boutron I. Association between trial registration and treatment effect estimates: a meta-epidemiological study. BMC Med. 2016;14(1):100.CrossRefPubMedPubMedCentralGoogle Scholar
  38. 38.
    Mocellin S, Lens MB, Pasquali S, Pilati P, Chiarion Sileni V. Interferon alpha for the adjuvant treatment of cutaneous melanoma. Cochrane Database Syst Rev. 2013;6:CD008955.Google Scholar
  39. 39.
    Ameur H, Ravaud P, Fayard F, Riveros C, Dechartres A. Systematic reviews of therapeutic interventions frequently consider patient-important outcomes. J Clin Epidemiol. 2017;84:70–77.Google Scholar
  40. 40.
    Furukawa TA, Watanabe N, Omori IM, Montori VM, Guyatt GH. Association between unreported outcomes and effect size estimates in Cochrane meta-analyses. JAMA. 2007;297(5):468–70.CrossRefPubMedGoogle Scholar
  41. 41.
    Wuytack F, Smith V, Clarke M, Williamson P, Gargon E. Towards core outcome set (COS) development: a follow-up descriptive survey of outcomes in Cochrane reviews. Syst Rev. 2015;4(1):73.CrossRefPubMedPubMedCentralGoogle Scholar
  42. 42.
    The COMET (Core Outcome Measures in Effectiveness Trials) Initiative [http://www.comet-initiative.org/]. Accessed 11 Apr 2017.
  43. 43.
    Methodology Committee of the Patient-Centered Outcomes Research I. Methodological standards and patient-centeredness in comparative effectiveness research: the PCORI perspective. JAMA. 2012;307(15):1636–40.CrossRefGoogle Scholar
  44. 44.
    Selby JV, Beal AC, Frank L. The Patient-Centered Outcomes Research Institute (PCORI) national priorities for research and initial research agenda. JAMA. 2012;307(15):1583–4.CrossRefPubMedGoogle Scholar
  45. 45.
    Clarke M, Williamson P. Core outcome sets and trial registries. Trials. 2015;16(1):216.CrossRefPubMedPubMedCentralGoogle Scholar
  46. 46.
    Gargon E, Williamson PR, Altman DG, Blazeby JM, Tunis S, Clarke M. The COMET initiative database: progress and activities update (2015). Trials. 2017;18(1):54.CrossRefPubMedPubMedCentralGoogle Scholar
  47. 47.
    Gargon E, Williamson PR, Altman DG, Blazeby JM, Clarke M. The COMET initiative database: progress and activities update (2014). Trials. 2015;16:515.CrossRefPubMedPubMedCentralGoogle Scholar
  48. 48.
    Gargon E, Williamson PR, Altman DG, Blazeby JM, Clarke M. The COMET initiative database: progress and activities from 2011 to 2013. Trials. 2014;15:279.CrossRefPubMedPubMedCentralGoogle Scholar
  49. 49.
    Riveros C, Dechartres A, Perrodeau E, Haneef R, Boutron I, Ravaud P. Timing and completeness of trial results posted at ClinicalTrials.gov and published in journals. PLoS Med. 2013;10(12):e1001566.CrossRefPubMedPubMedCentralGoogle Scholar
  50. 50.
    Peters JL, Mengersen KL. Meta-analysis of repeated measures study designs. J Eval Clin Pract. 2008;14(5):941–50.CrossRefPubMedGoogle Scholar
  51. 51.
    Useem J, Brennan A, LaValley M, Vickery M, Ameli O, Reinen N, Gill CJ. Systematic differences between Cochrane and non-Cochrane meta-analyses on the same topic: a matched pair analysis. PLoS One. 2015;10(12):e0144980.CrossRefPubMedPubMedCentralGoogle Scholar

Copyright information

© The Author(s). 2018

Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.

Authors and Affiliations

  1. 1.INSERM, U1153Hôpital Hôtel-DieusParisFrance
  2. 2.Sorbonne Universités, UPMC Paris Univ-06ParisFrance
  3. 3.Service des Urgences - Hôpital Saint Antoine, Assistance Publique–Hôpitaux de Paris (APHP)ParisFrance
  4. 4.Centre d’Épidémiologie CliniqueHôpital Hôtel Dieu, Assistance Publique–Hôpitaux de Paris (APHP)ParisFrance
  5. 5.Faculté de MédecineUniversité Paris Descartes, Sorbonne Paris CitéParisFrance
  6. 6.Cochrane FranceParisFrance
  7. 7.Department of EpidemiologyColumbia University, Mailman School of Public HealthNew YorkUSA

Personalised recommendations