Background

Randomised trials are the highest level of evidence in clinical research and are, therefore, pivotal in guiding clinical practice [1]. Accordingly, such trials are often published in prestigious high-impact journals. However, some randomised trials may be less robust than others. Trials with end points that are surrogates for clinically important (hard) outcomes may be less reliable because beneficial changes in surrogate end points do not necessarily translate to clinically meaningful benefits [2]. For example, a decrease in glycated haemoglobin, a surrogate end point frequently reported in trials of type 2 diabetes, is not invariably accompanied by reductions in clinical events such as incidence of diabetes complications or vascular events [3]. In oncology, only 5 of 23 drugs that obtained regulatory approval based on trials with surrogate end points between 2008 and 2012 and that were subsequently evaluated in trials with hard primary outcomes were found to confer a survival benefit [4]. There are numerous examples of treatments that entered clinical practice based on favourable effects on surrogate end points but which were subsequently found to be harmful [2].

The sponsorship of clinical trials might also influence their reliability. Systematic reviews indicate that industry-sponsored drug research is more likely to report beneficial effects and favourable safety profiles than non-industry-sponsored drug research [5, 6]. Research on new drugs and devices is almost exclusively sponsored by industry, and almost always includes the conduct of randomised trials with surrogate end points. The thrust of biomedicine to produce new and improved interventions might favour the reporting of clinical trials of new treatments in prestigious influential journals [7], particularly if those journals profit from such publications [8].

Clinical practice comprises a range of management strategies. Interventions include medications, medical devices and technologies, and procedures. The relative proportions of these interventions that are applied varies by discipline, but devices and procedures are an integral part of the working lives of most practitioners, and their use affects patient outcomes. Concerns have been expressed that medical devices can enter clinical practice without supporting evidence from clinical trials [9,10,11], perhaps because there are less stringent regulations for them than for pharmaceuticals [12].

In the current work, we set out to determine the prevalence of surrogate primary outcomes in clinical trials published in high-impact journals, the prevalence of the types of intervention assessed in such trials and whether there is a higher proportion of surrogate primary outcomes in industry-sponsored trials than non-industry-funded trials. Accordingly, we assessed the types of primary end points, types of intervention assessed and sponsorship in a set of randomised trials published in major internal medicine and specialty journals.

Methods

Dataset of clinical trials

We assessed randomised clinical trials published in 7 general and 10 specialty journals in 2013 and 2014. We focused on the journals that are likely to have the greatest readership and therefore, are most likely to influence clinical practice. The general medical journals were those with the highest impact factors: The New England Journal of Medicine (NEJM), Lancet, Journal of the American Medical Association (JAMA), The  British Medical Journal, JAMA Internal Medicine, Annals of Internal Medicine and PLoS Medicine. The specialty journals were the two journals with the highest impact factors that publish clinical research in each of five disciplines. For cardiology, we assessed publications in Journal of the American College of Cardiology and Circulation, for infectious diseases those in Lancet Infectious Diseases and Clinical Infectious Diseases, for neurology those in Lancet Neurology and Neurology, for oncology Lancet Oncology and Journal of Clinical Oncology, and for respiratory medicine Lancet Respiratory Medicine and American Journal of Respiratory and Critical Care Medicine.

One investigator (PG) searched Medline for reports of randomised clinical trials in each journal, collating the most recent publications before 31 December 2014 (Fig. 1). For each general medical journal, 30 publications were assessed (n = 210 in total) and 15 were assessed for each specialty journal (n = 30 for each speciality, n = 150 in total). The number of publications assessed was determined pragmatically. A second investigator (AG) checked the search and collation of publications.

Fig. 1
figure 1

Collation of reports of randomised trials

Full size image

We included all randomised trials conducted in humans, with primary outcomes relevant to human health. We excluded meta-analyses. To determine the eligibility of a publication, we reviewed its abstract and, if necessary, the full text.

Outcomes

For each included publication, we reviewed the full text of the trial report. We extracted data on trial sponsorship, primary outcome(s), trial design, type of intervention and trial results. All data were extracted by one investigator (PG) and checked by a second investigator (AG). In the event of disagreement, a third investigator (MB) provided an assessment and the majority view was recorded.

We categorised trial sponsorship as any industry (solely industry funding or mixed industry/non-industry funding) or non-industry. Primary outcomes of the trials were categorised as hard, surrogate or mixed, and single, co-primary or composite. We used the Institute of Medicine definition of surrogate outcomes as “biomarker[s] intended to substitute for a clinical endpoint [and] expected to predict clinical benefit (or harm …) based on epidemiologic, therapeutic, pathophysiologic, or other scientific evidence” [13]. Hard outcomes are patient-important end points that are definitive with respect to the disease process, and reflect how a patient feels, functions or survives [13]. Table 1 sets out examples from the current analysis of hard outcomes reported in trials in speciality journals. Interventions were categorised as drugs, devices, procedures or other. The latter category included interventions such as strategies for delivery of care, application of health systems or public health initiatives, health education, and counselling. Trial results for each intervention were categorised regarding the primary trial outcome as beneficial, neutral or harmful, based upon statistical assessments in the publication. All assessments were initially performed by one investigator (PG) and checked by a second (AG). In the event of disagreement, a third investigator (MB) provided an assessment and the majority view was recorded.

Table 1 Examples of hard outcomes (context) in trials reported in speciality journals
Full size table

Statistics

We used descriptive statistics to report the prevalence of types of sponsorship, primary outcome, trial design and trial results, and chi squared or Fisher’s exact test to compare categorical variables between trials with different types of sponsorship and in different types of journal.

Results

We assessed 360 trials that collectively reported the effects of 416 interventions (Table 2). We found that 233/416 interventions (56%) were drugs, 17/416 (4%) were devices and 49/416 (12%) were procedures. Primary outcomes were single in 237/360 trials (66%), co-primary in 47/360 trials (13%) and composite in 76/360 trials (21%). Primary outcomes were surrogate in 223/360 (61%) trials. A further 50 trials reported mixed surrogate/hard primary outcomes, meaning that in only 87/360 (24%) trials was the primary outcome hard. The primary outcome was clearly stated in the abstract of 286/360 (79%) of trial publications. The majority of trials were non-industry funded (220/360, 61%).

Table 2 Trial characteristics according to type of journal of publication
Full size table

Table 2 also shows the trial characteristics according to the type of journal. A greater proportion of trials studying interventions other than drugs, devices and procedures were published in general journals than specialty journals. Similar proportions of trials with solely surrogate primary end points were published in general and specialty journals (62% and 61%, respectively).

A higher proportion of trials published in specialty journals than in general journals had industry funding (83/150, 55% vs 53/210, 25%) (P < 0.001). In specialty journals, 67/150 (45%) trials were industry funded and included a drug intervention, compared to 37/210 (18%) in general journals (P < 0.001). In specialty journals, higher proportions of trials with both surrogate outcomes (61/111, 55%) and hard outcomes (22/36, 61%) had industry funding than those in general journals (39/161, 24% and 14/48, 29%, respectively, P < 0.01 for each comparison).

Table 3 shows the trial characteristics according to source of funding. Trials with interventions other than drugs, devices and procedures were more frequently sponsored by non-industry sources. No differences were observed in types of primary outcomes, or results for primary outcomes between trials with any industry funding and trials with no industry funding (Table 3).

Table 3 Trial characteristics by funding source
Full size table

Finally, we assessed types of outcome, types of interventions and funding of trials published by individual general journals and by the specialty journals within each medical discipline. Table 4 shows the results for the general journals. Among individual general journals, industry funding was associated with 7–63% of trials. Drug interventions were evaluated in 23–74% of trials and interventions other than drugs, devices and procedures were evaluated in 6–63% of trials. The two journals that published the highest proportions of trials of drug interventions (NEJM and Lancet) also published the lowest proportion of trials of interventions other than drugs, devices and procedures. The proportion of trials with solely surrogate primary outcomes was greater than 40% for each of the general journals (range 43–87%). Conversely, in none of the journals was the proportion of trials with a hard primary outcome greater than 40% (range 7–40%).

Table 4 Sponsorship, interventions and primary outcome characteristics in trials published in general journals
Full size table

Table 5 shows the types of outcomes, types of interventions and funding of trials published by the specialty journals in each medical discipline. Sponsorship by industry was associated with fewer than 50% of trials published in infectious diseases and respiratory medicine journals, and over 60% of trials published in neurology, oncology and cardiology journals. More than 50% of trials in each discipline assessed drugs or devices, and only 13–22% studied interventions other than drugs, devices and procedures. The prevalence of trials with solely surrogate primary outcomes was 60–80% in all disciplines except cardiology (33%). However, 27% of trials published in the cardiology journals reported mixed surrogate-hard primary outcomes, so 60% of cardiology trials included a surrogate primary outcome.

Table 5 Sponsorship, interventions and primary outcome characteristics in trials published in medical specialty journals
Full size table

Discussion

The majority of clinical trials published in major general medical and speciality journals in five clinical disciplines had a surrogate primary outcome; 61% of trials had only a surrogate primary outcome, and a further 15% included a surrogate outcome in a mixed primary outcome. Thus, only one quarter of the trials reported a solely hard primary outcome. Since surrogate outcomes may be unreliable in predicting hard outcomes [2, 4], our finding raises concerns that clinical trials published in these influential journals might contribute to the adoption of clinical practices that are subsequently contraindicated when trials with hard outcomes are conducted [2]. While trials with surrogate outcomes have a role in generating hypotheses about treatments and informing the design of trials with hard outcomes, their prominent publication in major journals might also have some disadvantageous effects.

The majority of trials we assessed (61%) were not sponsored by industry. Surrogate primary outcomes were present equally frequently in trials funded by industry and other sources. A higher proportion of trials published in specialty journals than general journals had industry funding. Since 86% of industry-funded trials were of either drugs or devices, compared to 44% of trials funded from non-industry sources, it is not surprising that a higher proportion of trials published in specialty journals compared to trials published in general journals were of drugs or devices. Some evidence suggests that industry-funded research disproportionately generates positive results [14, 15], but we found that the results for the primary outcome (benefit, neutral or harm) were similar in trials funded by industry or non-industry sources.

Medical devices and procedures are frequently used in medicine, and their widespread use can occur without strong evidence of efficacy and safety [10, 16]. In the current analysis, trials of devices and procedures comprised only 16% of trials published in these influential journals, while those that assessed drug interventions comprised 56%, and those that assessed interventions other than drugs, devices and procedures, 28%. The low proportion of trials evaluating devices and procedures may not reflect the importance of those interventions to clinical practice. Neither does it suggest that concerns about inadequate evidence for the efficacy and safety of devices are being addressed [10, 12, 17]. The importance of evaluating procedures is underscored by a meta-analysis of placebo-controlled trials of surgical interventions, which identified only 53 trials, in half of which the procedure was not superior to a placebo [18].

General medical journals varied widely in the proportions of trials they published that had any industry funding (7–63%) and of the types of intervention assessed. These characteristics are likely co-dependent, as industry funding is more likely to support trials of drugs and devices than trials of health-care delivery or health behaviours. Thus, the two general journals that published the highest proportion of industry-funded trials also published the highest proportion of trials of drugs and devices. Consistently, the proportion of trials with a hard primary outcome published in each of these influential general journals was <30%. Among the specialty journals, although there was also some variability in the proportions of published trials with industry funding, it was in each case more than 33%. With the exception of cardiology, journals in each discipline were strikingly similar in publishing mainly trials of drug interventions (67–79%). Few published trials in each discipline (13–22%) were of interventions other than drugs, devices or procedures. For four of five disciplines, trials with hard outcomes represented <30% of published trials, and in all disciplines such trials were the minority of published trials.

Our study has several limitations. Some trials were reported to have both industry and non-industry funding, but we were unable to ascertain whether one source predominated. Some non-industry-funding sources might derive income from commercial sources, but we could not determine this. We reported the characteristics of published trials but could not ascertain whether the results we observed reflect the characteristics of trials submitted for journal review; nonetheless our findings reflect the set of trial results most visible to practitioners. Categorising outcomes as surrogate or hard can sometimes be difficult [19], but we made independent assessments and recorded the majority opinion in the case of disagreement. Our analysis was confined to results published over 2 years, so does not permit inference about temporal patterns of trial publishing. We assessed the primary outcome in the trial publication rather than that in the trial registration document. We categorised trial results according to reported statistical significance, which might not reflect clinical importance. We did not undertake an assessment of risk of bias in the included trials, so cannot determine whether trial quality is associated with the variables we assessed.

Conclusion

The majority of trials published in high-impact general and specialty medical journals have surrogate primary outcomes and assess drug interventions. Trials of devices and procedures are infrequently published in such journals, despite the prominence of each type of intervention in clinical practice. Industry funding is more common for trials published in specialty than general journals but is not associated with more positive results for trial primary outcomes or with a greater preponderance of surrogate outcomes.