FormalPara Key Points

Randomized trials on drugs in our cohort were more likely to be stopped prematurely and receive more citations and mentions than randomized trials on devices, but this difference is largely attributed to several confounding factors other than the intervention itself.

Studies published simultaneously with their presentation were more likely to gain citations and mentions, and randomized trials on drugs with positive results obtained more citations than those with neutral results.

Our findings highlight the influence of dissemination strategies and study outcomes in determining the academic and social impact of randomized trials.

1 Introduction

Randomized controlled trials (RCTs) test the safety and efficacy of therapeutics, ensuring that only interventions meeting stringent criteria are integrated into patient care [1]. While frequently unified in their objective of evaluating safety and efficacy, RCTs investigating drugs and those assessing devices follow distinct regulatory pathways, reflecting their unique features and requirements [2].

The regulatory process for drugs is generally perceived as more rigorous and time consuming, especially for novel compounds, while medical devices can present unique challenges related to design, usage, and operator dependency [3]. Notably, with projected values reaching US$1156.00 billion for drugs compared to US$511.20 billion for devices by 2024, the drug market outperforms the device market [4,5,6]. This disparity suggests that drug RCTs may attract more interest and gain a wider impact, propelled by the pharmaceutical industry's growth due to aging populations, biotechnological progress, and rising chronic disease prevalence [4]. Therefore, discerning the differential impact of these interventions and its extent can be important in guiding resource allocation, research focus, and healthcare policy.

The successful conduct of RCTs depends on numerous factors, including the type of drug or device, the target condition, the regulatory environment, and the specific requirements of the study [7]. Citations and mentions serve as primary metrics for measuring the impact of RCTs. However, it remains unclear to what extent this impact is attributable to the type of intervention (i.e., drug or device) versus other shared factors like baseline characteristics or study outcomes (i.e., neutral or positive) [8,9,10,11,12,13,14,15,16,17]. The relationship between these factors and their collective effects on research visibility and impact has not been thoroughly explored.

This study aims to bridge this gap, exploring the impact of drug versus device RCTs in cardiovascular medicine and assessing if other factors, such as baseline characteristics and study outcomes, contribute to their overall influence on research and clinical practice.

2 Methods

2.1 Study Eligibility

Late-breaking RCTs presented at the American College of Cardiology (ACC), American Heart Association (AHA), European Society of Cardiology (ESC), and Transcatheter Cardiovascular Therapeutics (TCT) conferences were scrutinized from January 2015 to December 2021. Two investigators independently searched on PubMed, systematically reviewed, and agreed on the eligibility of peer-reviewed publications corresponding to such RCTs. This process was aimed at identifying a sample of peer-reviewed and published RCTs characterized by ample mediatic exposure, likely representing cutting-edge research in the field and impacting on practice, with less emphasis on studies with a lower potential for dissemination and publication in top-tier journals. The search strategy is described in detail in the appendix (see “Supplementary Methods 1” in the electronic supplementary material).

The studies were included if they were (1) RCTs of drugs or devices, (2) active- or placebo-controlled, (3) powered for superiority, non-inferiority, or equivalence, and (4) published in a peer-reviewed medical journal. Sub-studies and sub-analyses of RCTs were excluded, as well as RCTs of treatment strategies. RCTs encompassing two co-primary endpoints were included as a single entity, as they correspond to a presentation and its corresponding publication. Follow-up extensions of RCTs were included only if they were pre-specified by the original trial statistical plan. Studies selected for inclusion were grouped based on the objective of their investigations (i.e., drugs or devices).

2.2 Data Collection

For each study analyzed, we collected baseline characteristics and measures of conduct and publication from the published manuscripts, main clinical trials registries, and cardiovascular society websites (i.e., ACC, AHA, ESC, and TCT) [18,19,20,21].

The impact of each study was quantified using Altmetric Explorer and "dimensions" services to obtain the number of mentions and citations, respectively, following research authorization [22]. Total, 1-year, and 2-year citations (e.g., in the published literature) and total and 1-month mentions (e.g., in social media and news outlets) were collected. Because Altmetric Explorer monitor article citations by calendar year, when citations spanned 2 calendar years, we employed a weighted aggregation method to ensure methodological consistency while accounting for temporal overlaps.

2.3 Endpoints

The primary endpoint was the total number of citations. Secondary endpoints included the number of citations at 1 and 2 years and the number of study mentions (i.e., total and at 1 month), as well as several measures of study conduct and publication (i.e., number of studies stopped prematurely, number of studies with neutral results, time elapsed between study registration and publication, time from presentation to publication, revision time, publication in journals by impact factor quartile, number of ahead-of-print and simultaneous publications, and the number of articles accompanied by an editorial).

2.4 Statistical Analysis

Continuous variables are reported as means and standard deviations or medians with interquartile ranges (IQRs) based on data distribution, and were compared with t Student or Mann–Whitney tests, as appropriate. Categorical variables are reported as frequencies and percentages, and were compared with χ2 or Fisher’s exact tests, as appropriate. Counts for missing baseline characteristics are reported alongside the results.

Multivariable regression analyses, including linear regression for continuous variables and logistic regression for categorical variables, were conducted to explore the effect of drug versus device RCTs after accounting for several predefined confounders, including simultaneous publication, revision time, publication in journals in the first impact factor quartile, neutral results, studies stopped prematurely, sample size, superiority design, double-blind methodology, multicenter studies, industry funding, time from registration to publication, and the presence of a hard primary endpoint. Additionally, the interaction of the study endpoints with the outcome of the study (e.g., the primary endpoint was met or not) was explored. We also examined differences in the incidence of the primary endpoint before and during the coronavirus disease 2019 (COVID-19) pandemic (i.e., ≤ 2019 vs. ≥ 2020) as a potential factor that could alter the conduct and publication patterns of randomized trials. All analyses were conducted at a two-sided 5% significance level and performed with SPSS (Statistical Package for the Social Sciences). The study is reported in accordance with STROBE guidelines for reporting of observational research (see Supplementary Table 1 in the electronic supplementary material).

3 Results

3.1 Study Population

After screening of the selected medical conferences, 955 RCTs were identified, and 382 matched the eligibility criteria. Reasons for exclusion are reported in Fig. 1. A total of 227 studies (59.4%) were RCTs of drugs and 155 (40.6%) were RCTs of devices.

Fig. 1
figure 1

Study flow chart. ACC American College of Cardiology, AHA American Heart Association, ESC European Society of Cardiology, LBS Late-breaking science, TCT Transcatheter Cardiovascular Therapeutics

Full size image

The characteristics of the study population are detailed in Table 1. Compared to RCTs of devices, RCTs of drugs more frequently had a superiority design (84.1 vs. 63.9%; p < 0.001), were double-blinded (58.1 vs. 7.7%; p < 0.001), and had a larger median sample of patients (535 [136–1432] vs. 480 [249–1602] patients; p = 0.003). Furthermore, they more frequently had a sponsor involved in study design (50.4 vs. 34.9%; p = 0.003) or analysis (36.3 vs. 23.3%; p = 0.008).

Table 1 Baseline characteristics of drug or device randomized trials in cardiovascular medicine
Full size table

Conversely, device trials more frequently had a non-inferiority design (41.3 vs. 17.2%; p < 0.001), were open-label (71.0 vs. 37.4%; p < 0.001), were single-blinded (21.3 vs. 4.4%; p < 0.001), and were performed in a single continent (59.0 vs. 48.9%; p = 0.050). Also, they were more frequently presented in the main auditorium (80.2 vs. 65.6%; p = 0.021) at the time of congress presentation.

3.2 Measures of Study Impact

Measures of study impact in drug versus device RCTs are displayed in Figs. 2, 3, and Table 2. Compared to device RCTs, RCTs of drugs received a higher median number of total citations (93 [IQR 48–137] vs. 82 [IQR 39–192]; p = 0.025) and total mentions (60 [IQR 16–151] vs. 33 [16–73]; p < 0.001). Similarly, they had a higher number of citations at 1 and 2 years and mentions at 1 month. The total number of citations obtained from trials published before the COVID-19 pandemic showed no significant difference between drug and device trials [198 (IQR 91–477) vs. 186 (IQR 60–433); p = 0.123]. However, accounting for those published during the pandemic favored drug trials over device trials [126 (IQR 40–302) vs. 95 (IQR 28–119); p = 0.008].

Fig. 2
figure 2

Number of citations obtained by drug or device trials in cardiovascular medicine at different timeframes. The box plot illustrates the distribution of total citations, 1-year citations, and 2-year citations for randomized trials in cardiovascular medicine, categorized by therapeutic intervention (i.e., devices or drugs). The lower and upper edges of each box represent the first and third quartiles, respectively, and the line within the box marks the median. Outliers are represented by diamonds. The data suggest that studies focusing on drugs tend to have a higher median number of citations across all timepoints evaluated

Full size image
Fig. 3
figure 3

Number of mentions obtained by drug or device trials in cardiovascular medicine at different timeframes. The box plot illustrates the distribution of total and 1-month mentions for randomized trials in cardiovascular medicine, categorized by therapeutic intervention (i.e., devices or drugs). The lower and upper edges of each box represent the first and third quartiles, respectively, and the line within the box marks the median. Outliers are represented by diamonds. The data suggest that studies focusing on drugs tend to have a higher median number of mentions across the evaluated timepoints

Full size image
Table 2 Impact of device versus drugs randomized trials in cardiovascular medicine
Full size table

However, after adjusting for potential confounders, there were no significant associations between drug versus device RCTs and number of citations and mentions (total and at different timepoints). Simultaneous publication was the only independent predictor of total citations (adjusted B = 53.6, 95% CI 0.779–106.6; adjusted p = 0.047) (Table 3).

Table 3 Univariable and multivariable regression analyses among selected variables and significant study endpoints
Full size table

3.3 Measures of Study Conduct and Publication

Measures of study conduct and publication are listed in Table 2 and Fig. 4. Drug RCTs were more frequently stopped prematurely compared to device RCTs (8.8 vs. 1.9%; p = 0.006) (Fig. 3). No significant differences were observed in median time from presentation to publication, time from registration to publication, and revision time. No differences were also observed in the proportion of studies with neutral results (31.6% for devices vs. 37.4% for drugs studies; p = 0.241). Additionally, there were similar proportions of RCTs published in journals of the first quartile, with an accompanying editorial, ahead of print, and published simultaneously.

Fig. 4
figure 4

Measures of study conduct and publication across randomized controlled trials in cardiovascular medicine. The bar charts illustrate the prevalence of categorical measures of study conduct and publication categorized by therapeutic intervention (i.e., devices or drugs). Similarly, box plots illustrate the distribution of continuous measures of study conduct and publication. The lower and upper edges of each box represent the first and third quartiles, respectively, and the line within the box marks the median. Outliers are represented by diamonds. Data indicated by an asterisk (*) labels variables whose statistical significance persisted after adjustment

Full size image

After adjusting for potential confounders, drug RCTs were significantly more likely to be stopped prematurely (adjusted odds ratio [OR], 1.15; 95% confidence interval [CI] 1.03–1.28; p = 0.009).

3.4 Sensitivity Analyses

Sensitivity analyses are detailed in Table 4. Compared to device RCTs, drug RCTs that met the primary endpoint were associated with a significant increase in total citations (B = 185.6; 95% CI 18.7–352.6; p = 0.029), 1-year citations (B = 54.4; 95% CI 19.6–89.2; p = 0.002), and 2-year citations (B = 112.8; 95% CI 32.0–193.7; p = 0.006). The interaction effect for these outcomes between studies that did and did not meet the primary endpoint was statistically significant (p for interaction 0.049, 0.016, and 0.014, respectively). Drug trials that met the primary endpoint were also more likely to be stopped prematurely (OR 1.05; 95% CI 1.00–1.10; p = 0.033).

Table 4 Regression analysis examining the influence of study outcomes in drugs vs. devices comparison with interactions for study results
Full size table

4 Discussion

This study sheds light on the varying characteristics and impacts of device versus drug RCTs in cardiovascular medicine. Key results can be summarized as follows. Firstly, drug RCTs received more citations and mentions than device RCTs, but this difference was no longer statistically significant after adjusting for potential confounders. Secondly, regardless of being drug or device RCTs, studies with simultaneous publication were more likely to gain citations. Thirdly, drug RCTs were more frequently stopped prematurely than device RCTs. Lastly, drug RCTs with positive outcomes were more likely to attain citations compared to those reporting neutral outcomes.

In this study, drug and device RCTs exhibited distinctive baseline characteristics. For example, drug trials often adopted superiority designs, were double-blinded, had larger sample sizes, and had higher sponsor involvement in design and analysis. Conversely, device RCTs more likely had non-inferiority designs, were more often open-label, and were usually conducted by investigators within the same continent. These differences typically reflect challenges and requirements that are specific to the nature of devices versus drugs [7]. This includes general complexities like blinding, varying sample size based on required power for the primary endpoint, challenges in affiliating enrolling centers with expert operators across different countries, and influence of sponsor involvement in ensuring rigorous trial design and analysis [23]. Specific complexities can arise, for example, when designing non-inferiority trials, because they require the determination of the type and magnitude of the non-inferiority margin. In fact, studies that adopt absolute metrics and terminate with lower-than-expected event rates have sometimes led to different outcomes when recalculating non-inferiority margins based on actual event rates in cardiovascular trials. In a study of margin recalculation, device trials more frequently reported absolute metrics compared to drug trials [24]. This implies that authors, reviewers, and editors should be aware of the risks related to the choice of the non-inferiority margin when designing, reporting, interpreting, and commenting on non-inferiority trials, particularly if absolute metrics are used or in the case of a large discrepancy between observed and anticipated event rates in the control arm [25].

Importantly, non-significant differences emerged in the vast majority of baseline characteristics, such as funding by industries or institutions, among the two groups. After accounting for these modifiers, there was no difference in the citations of drug versus device RCTs. Simultaneous publication was the only independent predictor of total citation count, which is consistent with prior literature [26, 27]. When stratifying the data for the primary endpoint by the COVID-19 pandemic timeline, trials published during the pandemic period showed higher citation counts for drug trials. In contrast, studies published before the pandemic, consistent with our main analysis, exhibited no significant differences in citation counts. This continues to highlight the COVID-19 period as a significant factor affecting trial conduct and impact for multiple aspects [28, 29].

Secondary measures of conduct and publication were similar between drug and device RCTs, indicating consistent research design and dissemination principles across diverse therapeutic trials, notwithstanding differing regulatory pathways. This consistency underscores the overarching impact of standardized research methodologies, ethical guidelines, and publication practices in the scientific research landscape, transcending specific regulatory contexts [7, 30,31,32,33]. Nonetheless, the higher frequency of premature cessation in drug studies, particularly those halted for harm, aligns with existing literature and may signify heightened regulatory scrutiny and safety standards in drug RCTs [34]. On the other hand, trials stopped early for benefit often overestimate treatment effects by concluding prematurely when results appear significantly favorable, potentially ignoring long-term effects or rare adverse outcomes. These exaggerated results can lead to rapid incorporation into clinical guidelines and widespread use of therapies that may not be as effective or could be harmful. Experts call caution against ending trials early based on preliminary benefits alone, recommending that guidelines acknowledge potential uncertainties and overestimations [35].

The finding that drug trials with positive outcomes were more likely to attain citations compared to those with neutral outcomes is consistent with prior evidence, and numerous potential explanations exist for this finding, including citation bias—although this concept is subject to debate [16, 36]. A study of 24,000 sponsored RCTs emphasized the complex nature of trial success, including quality, speed, setting, and communication as positive determinants of study citations [16]. However, other factors related to the likelihood of a trial’s success, which may be influenced by the type of sponsor (i.e., industry or institutions), comprehend the inclusion rate of these studies (e.g., the percentage of subjects enrolled after initial screening). Investigating the inclusion rate could provide insights into the efficiency and transparency of recruitment strategies, which are essential for ensuring the generalizability and validity of study findings. Additionally, understanding how inclusion rates vary across different types of trials and funding sources could reveal disparities in participant recruitment and inform strategies to optimize this process. Importantly, this information is often not reported in a significant number of studies, and investigations into this issue are inadequate, thus necessitating more attention in future research [37].

To the best of our knowledge, this is the first study to assess the characteristics and impact of device or drug RCTs in cardiovascular medicine. The significance of our findings extends potentially beyond cardiovascular medicine, providing insights into the comparative analysis of device and drug RCTs in other disciplines. Furthermore, our study offers a better understanding of how certain study design aspects influence the dissemination and impact of research findings.

On the other hand, our study has some limitations. Firstly, although we assumed that a larger number of citations and mentions is a proxy for broader acceptance of the study results by the scientific community, we did not investigate other important aspects of the impact of device versus drug RCTs. These include the generalizability of results (e.g., external validity of study findings in randomized and non-randomized settings), the inclusion of trials in top-tier meta-analyses and their impact on the cumulative evidence, the influence of trial findings on clinical practice guidelines with strength of the connected recommendations, as well as any changes in recommendation class in case of confirmatory trials. These factors, crucial for a comprehensive understanding of a trial's impact beyond mere citation counts, should be considered in future research. Secondly, we cannot determine whether and to what extent other unidentified characteristics that also contribute to the number of citations and mentions may have influenced the study findings. For instance, studies that address a critical gap in knowledge might gain more impact compared to those investigating well-explored fields. Additionally, factors such as the open-access status of publications or whether trials continued as open-label after their formal closure leading to further publications might have provided broader access and visibility. Ultimately, our study only focused on a timeframe of 7 years due to the absence of good quality data before 2015. However, this is a quite wide timeframe that enables the inclusion of an adequate number of RCTs while avoiding excess heterogeneity stemming from temporal bias.

5 Conclusions

Drug RCTs are more likely to be stopped prematurely and have more citations and mentions than device RCTs, but this difference is largely attributed to several confounding factors other than the intervention itself. Studies with simultaneous publication are more likely to gain citations and mentions, and drug RCTs with positive results obtain more citations than those with neutral results. These findings highlight the influence of dissemination strategies and study outcomes in determining the impact of RCTs.