BACKGROUND

Potential bias in clinical trials related to relationships with the pharmaceutical industry is a longstanding concern.1 Between 2005 and 2013, JAMA journals required industry-sponsored studies to conduct independent statistical analysis (ISA), defined as data analysis by an “independent statistician at an academic institution” using the raw data set.2 While no journals currently require ISA, the term may be used to denote impartiality and robustness in data analysis.2, 3 However, its meaning, frequency of use, and association with study characteristics are not clear. Our study’s purpose was to investigate the prevalence and characteristics of ISA in published RCTs focused on drug efficacy and their adherence to JAMA’s definition.

METHODS

We searched MEDLINE and randomly selected 646 drug efficacy RCTs from 2013, as described previously;4 190 met inclusion criteria. Two of four reviewers (AA, RA, AW, SS) independently abstracted data regarding trial characteristics, clinical area, results, funding source, investigator/manufacturer financial ties, and description of ISA or independent statistician. Among papers reporting ISA, we abstracted in duplicate information concerning the analysis the sponsor’s relationship to data and analyses, and statistician(s) identity. Disagreements were resolved by consensus. When ISA was described, we determined conformity with its definitional components (academic statistician affiliation and use of the full dataset) and the relationship between ISA and study characteristics and outcome. We used the Mann–Whitney test for continuous variables and Chi-squared for categorical variables (SAS, V9).

RESULTS

Statistical Analysis Characteristics

Among the 190 trials, 17 (8.9%) reported ISA; the majority (15, 88%) were industry-funded and published in high impact journals (IF > 10) (12, 71%) (Table 1). Most identified the independent statistician(s) by name (11, 65%). Roles of independent statisticians varied; they led the analysis in eight trials (47%), validated the sponsors analysis in four (24%), provided statistical assistance in three (18%), and had an unspecified role in two (12%). ISA adhered to both components of the definition in seven trials (41%); independent analysts had academic affiliation in 13 trials (76%) and full dataset access in 11 (65%).

Table 1 Prevalence of Independent Statistical Analysis by Trial Characteristics (N = 195)
Full size table

Relationship to Study Characteristics

ISA was not associated with industry funding (p value = 0.07), positive study outcome (p value = 0.31), or financial ties to the manufacturer (p value = 0.42). ISA was strongly associated with sample size (p value < 0.0001) and clinical area (p value < 0.001), notably cardiology. ISA was not associated with trial registration, analysis type, phase, comparator, outcome measure, or first author country (Table 1).

DISCUSSION

We found that drug efficacy RCTs rarely self-reported ISA, though the term was used more commonly in large, industry-funded studies published in high impact journals. The meaning of ISA varied among trials with some statisticians controlling the analysis and others serving as collaborators or consultants.

In the past, JAMA clearly defined ISA and required it to ensure integrity and minimize bias,3 but this requirement resulted in fewer manuscript submissions by industry and was dropped.5, 6 Regardless, the term remains in use. Our findings demonstrate ambiguity around its meaning, possibly resulting in an unwarranted implication of rigor and integrity. Given this ambiguity, readers of the literature should not assume that ISA represents methodological rigor. Instead, readers concerned about the integrity of data analysis should note details of the identity, role, and affiliation of authors or statisticians performing analyses and their involvement in the protocol and access to data. Similarly, editors with concerns about the integrity of data analysis should ask authors for transparency regarding these issues.

Our study was limited by the low prevalence of ISA and may not have been powered to detect significant differences in study characteristics. In particular, there was a trend toward an association between ISA and industry funding that we may have been underpowered to detect.

In conclusion, while the term independent statistical analysis is used in scientific literature, its meaning varies across studies and it may be incorrectly associated with data integrity. Given the lack of consensus around its meaning, transparency regarding statisticians’ roles and access to the primary data may be better options for ensuring the integrity of the literature.