Advertisement

Journal of General Internal Medicine

, Volume 33, Issue 6, pp 786–788 | Cite as

What Constitutes an Independent Statistical Analysis?

  • Ann Abraham
  • Rosa Ahn
  • Alexandra Woodbridge
  • Erin Madden
  • Salomeh Keyhani
  • Deborah KorensteinEmail author
Concise Research Reports

KEY WORDS

independent statistical analysis evidence-based medicine randomized-controlled trials 

BACKGROUND

Potential bias in clinical trials related to relationships with the pharmaceutical industry is a longstanding concern.1 Between 2005 and 2013, JAMA journals required industry-sponsored studies to conduct independent statistical analysis (ISA), defined as data analysis by an “independent statistician at an academic institution” using the raw data set.2 While no journals currently require ISA, the term may be used to denote impartiality and robustness in data analysis.2, 3 However, its meaning, frequency of use, and association with study characteristics are not clear. Our study’s purpose was to investigate the prevalence and characteristics of ISA in published RCTs focused on drug efficacy and their adherence to JAMA’s definition.

METHODS

We searched MEDLINE and randomly selected 646 drug efficacy RCTs from 2013, as described previously;4 190 met inclusion criteria. Two of four reviewers (AA, RA, AW, SS) independently abstracted data regarding trial characteristics, clinical area, results, funding source, investigator/manufacturer financial ties, and description of ISA or independent statistician. Among papers reporting ISA, we abstracted in duplicate information concerning the analysis the sponsor’s relationship to data and analyses, and statistician(s) identity. Disagreements were resolved by consensus. When ISA was described, we determined conformity with its definitional components (academic statistician affiliation and use of the full dataset) and the relationship between ISA and study characteristics and outcome. We used the Mann–Whitney test for continuous variables and Chi-squared for categorical variables (SAS, V9).

RESULTS

Statistical Analysis Characteristics

Among the 190 trials, 17 (8.9%) reported ISA; the majority (15, 88%) were industry-funded and published in high impact journals (IF > 10) (12, 71%) (Table 1). Most identified the independent statistician(s) by name (11, 65%). Roles of independent statisticians varied; they led the analysis in eight trials (47%), validated the sponsors analysis in four (24%), provided statistical assistance in three (18%), and had an unspecified role in two (12%). ISA adhered to both components of the definition in seven trials (41%); independent analysts had academic affiliation in 13 trials (76%) and full dataset access in 11 (65%).
Table 1

Prevalence of Independent Statistical Analysis by Trial Characteristics (N = 195)

 

N

Independent statistical analysis present, N (%)

Independent statistical analysis absent, N (%)

p value

Outcome

 Positive

136

10 (7.4)

126 (92.6)

0.31

 Negative

59

7 (11.9)

52 (88.1)

 

Funding source

 Any industry funding

134

15 (11.2)

119 (88.8)

0.069

 No industry funding

61

2 (3.3)

59 (96.7)

 

Financial ties

 Financial ties present

132

13 (9.8)

119 (90.2)

0.42

 Financial ties absent

63

4 (6.3)

59 (93.7)

 

Impact factor

  ≥ 10

100

13 (13.0)

87 (87.0)

0.03*

  < 10

95

4 (4.2)

91 (95.8)

 

RCT phase

 Phase 3

102

10 (9.8)

92 (90.2)

0.57

 Other

93

7 (7.5)

86 (92.5)

 

RCT type

 Double-blinded

147

12 (8.2)

135 (91.8)

0.63

 Other

48

5 (10.4)

43 (89.6)

 

Sample size

 Q1 (13–118)

49

1 (2.0)

48 (98.0)

< 0.001

 Q2 (119–315)

49

2 (4.1)

47 (95.9)

 

 Q3 (316–615)

49

3 (6.1)

46 (93.9)

 

 Q4 (616–21,105)

48

11 (22.9)

37 (77.1)

 

Clinical area

 Cardiology

31

9 (29.0)

22 (71.0)

< 0.001

 Oncology

22

2 (9.1)

20 (90.9)

 

 Other specialties

142

6 (4.2)

136 (95.8)

 

Trial registration

 Yes

184

17 (9.2)

167 (93.8)

0.29

 No

11

0 (0)

11 (100.0)

 

Type of analysis

 Superiority

174

14 (8.0)

160 (92.0)

0.34

 Non-inferiority

21

3 (14.3)

18 (85.7)

 

Comparator

 Placebo

146

13 (8.9)

133 (91.1)

0.87

 Active

49

4 (8.2)

45 (91.8)

 

Outcome measure

 Surrogate

65

4 (6.2)

61 (93.8)

0.37

 Clinical

130

13 (10.0)

117 (90.0)

 

First author

 US

74

4 (5.4)

70 (94.6)

0.20

 Other

121

13 (10.7)

108 (89.3)

 

*p value based on continuous pooled variance test: impact factor, 0.0029; sample size, < 0.0001

Papers which described an independent statistical analysis

1. Oettle H, Neuhaus P, Hochhaus A et al. Adjuvant Chemotherapy With Gemcitabine and Long-term Outcomes Among Patients With Resected Pancreatic CancerThe CONKO-001 Randomized Trial. JAMA.2013;310(14):1473–1481. doi: https://doi.org/10.1001/jama.2013.279201

2. Nicholls SJ, Bakris GL, Kastelein JJP et al. Effect of Aliskiren on Progression of Coronary Disease in Patients With PrehypertensionThe AQUARIUS Randomized Clinical Trial. JAMA. 2013;310(11):1135–1144. doi: https://doi.org/10.1001/jama.2013.277169

3. Opal SM, Laterre P, Francois B et al. Effect of Eritoran, an Antagonist of MD2-TLR4, on Mortality in Patients With Severe SepsisThe ACCESS Randomized Trial. JAMA. 2013;309(11):1154–1162. doi: https://doi.org/10.1001/jama.2013.2194

4. Steg PG, Mehta SR, Pollack CV et al. Anticoagulation With Otamixaban and Ischemic Events in Non–ST-Segment Elevation Acute Coronary SyndromesThe TAO Randomized Clinical Trial. JAMA. 2013;310(11):1145–1155. doi: https://doi.org/10.1001/jama.2013.277165

5. Steg PG, van ‘t Hof A, Hamm CW, et al. Bivalirudin started during emergency transport for primary PCI. N Engl J Med 2013;369:2207–2217

6. de Zeeuw D, Akizawa T, Audhya P, et al. Bardoxolone methyl in type 2 diabetes and stage 4 chronic kidney disease. N Engl J Med 2013;369:2492–2503

7. Armstrong PW, Gershlick AH, Goldstein P, et al. Fibrinolysis or primary PCI in ST-segment elevation myocardial infarction. N Engl J Med 2013;368:1379–1387

8. Kosiborod M, Arnold SV, Spertus JA, et al. Evaluation of Ranolazine in Patients With Type 2 Diabetes Mellitus and Chronic Stable Angina. Journal of the American College of Cardiology. 2013;61(20):2038–2045. doi: https://doi.org/10.1016/j.jacc.2013.02.011

9. Burton LA, Sumukadas D, Witham MD, Struthers AD, Mcmurdo ME. Effect of Spironolactone on Physical Performance in Older People with Self-reported Physical Disability. The American Journal of Medicine. 2013;126(7):590–597. doi: https://doi.org/10.1016/j.amjmed.2012.11.032

10. Gisslinger H, Gotic M, Holowiecki J, et al. Anagrelide compared with hydroxyurea in WHO-classified essential thrombocythemia: the ANAHYDRET Study, a randomized controlled trial. Blood. 2013;121(10):1720–1728. doi: https://doi.org/10.1182/blood-2012-07-443,770

11. Parfrey PS, Chertow GM, Block GA, et al. The Clinical Course of Treated Hyperparathyroidism Among Patients Receiving Hemodialysis and the Effect of Cinacalcet: The EVOLVE Trial. The Journal of Clinical Endocrinology & Metabolism. 2013;98(12):4834–4844. doi: https://doi.org/10.1210/jc.2013-2975

12. Banai S, Finkelstein A, Almagor Y, et al. Targeted anti-inflammatory systemic therapy for restenosis: The Biorest Liposomal Alendronate with Stenting sTudy (BLAST)—a double blind, randomized clinical trial. American Heart Journal. 2012;165(2). doi: https://doi.org/10.1016/j.ahj.2012.10.023

13. Youn YJ, Lee J-W, Ahn SG, et al. Multicenter randomized trial of 3-month cilostazol use in addition to dual antiplatelet therapy after biolimus-eluting stent implantation for long or multivessel coronary artery disease. American Heart Journal. 2013;167(2). doi: https://doi.org/10.1016/j.ahj.2013.08.028

14. Zhu F-C, Liang Z-L, Li X-L, et al. Immunogenicity and safety of an enterovirus 71 vaccine in healthy Chinese children and infants: a randomised, double-blind, placebo-controlled phase 2 clinical trial. The Lancet. 2013;381(9871):1037–1045. doi: https://doi.org/10.1016/s0140-6736(12)61764-4

15. Giugliano RP, Ruff CT, Braunwald E et al. Edoxaban versus warfarin in patients with atrial fibrillation. The New England Journal of Medicine. 2013; 369(22):2093–104. doi:  https://doi.org/10.1056/NEJMoa1310907

16. Ryan CJ, Smith MR, de Bono JS et al. Abiraterone in Metastatic Prostate Cancer without Previous Chemotherapy. New England Journal of Medicine. 2013;368(6):584–584. doi: https://doi.org/10.1056/nejmx130004

19. Pulido T, Adzerikho I, Channick RN et al. Macitentan and morbidity and mortality in pulmonary arterial hypertension. New England Journal of Medicine. 2013; 369(9):809–18. doi:  https://doi.org/10.1056/NEJMoa1213917

Relationship to Study Characteristics

ISA was not associated with industry funding (p value = 0.07), positive study outcome (p value = 0.31), or financial ties to the manufacturer (p value = 0.42). ISA was strongly associated with sample size (p value < 0.0001) and clinical area (p value < 0.001), notably cardiology. ISA was not associated with trial registration, analysis type, phase, comparator, outcome measure, or first author country (Table 1).

DISCUSSION

We found that drug efficacy RCTs rarely self-reported ISA, though the term was used more commonly in large, industry-funded studies published in high impact journals. The meaning of ISA varied among trials with some statisticians controlling the analysis and others serving as collaborators or consultants.

In the past, JAMA clearly defined ISA and required it to ensure integrity and minimize bias,3 but this requirement resulted in fewer manuscript submissions by industry and was dropped.5, 6 Regardless, the term remains in use. Our findings demonstrate ambiguity around its meaning, possibly resulting in an unwarranted implication of rigor and integrity. Given this ambiguity, readers of the literature should not assume that ISA represents methodological rigor. Instead, readers concerned about the integrity of data analysis should note details of the identity, role, and affiliation of authors or statisticians performing analyses and their involvement in the protocol and access to data. Similarly, editors with concerns about the integrity of data analysis should ask authors for transparency regarding these issues.

Our study was limited by the low prevalence of ISA and may not have been powered to detect significant differences in study characteristics. In particular, there was a trend toward an association between ISA and industry funding that we may have been underpowered to detect.

In conclusion, while the term independent statistical analysis is used in scientific literature, its meaning varies across studies and it may be incorrectly associated with data integrity. Given the lack of consensus around its meaning, transparency regarding statisticians’ roles and access to the primary data may be better options for ensuring the integrity of the literature.

Notes

Acknowledgements

The authors would like to acknowledge Susan Saba for assisting with data collection.

Authors Contribution

SK and DK conceived the study. SK, DK, AA, RA, and AW created the study design. AA, RA, AW, and SS collected the data. EM, SK, AA, RA, AW, and DK analyzed and interpreted the data. AA, RA, SK, AW, DK, and EM wrote and revised the manuscript. All authors critically revised the manuscript and approved the final version for submission. DK supervised the study and is the guarantor.

Funding

This project was not directly supported by any research funds. Dr. Keyhani is funded by grants from the NIH grants (RO1 HL116522-01A1, RO1 HL114563-01A1) and VA HSR&D (1IP1HX001994). Dr. Korenstein’s work on this paper was supported by a Cancer Center Support Grant from the National Cancer Institute to Memorial Sloan Kettering Cancer Center (award number P30 CA008748).

Compliance with Ethical Standards

Conflict of Interest

The authors declare that they have no conflict of interest.

Ethical Approval

Not needed

Data Sharing

Dataset available from corresponding author on request.

Transparency

The manuscript’s guarantor (DK) affirms that this manuscript is an honest, accurate, and transparent account of the study being reported that no important aspects of the study have been omitted and that any discrepancies from the study as planned have been explained.

This is an open-access article distributed under the terms of the Creative Commons Attribution Non-commercial License, which permits use, distribution, and reproduction in any medium, provided the original work is properly cited, the use is noncommercial and is otherwise in compliance with the license. See: http://creativecommons.org/licenses/by-nc/2.0/ and http://creativecommons.org/licenses/by-nc/2.0/legalcode.

References

  1. 1.
    Lexchin J, Bero LA, Djulbegovic B, Clark O. Pharmaceutical industry sponsorship and research outcome and quality: systematic review. BMJ: British Medical Journal. 2003;326(7400):1167.CrossRefGoogle Scholar
  2. 2.
    Responsibilities in the Submission and Peer-Review Process. (2017). Retrieved May 26, 2017, from http://www.icmje.org/recommendations/browse/roles-and-responsibilities/responsibilities-in-the-submission-and-peer-peview-process.html.
  3. 3.
    Fontanarosa PB, Flanagin A, DeAngelis CD. (2005) Reporting Conflicts of Interest, Financial Aspects of Research, and Role of Sponsors in Funded Studies. JAMA;294(1):110–111. doi: https://doi.org/10.1001/jama.294.1.110 CrossRefPubMedGoogle Scholar
  4. 4.
    Ahn R, Woodbridge A, Abraham A. (2017). Financial ties of principal investigators and randomized controlled trial outcomes: cross sectional study. BMJ, I6770. doi: https://doi.org/10.1136/bmj.i6770
  5. 5.
    Wager E, Mhaskar R, Warburton S, Djulbegovic B (2010) JAMA Published Fewer Industry-Funded Studies after Introducing a Requirement for Independent Statistical Analysis. PLoS ONE 5(10): e13591. doi  https://doi.org/10.1371/journal.pone.0013591 CrossRefPubMedPubMedCentralGoogle Scholar
  6. 6.
    Lundh A, Krogsbøll LT, Gøtzsche PC (2012). Sponsors’ participation in conduct and reporting of industry trials: a descriptive study. Trials, 13(1). doi: https://doi.org/10.1186/1745-6215-13-146

Copyright information

© Society of General Internal Medicine 2018

Authors and Affiliations

  • Ann Abraham
    • 1
  • Rosa Ahn
    • 2
  • Alexandra Woodbridge
    • 3
  • Erin Madden
    • 1
  • Salomeh Keyhani
    • 1
    • 4
  • Deborah Korenstein
    • 5
    Email author
  1. 1.San Francisco VA Medical CenterSan FranciscoUSA
  2. 2.Oregon Health and Science UniversityPortlandUSA
  3. 3.Tulane University School of MedicineNew OrleansUSA
  4. 4.University of CaliforniaSan FranciscoUSA
  5. 5.Memorial Sloan Kettering Cancer CenterNew YorkUSA

Personalised recommendations