Original Research Article

Drug Safety

, Volume 36, Supplement 1, pp 143-158

First online:

A Comparison of the Empirical Performance of Methods for a Risk Identification System

  • Patrick B. RyanAffiliated withJanssen Research and Development LLCObservational Medical Outcomes Partnership, Foundation for the National Institutes of Health Email author 
  • , Paul E. StangAffiliated withJanssen Research and Development LLCObservational Medical Outcomes Partnership, Foundation for the National Institutes of Health
  • , J. Marc OverhageAffiliated withSiemens Health ServicesObservational Medical Outcomes Partnership, Foundation for the National Institutes of Health
  • , Marc A. SuchardAffiliated withDepartments of Biomathematics and Human Genetics, David Geffen School of Medicine at UCLA, University of CaliforniaDepartment of Biostatistics, UCLA Fielding School of Public Health, University of CaliforniaObservational Medical Outcomes Partnership, Foundation for the National Institutes of Health
  • , Abraham G. HartzemaAffiliated withCollege of Pharmacy, University of FloridaObservational Medical Outcomes Partnership, Foundation for the National Institutes of Health
  • , William DuMouchelAffiliated withOracle Health SciencesObservational Medical Outcomes Partnership, Foundation for the National Institutes of Health
  • , Christian G. ReichAffiliated withAstraZenecaObservational Medical Outcomes Partnership, Foundation for the National Institutes of Health
  • , Martijn J. SchuemieAffiliated withDepartment of Medical Informatics, Erasmus University Medical Center RotterdamObservational Medical Outcomes Partnership, Foundation for the National Institutes of Health
  • , David MadiganAffiliated withDepartment of Statistics, Columbia UniversityObservational Medical Outcomes Partnership, Foundation for the National Institutes of Health

Rent the article at a discount

Rent now

* Final gross prices may vary according to local VAT.

Get Access

Abstract

Background

Observational healthcare data offer the potential to enable identification of risks of medical products, and the medical literature is replete with analyses that aim to accomplish this objective. A number of established analytic methods dominate the literature but their operating characteristics in real-world settings remain unknown.

Objectives

To compare the performance of seven methods (new user cohort, case control, self-controlled case series, self-controlled cohort, disproportionality analysis, temporal pattern discovery, and longitudinal gamma poisson shrinker) as tools for risk identification in observational healthcare data.

Research Design

The experiment applied each method to 399 drug-outcome scenarios (165 positive controls and 234 negative controls across 4 health outcomes of interest) in 5 real observational databases (4 administrative claims and 1 electronic health record).

Measures

Method performance was evaluated through Area Under the receiver operator characteristics Curve (AUC), bias, mean square error, and confidence interval coverage probability.

Results

Multiple methods offer strong predictive accuracy, with AUC > 0.70 achievable for all outcomes and databases with more than one analytical approach. Self-controlled methods (self-controlled case series, temporal pattern discovery, self-controlled cohort) had higher predictive accuracy than cohort and case–control methods across all databases and outcomes. Methods differed in the expected value and variance of the error distribution. All methods had lower coverage probability than the expected nominal properties.

Conclusions

Observational healthcare data can inform risk identification of medical product effects on acute liver injury, acute myocardial infarction, acute renal failure and gastrointestinal bleeding. However, effect estimates from all methods require calibration to address inconsistency in method operating characteristics. Further empirical evaluation is required to gauge the generalizability of these findings to other databases and outcomes.