Drug Safety

, Volume 36, Supplement 1, pp 143–158

A Comparison of the Empirical Performance of Methods for a Risk Identification System

  • Patrick B. Ryan
  • Paul E. Stang
  • J. Marc Overhage
  • Marc A. Suchard
  • Abraham G. Hartzema
  • William DuMouchel
  • Christian G. Reich
  • Martijn J. Schuemie
  • David Madigan
Original Research Article

Abstract

Background

Observational healthcare data offer the potential to enable identification of risks of medical products, and the medical literature is replete with analyses that aim to accomplish this objective. A number of established analytic methods dominate the literature but their operating characteristics in real-world settings remain unknown.

Objectives

To compare the performance of seven methods (new user cohort, case control, self-controlled case series, self-controlled cohort, disproportionality analysis, temporal pattern discovery, and longitudinal gamma poisson shrinker) as tools for risk identification in observational healthcare data.

Research Design

The experiment applied each method to 399 drug-outcome scenarios (165 positive controls and 234 negative controls across 4 health outcomes of interest) in 5 real observational databases (4 administrative claims and 1 electronic health record).

Measures

Method performance was evaluated through Area Under the receiver operator characteristics Curve (AUC), bias, mean square error, and confidence interval coverage probability.

Results

Multiple methods offer strong predictive accuracy, with AUC > 0.70 achievable for all outcomes and databases with more than one analytical approach. Self-controlled methods (self-controlled case series, temporal pattern discovery, self-controlled cohort) had higher predictive accuracy than cohort and case–control methods across all databases and outcomes. Methods differed in the expected value and variance of the error distribution. All methods had lower coverage probability than the expected nominal properties.

Conclusions

Observational healthcare data can inform risk identification of medical product effects on acute liver injury, acute myocardial infarction, acute renal failure and gastrointestinal bleeding. However, effect estimates from all methods require calibration to address inconsistency in method operating characteristics. Further empirical evaluation is required to gauge the generalizability of these findings to other databases and outcomes.

Copyright information

© Springer International Publishing Switzerland 2013

Authors and Affiliations

  • Patrick B. Ryan
    • 1
    • 10
  • Paul E. Stang
    • 1
    • 10
  • J. Marc Overhage
    • 2
    • 10
  • Marc A. Suchard
    • 3
    • 4
    • 10
  • Abraham G. Hartzema
    • 5
    • 10
  • William DuMouchel
    • 6
    • 10
  • Christian G. Reich
    • 7
    • 10
  • Martijn J. Schuemie
    • 8
    • 10
  • David Madigan
    • 9
    • 10
  1. 1.Janssen Research and Development LLCTitusvilleUSA
  2. 2.Siemens Health ServicesMalvernUSA
  3. 3.Departments of Biomathematics and Human Genetics, David Geffen School of Medicine at UCLAUniversity of CaliforniaLos AngelesUSA
  4. 4.Department of Biostatistics, UCLA Fielding School of Public HealthUniversity of CaliforniaLos AngelesUSA
  5. 5.College of PharmacyUniversity of FloridaGainesvilleUSA
  6. 6.Oracle Health SciencesBurlingtonUSA
  7. 7.AstraZenecaWalthamUSA
  8. 8.Department of Medical InformaticsErasmus University Medical Center RotterdamRotterdamThe Netherlands
  9. 9.Department of StatisticsColumbia UniversityNew YorkUSA
  10. 10.Observational Medical Outcomes Partnership, Foundation for the National Institutes of HealthBethesdaUSA

Personalised recommendations