Drug Safety

, Volume 36, Supplement 1, pp 59–72

Empirical Performance of a New User Cohort Method: Lessons for Developing a Risk Identification and Analysis System

  • Patrick B. Ryan
  • Martijn J. Schuemie
  • Susan Gruber
  • Ivan Zorych
  • David Madigan
Original Research Article

DOI: 10.1007/s40264-013-0099-6

Cite this article as:
Ryan, P.B., Schuemie, M.J., Gruber, S. et al. Drug Saf (2013) 36(Suppl 1): 59. doi:10.1007/s40264-013-0099-6

Abstract

Background

Observational healthcare data offer the potential to enable identification of risks of medical products, but appropriate methodology has not yet been defined. The new user cohort method, which compares the post-exposure rate among the target drug to a referent comparator group, is the prevailing approach for many pharmacoepidemiology evaluations and has been proposed as a promising approach for risk identification but its performance in this context has not been fully assessed.

Objectives

To evaluate the performance of the new user cohort method as a tool for risk identification in observational healthcare data.

Research Design

The method was applied to 399 drug-outcome scenarios (165 positive controls and 234 negative controls across 4 health outcomes of interest) in 5 real observational databases (4 administrative claims and 1 electronic health record) and in 6 simulated datasets with no effect and injected relative risks of 1.25, 1.5, 2, 4, and 10, respectively.

Measures

Method performance was evaluated through Area Under ROC Curve (AUC), bias, and coverage probability.

Results

The new user cohort method achieved modest predictive accuracy across the outcomes and databases under study, with the top-performing analysis near AUC >0.70 in most scenarios. The performance of the method was particularly sensitive to the choice of comparator population. For almost all drug-outcome pairs there was a large difference, either positive or negative, between the true effect size and the estimate produced by the method, although this error was near zero on average. Simulation studies showed that in the majority of cases, the true effect estimate was not within the 95 % confidence interval produced by the method.

Conclusion

The new user cohort method can contribute useful information toward a risk identification system, but should not be considered definitive evidence given the degree of error observed within the effect estimates. Careful consideration of the comparator selection and appropriate calibration of the effect estimates is required in order to properly interpret study findings.

Copyright information

© Springer International Publishing Switzerland 2013

Authors and Affiliations

  • Patrick B. Ryan
    • 1
    • 2
    • 6
  • Martijn J. Schuemie
    • 3
    • 6
  • Susan Gruber
    • 4
  • Ivan Zorych
    • 6
  • David Madigan
    • 5
    • 6
  1. 1.Janssen Research and Development LLCTitusvilleUSA
  2. 2.TitusvilleUSA
  3. 3.Department of Medical InformaticsErasmus University Medical Center RotterdamRotterdamThe Netherlands
  4. 4.Harvard School of Public HealthCambridgeUSA
  5. 5.Department of StatisticsColumbia UniversityNew YorkUSA
  6. 6.Observational Medical Outcomes PartnershipFoundation for the National Institutes of HealthBethesdaUSA

Personalised recommendations