Skip to main content

Evaluation of Disproportionality Safety Signaling Applied to Healthcare Databases

Drug Safety volume 36, pages 123–132 (2013)Cite this article

Abstract

Objective

To evaluate the performance of a disproportionality design, commonly used for analysis of spontaneous reports data such as the FDA Adverse Event Reporting System database, as a potential analytical method for an adverse drug reaction risk identification system using healthcare data.

Research Design

We tested the disproportionality design in 5 real observational healthcare databases and 6 simulated datasets, retrospectively studying the predictive accuracy of the method when applied to a collection of 165 positive controls and 234 negative controls across 4 outcomes: acute liver injury, acute myocardial infarction, acute kidney injury, and upper gastrointestinal bleeding.

Measures

We estimate how well the method can be expected to identify true effects and discriminate from false findings and explore the statistical properties of the estimates the design generates. The primary measure was the area under the curve (AUC) of the receiver operating characteristic (ROC) curve.

Results

For each combination of 4 outcomes and 5 databases, 48 versions of disproportionality analysis (DPA) were carried out and the AUC computed. The majority of the AUC values were in the range of 0.35 < AUC < 0.6, which is considered to be poor predictive accuracy, since the value AUC = 0.5 would be expected from mere random assignment. Several DPA versions achieved AUC of about 0.7 for the outcome Acute Renal Failure within the GE database. The overall highest DPA version across all 20 outcome-database combinations was the Bayesian Information Component method with no stratification by age and gender, using first occurrence of outcome and with assumed time-at-risk equal to duration of exposure + 30d, but none were uniformly optimal. The relative risk estimates for the negative control drug-event combinations were very often biased either upward or downward by a factor of 2 or more. Coverage probabilities of confidence intervals from all methods were far below nominal.

Conclusions

The disproportionality methods that we evaluated did not discriminate true positives from true negatives using healthcare data as they seem to do using spontaneous report data.

This is a preview of subscription content, access via your institution.

Access options

Buy single article

Instant access to the full article PDF.

USD 49.95

Price includes VAT (USA)
Tax calculation will be finalised during checkout.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig.5

References

  1. Mantel N, Haenszel W. Statistical aspects of the analysis of data from retrospective studies of disease. J Natl Cancer Inst. 1959;22(4):719–48.

    PubMed  CAS  Google Scholar 

  2. Almenoff JS, Pattishall EN, Gibbs TG, DuMouchel W, Evans SJ, Yuen N. Novel statistical tools for monitoring the safety of marketed drugs. Clin Pharmacol Ther. 2007;82(2):157–66.

    PubMed  Article  CAS  Google Scholar 

  3. DuMouchel W, Pregibon D. Empirical Bayes screening for multi-item associations. In: Proceedings of the seventh ACM SIGKDD international conference on Knowledge discovery and data mining. San Francisco: ACM; 2001. p. 67–76.

  4. Dumouchel W. Bayesian data mining in large frequency tables, with an application to the FDA spontaneous reporting system. Am Stat. 1999;53(3):177–90.

    Google Scholar 

  5. Norén GN, Hopstadius J, Bate A. Shrinkage observed-to-expected ratios for robust and transparent large-scale pattern discovery. Stat Methods Med Res 2013;22(1):57–69.

    PubMed  Article  Google Scholar 

  6. Hauben M, Madigan D, Gerrits CM, Walsh L, Van Puijenbroek EP. The role of data mining in pharmacovigilance. Expert Opin Drug Saf. 2005;4(5):929–48.

    PubMed  Article  CAS  Google Scholar 

  7. Hauben M, Reich L. Safety related drug-labelling changes: findings from two data mining algorithms. Drug Saf. 2004;27(10):735–44.

    PubMed  Article  Google Scholar 

  8. Szarfman A, Machado SG, O’Neill RT. Use of screening algorithms and computer systems to efficiently signal higher-than-expected combinations of drugs and events in the US FDA’s spontaneous reports database. Drug Saf. 2002;25(6):381–92.

    PubMed  Article  CAS  Google Scholar 

  9. Zorych I, Madigan D, Ryan P, Bate A. Disproportionality methods for pharmacovigilance in longitudinal observational databases. Stat Methods Med Res. 2013;22(1):39–56.

    Google Scholar 

  10. Evans SJ, Waller PC, Davis S. Use of proportional reporting ratios (PRRs) for signal generation from spontaneous adverse drug reaction reports. Pharmacoepidemiol Drug Saf. 2001;10(6):483–6.

    PubMed  Article  CAS  Google Scholar 

  11. Bate A, Lindquist M, Edwards IR, Olsson S, Orre R, Lansner A, et al. A Bayesian neural network method for adverse drug reaction signal generation. Eur J Clin Pharmacol. 1998;54(4):315–21.

    PubMed  Article  CAS  Google Scholar 

  12. Ryan PB, Schuemie M. Evaluating performance of risk identification methods through a large-scale simulation of observational data. Drug Saf. 2013 (in this supplement issue). doi:10.1007/s40264-013-0110-2.

  13. Overhage JM, Ryan PB, Schuemie MJ, Stang PE. Desideratum for evidence based epidemiology. Drug Saf. 2013 (in this supplement issue). doi:10.1007/s40264-013-0102-2.

  14. Hartzema AG, Reich CG, Ryan PB, Stang PE, Madigan D, Welebob E, et al. Managing data quality for a drug safety surveillance system. Drug Saf. 2013 (in this supplement issue). doi:10.1007/s40264-013-0098-7.

  15. Cantor SB, Kattan MW. Determining the area under the ROC curve for a binary diagnostic test. Med Decis Making. 2000;20(4):468–70.

    PubMed  Article  CAS  Google Scholar 

  16. Fram DM, Almenoff JS, DuMouchel W. Empirical Bayesian data mining for discovering patterns in post-marketing drug safety. In: Proceedings of the Ninth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. Washington, DC: ACM; 2003. p. 359–68.

  17. Almenoff JS, DuMouchel W, Kindman LA, Yang X, Fram D. Disproportionality analysis using empirical Bayes data mining: a tool for the evaluation of drug interactions in the post-marketing setting. Pharmacoepidemiol Drug Saf. 2003;12(6):517–21.

    PubMed  Article  Google Scholar 

  18. DuMouchel W, Smith ET, Beasley R, Nelson H, Yang X, Fram D, et al. Association of asthma therapy and Churg-Strauss syndrome: an analysis of postmarketing surveillance data. Clin Ther. 2004;26(7):1092–104.

    PubMed  Article  Google Scholar 

  19. Almenoff JS, LaCroix KK, Yuen NA, Fram D, DuMouchel W. Comparative performance of two quantitative safety signalling methods: implications for use in a pharmacovigilance department. Drug Saf. 2006;29(10):875–87.

    PubMed  Article  CAS  Google Scholar 

  20. Solomon R, Dumouchel W. Contrast media and nephropathy: findings from systematic analysis and Food and Drug Administration reports of adverse effects. Invest Radiol. 2006;41(8):651–60.

    PubMed  Article  Google Scholar 

  21. DuMouchel W, Fram D, Yang X, Mahmoud RA, Grogg AL, Engelhart L, et al. Antipsychotics, glycemic disorders, and life-threatening diabetic events: a Bayesian data-mining analysis of the FDA adverse event reporting system (1968-2004). Ann Clin Psychiatry. 2008;20(1):21–31.

    PubMed  Article  Google Scholar 

  22. Harpaz R, DuMouchel W, LePendu P, Bauer-Mehren A, Ryan P, Shah NH. Performance of pharmacovigilance signal-detection algorithms for the FDA adverse event reporting system. Clin Pharmacol Ther. 2013;93(6):539–46.

    Google Scholar 

  23. Schuemie MJ, Coloma PM, Straatman H, et al. Using electronic healthcare records for drug safety signal detection; a comparative evaluation of statistical methods. Med Care. 2012;50(10):890–7.

    PubMed  Article  Google Scholar 

Download references

Acknowledgments

The Observational Medical Outcomes Partnership is funded by the Foundation for the National Institutes of Health (FNIH) through generous contributions from the following: Abbott, Amgen Inc., AstraZeneca, Bayer Healthcare Pharmaceuticals, Inc., Biogen Idec, Bristol-Myers Squibb, Eli Lilly & Company, GlaxoSmithKline, Janssen Research and Development, Lundbeck, Inc., Merck & Co., Inc., Novartis Pharmaceuticals Corporation, Pfizer Inc, Pharmaceutical Research Manufacturers of America (PhRMA), Roche, Sanofi-aventis, Schering-Plough Corporation, and Takeda. Dr. DuMouchel is an employee of Oracle Health Sciences. Drs. Ryan and Schuemie are employees of Janssen Research and Development. Dr. Schuemie received a fellowship from the Office of Medical Policy, Center for Drug Evaluation and Research, Food and Drug Administration. Drs. Schuemie and Madigan have received grants from FNIH.

This article was published in a supplement sponsored by the Foundation for the National Institutes of Health (FNIH). The supplement was guest edited by Stephen J.W. Evans. It was peer reviewed by Olaf H. Klungel who received a small honorarium to cover out-of-pocket expenses. S.J.W.E has received travel funding from the FNIH to travel to the OMOP symposium and received a fee from FNIH for the review of a protocol for OMOP. O.H.K has received funding for the IMI-PROTECT project.from the Innovative Medicines Initiative Joint Undertaking (http://www.imi.europa.eu) under Grant Agreement no 115004, resources of which are composed of financial contribution from the European Union’s Seventh Framework Programme (FP7/2007-2013) and EFPIA companies’ in kind contribution.

Author information

Authors and Affiliations

  1. Oracle Health Sciences, Burlington, Massachusetts, USA

    William DuMouchel

  2. Janssen Research and Development LLC, Titusville, NJ, USA

    Patrick B. Ryan

  3. Department of Medical Informatics, Erasmus University Medical Center Rotterdam, Rotterdam, The Netherlands

    Martijn J. Schuemie

  4. Department of Statistics, Columbia University, New York, NY, USA

    David Madigan

  5. Oracle Health Sciences Global Business Unit, 7164 N Mercer Spring Pl, Tucson, AZ, 85718, USA

    William DuMouchel

  6. Observational Medical Outcomes Partnership, Foundation for the National Institutes of Health, Bethesda, MD, USA

    William DuMouchel, Patrick B. Ryan, Martijn J. Schuemie & David Madigan

Authors
  1. William DuMouchel
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Patrick B. Ryan
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Martijn J. Schuemie
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. David Madigan
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to William DuMouchel.

Additional information

The OMOP research used data from Truven Health Analytics (formerly the Health Business of Thomson Reuters), and includes MarketScan® Research Databases, represented with MarketScan Lab Supplemental (MSLR, 1.2 m persons), MarketScan Medicare Supplemental Beneficiaries (MDCR, 4.6 m persons), MarketScan Multi-State Medicaid (MDCD, 10.8 m persons), MarketScan Commercial Claims and Encounters (CCAE, 46.5 m persons). Data also provided by Quintiles® Practice Research Database (formerly General Electric’s Electronic Health Record, 11.2 m persons) database. GE is an electronic health record database while the other four databases contain administrative claims data.

Appendix

Appendix

figure a

Disproportionality estimates for all test cases, by database. MSLR MarketScan Lab Supplemental, MDCD MarketScan Multi‐state Medicaid, MDCR MarketScan Medicare Supplemental Beneficiaries, CCAE MarketScan Commercial Claims and Encounters, GE GE centricity. Blue negative controls, Orange positive controls; each line represents point estimate and 95 %

Rights and permissions

Reprints and Permissions

About this article

Cite this article

DuMouchel, W., Ryan, P.B., Schuemie, M.J. et al. Evaluation of Disproportionality Safety Signaling Applied to Healthcare Databases. Drug Saf 36, 123–132 (2013). https://doi.org/10.1007/s40264-013-0106-y

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s40264-013-0106-y

Keywords

  • Sitagliptin
  • Coverage Probability
  • Acute Liver Injury
  • Proportional Reporting Ratio
  • True Effect Size