Advertisement

International Journal of Clinical Pharmacy

, Volume 37, Issue 1, pp 94–104 | Cite as

Evaluating performance of electronic healthcare records and spontaneous reporting data in drug safety signal detection

  • Vaishali K. PatadiaEmail author
  • Martijn J. Schuemie
  • Preciosa Coloma
  • Ron Herings
  • Johan van der Lei
  • Sabine Straus
  • Miriam Sturkenboom
  • Gianluca Trifirò
Research Article
First Online:

Abstract

Background Electronic reporting and processing of suspected adverse drug reactions (ADRs) is increasing and has facilitated automated screening procedures. It is crucial for healthcare professionals to understand the nature and proper use of data available in pharmacovigilance practice. Objectives To (a) compare performance of EU-ADR [electronic healthcare record (EHR) exemplar] and FAERS [spontaneous reporting system (SRS) exemplar] databases in detecting signals using “positive” and “negative” drug-event reference sets; and (b) evaluate the impact of timing bias on sensitivity thresholds by comparing all data to data restricted to the time before a warning/regulatory action. Methods Ten events with known positive and negative reference sets were selected. Signals were identified when respective statistics exceeded defined thresholds. Main outcome measure Performance metrics, including sensitivity, specificity, positive predictive value and accuracy were calculated. In addition, the effect of regulatory action on the performance of signal detection in each data source was evaluated. Results The sensitivity for detecting signals in EHR data varied depending on the nature of the adverse events and increased substantially if the analyses were restricted to the period preceding the first regulatory action. Across all events, using data from all years, a sensitivity of 45–73 % was observed for EU-ADR and 77 % for FAERS. The specificity was high and similar for EU-ADR (82–96 %) and FAERS (98 %). EU-ADR data showed range of PPV (78–91 %) and accuracy (78–72 %) and FAERS data yielded a PPV of 97 % with 88 % accuracy. Conclusion Using all cumulative data, signal detection in SRS data achieved higher specificity and sensitivity than EHR data. However, when data were restricted to time prior to a regulatory action, performance characteristics changed in a manner consistent with both the type of data and nature of the ADR. Further research focusing on prospective validation of is necessary to learn more about the performance and utility of these databases in modern pharmacovigilance practice.

Keywords

Adverse drug reactions Data mining Electronic health records EU-ADR FAERS Performance metrics Pharmacovigilance Signal detection Spontaneous reporting systems 
This is a preview of subscription content, log in to check access.

Notes

Acknowledgments

The authors would like to thank the EU-ADR consortium and the EU-ADR database owners for their contribution and support, particularly Rosa Gini (Agenzia Regionale Sanità Toscana, Florence, Italy), Ron Herings (PHARMO Institute, Utrecht, Netherlands), Giampiero Mazzaglia (Società Italiana Medicina Generale, Florence, Italy), Gino Picelli (Pedianet, Padova, Italy), Carla Fornari (Department of Statistics and Quantitative Methods, Università Milano-Bicocca, Milan, Italy), and Lars Pedersen (Aarhus University Hospital, Aarhus, Denmark) and Colette Saccomanno for editorial assistance.

Funding

This project has received funding from the European Commission’s Seventh Framework Programme (FP7/2007-2013) under Grant Agreement No 215847. The funding agency has not been involved in the collection of data, the analysis or interpretation of the data, or the decision to submit.

Conflicts of interest

The authors do not declare any conflict of interest.

Supplementary material

11096_2014_44_MOESM1_ESM.docx (46 kb)
Supplementary material 1 (DOCX 46 kb)

References

  1. 1.
    Department of Health and Human Services (US), Food and Drug Administration, Center for Drug Evaluation and Research, Center for Biologics Evaluation and Research. Guidance for Industry: premarketing Risk Assessment; March 2005.Google Scholar
  2. 2.
    International Conference on Harmonization of Technical Requirements for Registration of Pharmaceuticals for Human Use. Guideline for Good Clinical Practice E6(R1); June 1996.Google Scholar
  3. 3.
    Department of Health and Human Services (US), Food and Drug Administration, Center for Drug Evaluation and Research, Center for Biologics Evaluation and Research. Guidance for Industry: E6 Good Clinical Practice: consolidated Guidance; April 1996.Google Scholar
  4. 4.
    Council for International Organizations of Medical Sciences. Management of Safety Information from Clinical Trials: report of CIOMS Working Group VI; CIOMS; April 2005.Google Scholar
  5. 5.
    Department of Health and Human Services (US), Food and Drug Administration, Center for Drug Evaluation and Research, Center for Biologics Evaluation and Research. Good Pharmacovigilance Practices and Pharmacoepidemiologic Assessment; March 2005.Google Scholar
  6. 6.
    European Medicines Agency. Volume 9A: guidelines on Pharmacovigilance for Medicinal Products for Human Use. In: The Rules governing medicinal products in the European Union: London (UK); September 2008.Google Scholar
  7. 7.
    Council for International Organizations of Medical Sciences. Practical Aspects of Signal Detection in Pharmacovigilance: report of CIOMS Working Group VIII; CIOMS; September 2010.Google Scholar
  8. 8.
    Hauben M, Aronson JK. Defining ‘signal’ and its subtypes in pharmacovigilance based on a systematic review of previous definitions. Drug Saf. 2009;32(2):99–110.PubMedCrossRefGoogle Scholar
  9. 9.
    Szarfman A, Machado SG, O’Neill RT. Use of screening algorithms and computer systems to efficiently signal higher-than-expected combinations of drugs and events in the US FDA’s spontaneous reports database. Drug Saf. 2002;25(6):381–92.PubMedCrossRefGoogle Scholar
  10. 10.
    Szarfman A, Tonning JM, Doraiswamy PM. Pharmacovigilance in the 21st century: new systematic tools for an old problem. Pharmacotherapy. 2004;24(9):1099–104.PubMedCrossRefGoogle Scholar
  11. 11.
    Bailey S, Singh A, Azadian R, Huber P, Blum M. Prospective data mining of six products in the US FDA Adverse Event Reporting System: disposition of events identified and impact on product safety profiles. Drug Saf. 2010;33(2):139–46.PubMedCrossRefGoogle Scholar
  12. 12.
    Bate A, Lindquist M, Orre R, Edwards I, Meyboom R. Data-mining analyses of pharmacovigilance signals in relation to relevant comparison drugs. Eur J Clin Pharmacol. 2002;58(7):483–90.PubMedCrossRefGoogle Scholar
  13. 13.
    Edwards IR, Star K, Kiuru A. Statins, neuromuscular degenerative disease and an amyotrophic lateral sclerosis-like syndrome: an analysis of individual case safety reports from vigibase. Drug Saf. 2007;30(6):515–25.PubMedCrossRefGoogle Scholar
  14. 14.
    McAdams M, Staffa J, Dal Pan G. Estimating the extent of reporting to FDA: a case study of statin-associated rhabdomyolysis. Pharmacoepidemiol Drug Saf. 2008;17(3):229–39.PubMedCrossRefGoogle Scholar
  15. 15.
    Hauben M, Reich L, DeMicco J, Kim K. ‘Extreme duplication’ in the US FDA Adverse Events Reporting System database. Drug Saf. 2007;30(6):551–4.PubMedCrossRefGoogle Scholar
  16. 16.
    Norén GN, Hopstadius J, Bate A, Edwards IR. Safety surveillance of longitudinal databases: methodological considerations. Pharmacoepidemiol Drug Saf. 2011;20(7):714–7.PubMedCrossRefGoogle Scholar
  17. 17.
    Curtis JR, Cheng H, Delzell E, Fram D, Kilgore M, Saag K, et al. Adaptation of Bayesian data mining algorithms to longitudinal claims data: coxib safety as an example. Med Care. 2008;46(9):969–75.PubMedCentralPubMedCrossRefGoogle Scholar
  18. 18.
    Hartzema AG, Racoosin JA, MaCurdy TE, Gibbs JM, Kelman JA. Utilizing Medicare claims data for real-time drug safety evaluations: is it feasible? Pharmacoepidemiol Drug Saf. 2011;20(7):684–8.PubMedCrossRefGoogle Scholar
  19. 19.
    FDAs Sentinel Initiative [Internet]. US Department of Health and Human Services, Food and Drug Administration; Jun 06 2014 [cited 17 Sept 2014]. http://www.fda.gov/Safety/FDAsSentinelinitiative/.
  20. 20.
    Zorych I, Madigan D, Ryan P, Bate A. Disproportionality methods for pharmacovigilance in longitudinal observational databases. Stat Methods Med Res. 2013;22(1):39–56.PubMedCrossRefGoogle Scholar
  21. 21.
    Ryan PB, Madigan D, Stang PE, Overhage JM, Racoosin JA, Hartzema AG. Empirical assessment of methods for risk identification in healthcare data: results from the experiments of the Observational Medical Outcomes Partnership. Stat Med. 2012;31(30):4401–15.PubMedCrossRefGoogle Scholar
  22. 22.
    Coloma PM, Schuemie MJ, Trifirò G, Gini R, Herings R, Hippisley-Cox J, et al. Combining electronic healthcare databases in Europe to allow for large-scale drug safety monitoring: the EU-ADR Project. Pharmacoepidemiol Drug Saf. 2011;20(1):1–11.PubMedCrossRefGoogle Scholar
  23. 23.
    Ishiguro C, Hinomura Y, Uemura K, Matsuda T. Analysis of the factors influencing the spontaneous reporting frequency of drug safety issues addressed in the FDA’s drug safety communications, using FAERS data. Pharm Med. 2014;28(1):7–19.CrossRefGoogle Scholar
  24. 24.
    Trifirò G, Pariente A, Coloma PM, Kors JA, Polimeni G, Miremont-Salamé G, et al. Data mining on electronic health record databases for signal detection in pharmacovigilance: which events to monitor? Pharmacoepidemiol Drug Saf. 2009;18(12):1176–84.PubMedCrossRefGoogle Scholar
  25. 25.
    Coloma PM, Avillach P, Salvo F, Schuemie MJ, Ferrajolo C, Pariente A, et al. A reference standard for evaluation of methods for drug safety signal detection using electronic healthcare record databases. Drug Saf. 2013;36(1):13–23.PubMedCrossRefGoogle Scholar
  26. 26.
    Avillach P, Coloma PM, Gini R, Schuemie M, Mougin F, Dufour JC, et al. Harmonization process for the identification of medical events in eight European healthcare databases: the experience from the EU-ADR project. J Am Med Inform Assoc. 2013;20(1):184–92.PubMedCentralPubMedCrossRefGoogle Scholar
  27. 27.
    FDA Adverse Event Reporting System (FAERS) [Internet]. US Department of Health and Human Services, Food and Drug Administration; Sep 10 2012 [cited 9 May 2014]. http://www.fda.gov/Drugs/GuidanceComplianceRegulatoryInformation/Surveillance/AdverseDrugEffects.
  28. 28.
    Schuemie MJ. Methods for drug safety signal detection in longitudinal observational databases: LGPS and LEOPARD. Pharmacoepidemiol Drug Saf. 2011;20(3):292–9.PubMedCrossRefGoogle Scholar
  29. 29.
    Graham DJ, Ouellet-Hellstrom R, MaCurdy TE, Ali F, Sholley C, Worrall C, et al. Risk of acute myocardial infarction, stroke, heart failure, and death in elderly Medicare patients treated with rosiglitazone or pioglitazone. JAMA. 2010;304(4):411–8.PubMedCrossRefGoogle Scholar
  30. 30.
    Nissen SE, Wolski K. Rosiglitazone revisited: an updated meta-analysis of risk for myocardial infarction and cardiovascular mortality. Arch Intern Med. 2010;170(14):1191–201.PubMedCrossRefGoogle Scholar

Copyright information

© Koninklijke Nederlandse Maatschappij ter bevordering der Pharmacie 2014

Authors and Affiliations

  • Vaishali K. Patadia
    • 1
    • 2
    Email author
  • Martijn J. Schuemie
    • 1
  • Preciosa Coloma
    • 1
  • Ron Herings
    • 3
  • Johan van der Lei
    • 1
  • Sabine Straus
    • 4
    • 5
  • Miriam Sturkenboom
    • 1
    • 4
  • Gianluca Trifirò
    • 1
    • 6
  1. 1.Department of Medical InformaticsErasmus University Medical CenterRotterdamThe Netherlands
  2. 2.Astellas Pharma Global Development Inc.NorthbrookUSA
  3. 3.PHARMO InstituteUtrechtThe Netherlands
  4. 4.Department of EpidemiologyErasmus Medical CenterRotterdamThe Netherlands
  5. 5.Medicines Evaluation BoardUtrechtThe Netherlands
  6. 6.Department of Clinical and Experimental MedicineUniversity of MessinaMessinaItaly

Personalised recommendations

Cite article

Buy options