Skip to main content
SpringerLink
Log in

Evaluating performance of electronic healthcare records and spontaneous reporting data in drug safety signal detection

  • Research Article
  • Published:
International Journal of Clinical Pharmacy Aims and scope Submit manuscript

Abstract

Background Electronic reporting and processing of suspected adverse drug reactions (ADRs) is increasing and has facilitated automated screening procedures. It is crucial for healthcare professionals to understand the nature and proper use of data available in pharmacovigilance practice. Objectives To (a) compare performance of EU-ADR [electronic healthcare record (EHR) exemplar] and FAERS [spontaneous reporting system (SRS) exemplar] databases in detecting signals using “positive” and “negative” drug-event reference sets; and (b) evaluate the impact of timing bias on sensitivity thresholds by comparing all data to data restricted to the time before a warning/regulatory action. Methods Ten events with known positive and negative reference sets were selected. Signals were identified when respective statistics exceeded defined thresholds. Main outcome measure Performance metrics, including sensitivity, specificity, positive predictive value and accuracy were calculated. In addition, the effect of regulatory action on the performance of signal detection in each data source was evaluated. Results The sensitivity for detecting signals in EHR data varied depending on the nature of the adverse events and increased substantially if the analyses were restricted to the period preceding the first regulatory action. Across all events, using data from all years, a sensitivity of 45–73 % was observed for EU-ADR and 77 % for FAERS. The specificity was high and similar for EU-ADR (82–96 %) and FAERS (98 %). EU-ADR data showed range of PPV (78–91 %) and accuracy (78–72 %) and FAERS data yielded a PPV of 97 % with 88 % accuracy. Conclusion Using all cumulative data, signal detection in SRS data achieved higher specificity and sensitivity than EHR data. However, when data were restricted to time prior to a regulatory action, performance characteristics changed in a manner consistent with both the type of data and nature of the ADR. Further research focusing on prospective validation of is necessary to learn more about the performance and utility of these databases in modern pharmacovigilance practice.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2

Similar content being viewed by others

References

  1. Department of Health and Human Services (US), Food and Drug Administration, Center for Drug Evaluation and Research, Center for Biologics Evaluation and Research. Guidance for Industry: premarketing Risk Assessment; March 2005.

  2. International Conference on Harmonization of Technical Requirements for Registration of Pharmaceuticals for Human Use. Guideline for Good Clinical Practice E6(R1); June 1996.

  3. Department of Health and Human Services (US), Food and Drug Administration, Center for Drug Evaluation and Research, Center for Biologics Evaluation and Research. Guidance for Industry: E6 Good Clinical Practice: consolidated Guidance; April 1996.

  4. Council for International Organizations of Medical Sciences. Management of Safety Information from Clinical Trials: report of CIOMS Working Group VI; CIOMS; April 2005.

  5. Department of Health and Human Services (US), Food and Drug Administration, Center for Drug Evaluation and Research, Center for Biologics Evaluation and Research. Good Pharmacovigilance Practices and Pharmacoepidemiologic Assessment; March 2005.

  6. European Medicines Agency. Volume 9A: guidelines on Pharmacovigilance for Medicinal Products for Human Use. In: The Rules governing medicinal products in the European Union: London (UK); September 2008.

  7. Council for International Organizations of Medical Sciences. Practical Aspects of Signal Detection in Pharmacovigilance: report of CIOMS Working Group VIII; CIOMS; September 2010.

  8. Hauben M, Aronson JK. Defining ‘signal’ and its subtypes in pharmacovigilance based on a systematic review of previous definitions. Drug Saf. 2009;32(2):99–110.

    Article  PubMed  Google Scholar 

  9. Szarfman A, Machado SG, O’Neill RT. Use of screening algorithms and computer systems to efficiently signal higher-than-expected combinations of drugs and events in the US FDA’s spontaneous reports database. Drug Saf. 2002;25(6):381–92.

    Article  CAS  PubMed  Google Scholar 

  10. Szarfman A, Tonning JM, Doraiswamy PM. Pharmacovigilance in the 21st century: new systematic tools for an old problem. Pharmacotherapy. 2004;24(9):1099–104.

    Article  PubMed  Google Scholar 

  11. Bailey S, Singh A, Azadian R, Huber P, Blum M. Prospective data mining of six products in the US FDA Adverse Event Reporting System: disposition of events identified and impact on product safety profiles. Drug Saf. 2010;33(2):139–46.

    Article  PubMed  Google Scholar 

  12. Bate A, Lindquist M, Orre R, Edwards I, Meyboom R. Data-mining analyses of pharmacovigilance signals in relation to relevant comparison drugs. Eur J Clin Pharmacol. 2002;58(7):483–90.

    Article  CAS  PubMed  Google Scholar 

  13. Edwards IR, Star K, Kiuru A. Statins, neuromuscular degenerative disease and an amyotrophic lateral sclerosis-like syndrome: an analysis of individual case safety reports from vigibase. Drug Saf. 2007;30(6):515–25.

    Article  PubMed  Google Scholar 

  14. McAdams M, Staffa J, Dal Pan G. Estimating the extent of reporting to FDA: a case study of statin-associated rhabdomyolysis. Pharmacoepidemiol Drug Saf. 2008;17(3):229–39.

    Article  CAS  PubMed  Google Scholar 

  15. Hauben M, Reich L, DeMicco J, Kim K. ‘Extreme duplication’ in the US FDA Adverse Events Reporting System database. Drug Saf. 2007;30(6):551–4.

    Article  PubMed  Google Scholar 

  16. Norén GN, Hopstadius J, Bate A, Edwards IR. Safety surveillance of longitudinal databases: methodological considerations. Pharmacoepidemiol Drug Saf. 2011;20(7):714–7.

    Article  PubMed  Google Scholar 

  17. Curtis JR, Cheng H, Delzell E, Fram D, Kilgore M, Saag K, et al. Adaptation of Bayesian data mining algorithms to longitudinal claims data: coxib safety as an example. Med Care. 2008;46(9):969–75.

    Article  PubMed Central  PubMed  Google Scholar 

  18. Hartzema AG, Racoosin JA, MaCurdy TE, Gibbs JM, Kelman JA. Utilizing Medicare claims data for real-time drug safety evaluations: is it feasible? Pharmacoepidemiol Drug Saf. 2011;20(7):684–8.

    Article  PubMed  Google Scholar 

  19. FDAs Sentinel Initiative [Internet]. US Department of Health and Human Services, Food and Drug Administration; Jun 06 2014 [cited 17 Sept 2014]. http://www.fda.gov/Safety/FDAsSentinelinitiative/.

  20. Zorych I, Madigan D, Ryan P, Bate A. Disproportionality methods for pharmacovigilance in longitudinal observational databases. Stat Methods Med Res. 2013;22(1):39–56.

    Article  PubMed  Google Scholar 

  21. Ryan PB, Madigan D, Stang PE, Overhage JM, Racoosin JA, Hartzema AG. Empirical assessment of methods for risk identification in healthcare data: results from the experiments of the Observational Medical Outcomes Partnership. Stat Med. 2012;31(30):4401–15.

    Article  PubMed  Google Scholar 

  22. Coloma PM, Schuemie MJ, Trifirò G, Gini R, Herings R, Hippisley-Cox J, et al. Combining electronic healthcare databases in Europe to allow for large-scale drug safety monitoring: the EU-ADR Project. Pharmacoepidemiol Drug Saf. 2011;20(1):1–11.

    Article  PubMed  Google Scholar 

  23. Ishiguro C, Hinomura Y, Uemura K, Matsuda T. Analysis of the factors influencing the spontaneous reporting frequency of drug safety issues addressed in the FDA’s drug safety communications, using FAERS data. Pharm Med. 2014;28(1):7–19.

    Article  CAS  Google Scholar 

  24. Trifirò G, Pariente A, Coloma PM, Kors JA, Polimeni G, Miremont-Salamé G, et al. Data mining on electronic health record databases for signal detection in pharmacovigilance: which events to monitor? Pharmacoepidemiol Drug Saf. 2009;18(12):1176–84.

    Article  PubMed  Google Scholar 

  25. Coloma PM, Avillach P, Salvo F, Schuemie MJ, Ferrajolo C, Pariente A, et al. A reference standard for evaluation of methods for drug safety signal detection using electronic healthcare record databases. Drug Saf. 2013;36(1):13–23.

    Article  CAS  PubMed  Google Scholar 

  26. Avillach P, Coloma PM, Gini R, Schuemie M, Mougin F, Dufour JC, et al. Harmonization process for the identification of medical events in eight European healthcare databases: the experience from the EU-ADR project. J Am Med Inform Assoc. 2013;20(1):184–92.

    Article  PubMed Central  PubMed  Google Scholar 

  27. FDA Adverse Event Reporting System (FAERS) [Internet]. US Department of Health and Human Services, Food and Drug Administration; Sep 10 2012 [cited 9 May 2014]. http://www.fda.gov/Drugs/GuidanceComplianceRegulatoryInformation/Surveillance/AdverseDrugEffects.

  28. Schuemie MJ. Methods for drug safety signal detection in longitudinal observational databases: LGPS and LEOPARD. Pharmacoepidemiol Drug Saf. 2011;20(3):292–9.

    Article  CAS  PubMed  Google Scholar 

  29. Graham DJ, Ouellet-Hellstrom R, MaCurdy TE, Ali F, Sholley C, Worrall C, et al. Risk of acute myocardial infarction, stroke, heart failure, and death in elderly Medicare patients treated with rosiglitazone or pioglitazone. JAMA. 2010;304(4):411–8.

    Article  CAS  PubMed  Google Scholar 

  30. Nissen SE, Wolski K. Rosiglitazone revisited: an updated meta-analysis of risk for myocardial infarction and cardiovascular mortality. Arch Intern Med. 2010;170(14):1191–201.

    Article  CAS  PubMed  Google Scholar 

Download references

Acknowledgments

The authors would like to thank the EU-ADR consortium and the EU-ADR database owners for their contribution and support, particularly Rosa Gini (Agenzia Regionale Sanità Toscana, Florence, Italy), Ron Herings (PHARMO Institute, Utrecht, Netherlands), Giampiero Mazzaglia (Società Italiana Medicina Generale, Florence, Italy), Gino Picelli (Pedianet, Padova, Italy), Carla Fornari (Department of Statistics and Quantitative Methods, Università Milano-Bicocca, Milan, Italy), and Lars Pedersen (Aarhus University Hospital, Aarhus, Denmark) and Colette Saccomanno for editorial assistance.

Funding

This project has received funding from the European Commission’s Seventh Framework Programme (FP7/2007-2013) under Grant Agreement No 215847. The funding agency has not been involved in the collection of data, the analysis or interpretation of the data, or the decision to submit.

Conflicts of interest

The authors do not declare any conflict of interest.

Author information

Authors and Affiliations

  1. Department of Medical Informatics, Erasmus University Medical Center, Rotterdam, The Netherlands

    Vaishali K. Patadia, Martijn J. Schuemie, Preciosa Coloma, Johan van der Lei, Miriam Sturkenboom & Gianluca Trifirò

  2. Astellas Pharma Global Development Inc., Northbrook, IL, USA

    Vaishali K. Patadia

  3. PHARMO Institute, Utrecht, The Netherlands

    Ron Herings

  4. Department of Epidemiology, Erasmus Medical Center, Rotterdam, The Netherlands

    Sabine Straus & Miriam Sturkenboom

  5. Medicines Evaluation Board, Utrecht, The Netherlands

    Sabine Straus

  6. Department of Clinical and Experimental Medicine, University of Messina, Messina, Italy

    Gianluca Trifirò

Authors
  1. Vaishali K. Patadia
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Martijn J. Schuemie
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Preciosa Coloma
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Ron Herings
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Johan van der Lei
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Sabine Straus
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Miriam Sturkenboom
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Gianluca Trifirò
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Vaishali K. Patadia.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (DOCX 46 kb)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Patadia, V.K., Schuemie, M.J., Coloma, P. et al. Evaluating performance of electronic healthcare records and spontaneous reporting data in drug safety signal detection. Int J Clin Pharm 37, 94–104 (2015). https://doi.org/10.1007/s11096-014-0044-5

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11096-014-0044-5

Keywords

Search

Navigation