Skip to main content
SpringerLink
Log in

A Reference Standard for Evaluation of Methods for Drug Safety Signal Detection Using Electronic Healthcare Record Databases

  • Short Communication
  • Published:
Drug Safety Aims and scope Submit manuscript

Abstract

Background

The growing interest in using electronic healthcare record (EHR) databases for drug safety surveillance has spurred development of new methodologies for signal detection. Although several drugs have been withdrawn postmarketing by regulatory authorities after scientific evaluation of harms and benefits, there is no definitive list of confirmed signals (i.e. list of all known adverse reactions and which drugs can cause them). As there is no true gold standard, prospective evaluation of signal detection methods remains a challenge.

Objective

Within the context of methods development and evaluation in the EU-ADR Project (Exploring and Understanding Adverse Drug Reactions by integrative mining of clinical records and biomedical knowledge), we propose a surrogate reference standard of drug-adverse event associations based on existing scientific literature and expert opinion.

Methods

The reference standard was constructed for ten top-ranked events judged as important in pharmacovigilance. A stepwise approach was employed to identify which, among a list of drug-event associations, are well recognized (known positive associations) or highly unlikely (‘negative controls’) based on MEDLINE-indexed publications, drug product labels, spontaneous reports made to the WHO’s pharmacovigilance database, and expert opinion. Only drugs with adequate exposure in the EU-ADR database network (comprising ≈60 million person-years of healthcare data) to allow detection of an association were considered. Manual verification of positive associations and negative controls was independently performed by two experts proficient in clinical medicine, pharmacoepidemiology and pharmacovigilance. A third expert adjudicated equivocal cases and arbitrated any disagreement between evaluators.

Results

Overall, 94 drug-event associations comprised the reference standard, which included 44 positive associations and 50 negative controls for the ten events of interest: bullous eruptions; acute renal failure; anaphylactic shock; acute myocardial infarction; rhabdomyolysis; aplastic anaemia/pancytopenia; neutropenia/agranulocytosis; cardiac valve fibrosis; acute liver injury; and upper gastrointestinal bleeding. For cardiac valve fibrosis, there was no drug with adequate exposure in the database network that satisfied the criteria for a positive association.

Conclusion

A strategy for the construction of a reference standard to evaluate signal detection methods that use EHR has been proposed. The resulting reference standard is by no means definitive, however, and should be seen as dynamic. As knowledge on drug safety evolves over time and new issues in drug safety arise, this reference standard can be re-evaluated.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1

References

  1. Amery WK. Signal generation from spontaneous adverse event reports. Pharmacoepidemiol Drug Saf. 1999;8(2):147–50.

    Article  PubMed  CAS  Google Scholar 

  2. Coloma PM, Trifirò G, Schuemie MJ et al. On behalf of the EU-ADR Consortium. Electronic healthcare databases for active drug safety surveillance: is there enough leverage? Pharmacoepidemiol Drug Saf. Epub 2012 Feb 8.

  3. Hauben M, Reich L. Drug-induced pancreatitis: lessons in data mining. Br J Clin Pharmacol. 2004;58(5):560–2.

    Article  PubMed  Google Scholar 

  4. World Health Organization. Safety of medicines: a guide to detecting and reporting adverse drug reactions [online]. Available from URL: http://whqlibdoc.who.int/hq/2002/WHO_EDM_QSM_2002.2.pdf. Accessed 10 Sep 2011.

  5. Hauben M, Aronson JK. Defining ‘signal’ and its subtypes in pharmacovigilance based on a systematic review of previous definitions. Drug Saf. 2009;32(2):99–110.

    Article  PubMed  Google Scholar 

  6. Trifirò G, Fourrier-Reglat A, Sturkenboom MC, et al. The EU-ADR project: preliminary results and perspective. Stud Health Technol Inform. 2009;148:43–9.

    PubMed  Google Scholar 

  7. Coloma PM, Schuemie MJ, Trifiró G, on behalf of the EU-ADR Consortium, et al. Combining electronic healthcare databases in Europe to allow for large-scale drug safety monitoring: the EU-ADR Project. Pharmacoepidemiol Drug Saf. 2011;20(1):1–11.

    Article  PubMed  Google Scholar 

  8. Trifirò G, Pariente A, Coloma PM. On behalf of the EU-ADR consortium, et al. Data mining on electronic health record databases for signal detection in pharmacovigilance: which events to monitor? Pharmacoepidemiol Drug Saf. 2009;18(12):1176–84.

    Article  PubMed  Google Scholar 

  9. Avillach P, Dufour JC, Diallo G, et al. Design and validation of an automated method to detect known adverse drug reactions in MEDLINE: a contribution to the European EU-ADR project. AMIA 2010 Annual Symposium (2010).

  10. Bodenreider O. The unified medical language system (UMLS): integrating biomedical terminology. Nucleic Acids Res. 2004;32(database issue):D267–70.

    Article  PubMed  CAS  Google Scholar 

  11. European Medicines Agency. European public assessment reports [online]. Available from URL: http://www.ema.europa.eu/ema/index.jsp?curl=/pages/medicines/landing/epar_search.jsp&murl=menus/medicines/medicines.jsp&mid=WC0b01ac058001d125. Accessed 20 Sep 2011.

  12. DailyMed [online]. Available from URL: http://dailymed.nlm.nih.gov/dailymed/about.cfm. Accessed 20 Sep 2011.

  13. Electronic Medicines Compendium (for drugs licensed in the United Kingdom). http://www.medicines.org.uk/emc/. Accessed 20 Sep 2011.

  14. Micromedex. https://www.thomsonhc.com/hcs/librarian/. Accessed 20 Sep 2011.

  15. RxList [online]. Available from URL: http://www.rxlist.com/. Accessed 20 Sep 2011.

  16. Drugbank: open data drug and drug target database [online]. Available from URL: http://www.drugbank.ca/. Accessed 20 Sep 2011.

  17. Trifirò G, Patadia V, Schuemie MJ, et al. EU-ADR healthcare database network vs. spontaneous reporting system database: preliminary comparison of signal detection. Stud Health Technol Inform. 2011;166:25–30.

    PubMed  Google Scholar 

  18. DuMouchel W, Smith ET, Beasley R, et al. Association of asthma therapy and Churg-Strauss syndrome: an analysis of postmarketing surveillance data. Clin Ther. 2004;26(7):1092–104.

    Article  PubMed  Google Scholar 

  19. Edwards IR, Aronson JK. Adverse drug reactions: definitions, diagnosis, and management. Lancet. 2000;356(9237):1255–9.

    Article  PubMed  CAS  Google Scholar 

  20. Kennedy DT, Hayney MS, Lake KD. Azathioprine and allopurinol: the price of an avoidable drug interaction. Ann Pharmacother. 1996;30(9):951–4.

    PubMed  CAS  Google Scholar 

  21. Black N. Why we need observational studies to evaluate the effectiveness of health care. BMJ. 1996;312(7040):1215–8.

    Article  PubMed  CAS  Google Scholar 

  22. Papanikolaou PN, Christidi GD, Ioannidis JP. Comparison of evidence on harms of medical interventions in randomized and nonrandomized studies. CMAJ. 2006;174(5):635–41.

    PubMed  Google Scholar 

  23. Stricker BH, Psaty BM. Detection, verification, and quantification of adverse drug reactions. BMJ. 2004;329(7456):44–7.

    Article  PubMed  Google Scholar 

  24. Ray W. Population-based studies of adverse drug effects. N Engl J Med. 2003;349(17):1592–4.

    Article  PubMed  CAS  Google Scholar 

  25. Lindquist M, Ståhl M, Bate A, et al. A retrospective evaluation of a data mining approach to aid finding new adverse drug reaction signals in the WHO international database. Drug Saf. 2000;23(6):533–42.

    Article  PubMed  CAS  Google Scholar 

  26. Hauben M, Reich L. Safety related drug-labelling changes: findings from two data mining algorithms [published erratum appears in Drug Saf 2006; 29 (12): 1191]. Drug Saf. 2004;27(10):735–44.

    Article  PubMed  Google Scholar 

  27. Hochberg AM, Hauben M, Pearson RK, et al. An evaluation of three signal-detection algorithms using a highly inclusive reference event database. Drug Saf. 2009;32(6):509–25.

    Article  PubMed  Google Scholar 

  28. Waller P. Dealing with uncertainty in drug safety: lessons for the future from sertindole. Pharmacoepidemiol Drug Saf. 2003;12(4):283–7. (discussion 289–90).

    Article  PubMed  Google Scholar 

  29. Bate A, Edwards IR. Data mining techniques in pharmacovigilance. In: Hartzema AG, Tilson HH, Chan KA, editors. Pharmacoepidemiology and therapeutic risk management. Cincinnati: Harvey Whitney; 2008. p. 239–72.

    Google Scholar 

  30. Bhattacharyya S, Schapira AH, Mikhailidis DP, et al. Drug-induced fibrotic valvular heart disease. Lancet. 2009;374(9689):577–85.

    Article  PubMed  CAS  Google Scholar 

  31. Levine JG, Tonning JM, Szarfman A. Reply: the evaluation of data mining methods for the simultaneous and systematic detection of safety signals in large databases: lessons to be learned. Br J Clin Pharmacol. 2006;61(1):105–13 (author reply 115–7).

    Article  PubMed  CAS  Google Scholar 

Download references

Acknowledgments

This research has been funded by the European Commission Seventh Framework Programme (FP7/2007-2013) under grant no. 215847—The EU-ADR Project. The funding agency had no role in the design and conduct of the study, the collection and management of data, the analysis or interpretation of the data, and preparation, review or approval of the manuscript. The authors thank the anonymous reviewers and Anders Ottosson of Astra Zeneca for their valuable comments and insights. Mariam Molokhia has previously received grants from AstraZeneca, Pfizer and the Serious Adverse Events Consortium (not for profit collaboration of industry and academia) for studies of ADRs. Miriam Sturkenboom is running a research group that occasionally performs studies for pharmaceutical companies according to unconditional grants. These companies include AstraZeneca, Pfizer, Lilly and Boehringer. She has also been a consultant to Pfizer, Novartis, Consumer Health, Servier, Celgene and Lundbeck on issues not related to the study.

Author information

Authors and Affiliations

  1. Department of Medical Informatics, Erasmus Medical Center, Postbus 2040, 3000, CA, Rotterdam, The Netherlands

    Preciosa M. Coloma, Martijn J. Schuemie, Carmen Ferrajolo, Vaishali Patadia, Johan van der Lei, Miriam Sturkenboom & Gianluca Trifirò

  2. LESIM, ISPED, University Bordeaux 2, Bordeaux, France

    Paul Avillach

  3. LERTIM, EA 3283, Faculté de Médecine, Université Aix Marseille 2, Marseille, France

    Paul Avillach

  4. Campania Regional Center for Pharmacovigilance and Pharmacoepidemiology, Department of Experimental Medicine, Pharmacology Section, Second University of Naples, Naples, Italy

    Francesco Salvo & Carmen Ferrajolo

  5. Inserm U657, Pharmacology Department, Bordeaux, France

    Francesco Salvo, Antoine Pariente & Annie Fourrier-Réglat

  6. CHU Bordeaux, Bordeaux, France

    Francesco Salvo, Antoine Pariente & Annie Fourrier-Réglat

  7. Department of Pharmacology, University of Bordeaux, Bordeaux, France

    Antoine Pariente & Annie Fourrier-Réglat

  8. Primary Care and Population Sciences, Kings College, London, UK

    Mariam Molokhia

  9. Astellas Pharmaceuticals, Deerfield, IL, USA

    Vaishali Patadia

  10. Department of Epidemiology, Erasmus University Medical Center, Rotterdam, The Netherlands

    Miriam Sturkenboom

  11. Department of Clinical and Experimental Medicine and Pharmacology, Section of Pharmacology, University of Messina, Messina, Italy

    Gianluca Trifirò

Authors
  1. Preciosa M. Coloma
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Paul Avillach
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Francesco Salvo
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Martijn J. Schuemie
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Carmen Ferrajolo
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Antoine Pariente
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Annie Fourrier-Réglat
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Mariam Molokhia
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Vaishali Patadia
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Johan van der Lei
    View author publications

    You can also search for this author in PubMed Google Scholar

  11. Miriam Sturkenboom
    View author publications

    You can also search for this author in PubMed Google Scholar

  12. Gianluca Trifirò
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Preciosa M. Coloma.

Additional information

On behalf of the EU-ADR Consortium.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (PDF 167 kb)

Rights and permissions

Reprints and permissions

About this article

Cite this article

Coloma, P.M., Avillach, P., Salvo, F. et al. A Reference Standard for Evaluation of Methods for Drug Safety Signal Detection Using Electronic Healthcare Record Databases. Drug Saf 36, 13–23 (2013). https://doi.org/10.1007/s40264-012-0002-x

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s40264-012-0002-x

Keywords

Search

Navigation