Advertisement

Drug Safety

, Volume 38, Issue 6, pp 577–587 | Cite as

Comparison of Statistical Signal Detection Methods Within and Across Spontaneous Reporting Databases

  • Gianmario Candore
  • Kristina Juhlin
  • Katrin Manlik
  • Bharat Thakrar
  • Naashika Quarcoo
  • Suzie Seabroke
  • Antoni Wisniewski
  • Jim SlatteryEmail author
Original Research Article

Abstract

Background

Most pharmacovigilance departments maintain a system to identify adverse drug reactions (ADRs) through analysis of spontaneous reports. The signal detection algorithms (SDAs) and the nature of the reporting databases vary between operators and it is unclear whether any algorithm can be expected to provide good performance in a wide range of environments.

Objective

The objective of this study was to compare the performance of commonly used algorithms across spontaneous reporting databases operated by pharmaceutical companies and national and international pharmacovigilance organisations.

Methods

220 products were chosen and a reference set of ADRs was compiled. Within four company, one national and two international databases, 15 SDAs based on five disproportionality methods were tested. Signals of disproportionate reporting (SDRs) were calculated at monthly intervals and classified by comparison with the reference set. These results were summarised as sensitivity and precision for each algorithm in each database.

Results

Different algorithms performed differently between databases but no method dominated all others. Performance was strongly dependent on the thresholds used to define a statistical signal. However, the different disproportionality statistics did not influence the achievable performance. The relative performance of two algorithms was similar in different databases. Over the lifetime of a product there is a reduction in precision for any method.

Conclusions

In designing signal detection systems, careful consideration should be given to the criteria that are used to define an SDR. The choice of disproportionality statistic does not appreciably affect the achievable range of signal detection performance and so this can primarily be based on ease of implementation, interpretation and minimisation of computing resources. The changes in sensitivity and precision obtainable by replacing one algorithm with another are predictable. However, the absolute performance of a method is specific to the database and is best assessed directly on that database. New methods may be required to gain appreciable improvements.

Keywords

Reporting Odds Ratio Proportional Reporting Ratio Uppsala Monitoring Centre Spontaneous Reporting Database Disproportionate Reporting 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Notes

Acknowledgments

We would like to thank Jenny Wong, Jeffery Painter, Ramin Arani, Niklas Noren, Phil Tregunno, Michael Kayser and Magnus Lerch for their part in the study and constructive comments. The views expressed in this paper are those of the authors only and do not reflect the official policy or position of the IMI JU (Innovative Medicines Initiative Joint Undertaking), the European Union, European Federation of Pharmaceutical Industries and Associations (EFPIA) or the Medicines and Healthcare products Regulatory Agency.

Conflicts of interest

The PROTECT project has received support from the Innovative Medicine Initiative Joint Undertaking (http://www.imi.europa.eu) under Grant Agreement No. 115004, resources of which are composed of financial contribution from the European Union’s Seventh Framework Programme (FP7/2007–2013) and EFPIA companies’ in kind contribution. Gianmario Candore, Kristina Juhlin, Katrin Manlik, Bharat Thakrar, Naashika Quarcoo, Suzie Seabroke, Antoni Wisniewski and Jim Slattery have no financial interest in any commercial signal detection software. Katrin Manlik is an employee of Bayer Pharma AG, Bharat Thakrar is an employee of Roche and holds shares in both Roche and GlaxoSmithKline, Naashika Quarcoo is an employee of and holds shares in GlaxoSmithKline, and Antoni Wisniewski is an employee of and holds shares in AstraZeneca. Products from these companies were among those used to test the methodologies in this research.

References

  1. 1.
    Bate A, Lindquist M, Edwards IR, Olsson S, Orre R, Lansner A, et al. A Bayesian neural network method for adverse drug reaction signal generation. Eur J Clin Pharmacol. 1998;54(4):315–21.CrossRefPubMedGoogle Scholar
  2. 2.
    Evans SJW, Waller PC, Davis S. Use of proportional reporting ratios (PRRs) for signal generation from spontaneous adverse drug reaction reports. Pharmacoepidemiol Drug Saf. 2001;10(6):483–6.CrossRefPubMedGoogle Scholar
  3. 3.
    Lindquist M, Stahl M, Bate A, Edwards IR, Meyboom RHB. A retrospective evaluation of a data mining approach to aid finding new adverse drug reaction signals in the WHO international database. Drug Saf. 2000;23(6):533–42.CrossRefPubMedGoogle Scholar
  4. 4.
    Szarfman A, Machado SG, O’Neill RT. Use of screening algorithms and computer systems to efficiently signal higher-than-expected combinations of drugs and events in the US FDA’s spontaneous reports database. Drug Saf. 2002;25(6):381–92.CrossRefPubMedGoogle Scholar
  5. 5.
    Van Puijenbroek EP, Diemont WL, Van Grootheest K. Application of quantitative signal detection in the Dutch spontaneous reporting system for adverse drug reactions. Drug Saf. 2003;26(5):293–301.CrossRefPubMedGoogle Scholar
  6. 6.
    Almenoff J, Tonning JM, Gould AL, Szarfman A, Hauben M, Ouellet-Hellstrom R, et al. Perspectives on the use of data mining in pharmacovigilance. Drug Saf. 2005;28(11):981–1007.CrossRefPubMedGoogle Scholar
  7. 7.
    Roux E, Thiessard F, Fourrier A, Begaud B, Tubert-Bitter P. Evaluation of statistical association measures for the automatic signal generation in pharmacovigilance. IEEE Trans Inf Technol Biomed. 2005;9(4):518–27.CrossRefPubMedGoogle Scholar
  8. 8.
    Czarnecki A, Voss S. Safety signals using proportional reporting ratios from company and regulatory authority databases. Drug Inf J 2008. 2008;42(3):205–210.Google Scholar
  9. 9.
    Hauben M, Horn S, Reich L. Potential use of data-mining algorithms for the detection of ‘surprise’ adverse drug reactions. Drug Saf. 2007;30(2):143–55.CrossRefPubMedGoogle Scholar
  10. 10.
    Bate A, Lindquist M, Edwards IR. The application of knowledge discovery in databases to post-marketing drug safety: example of the WHO database. Fundam Clin Pharmacol. 2008;22(2):127–40.CrossRefPubMedGoogle Scholar
  11. 11.
    Banks D, Woo EJ, Burwen DR, Perucci P, Braun MM, Ball R. Comparing data mining methods on the VAERS database. Pharmacoepidemiol Drug Saf. 2005;14(9):601–9.CrossRefPubMedGoogle Scholar
  12. 12.
    Gipson G. A shrinkage-based comparative assessment of observed-to-expected disproportionality measures. Pharmacoepidemiol Drug Saf. 2012;21(6):589–96.CrossRefPubMedGoogle Scholar
  13. 13.
    Hauben M, Aronson JK. Defining ‘signal’ and its subtypes in pharmacovigilance based on a systematic review of previous definitions. Drug Saf. 2009;32(2):99–110.CrossRefPubMedGoogle Scholar
  14. 14.
    Finney DJ. Statistical logic in the monitoring of reactions to therapeutic drugs. Methods Inf Med. 1971;10(4):237–45.PubMedGoogle Scholar
  15. 15.
    Declerck G, Bousquet C, Jaulent MC. Automatic generation of MedDRA terms groupings using an ontology. Stud Health Technol Inform. 2012;180:73–7.PubMedGoogle Scholar
  16. 16.
    Hochberg AM, Hauben M, Pearson RK, OHara DJ, Reisinger SJ, Goldsmith DI, et al. An evaluation of three signal-detection algorithms using a highly inclusive reference event database. Drug Saf. 2009;32(6):509–25.CrossRefPubMedGoogle Scholar
  17. 17.
    Slattery J, Alvarez Y, Hidalgo A. Choosing thresholds for statistical signal detection with the proportional reporting ratio. Drug Saf. 2013;36(8):687–92.CrossRefPubMedGoogle Scholar
  18. 18.
    Gogolak VV. The effect of backgrounds in safety analysis: The impact of comparison cases on what you see. Pharmacoepidemiol Drug Saf. 2003;12(3):249–52.CrossRefPubMedGoogle Scholar
  19. 19.
    Hammond IW, Gibbs TG, Seifert HA, Rich DS. Database size and power to detect safety signals in pharmacovigilance. Expert Opin Drug Saf. 2007;6(6):713–21.CrossRefPubMedGoogle Scholar
  20. 20.
    Hammond IW, Rich DS, Gibbs TG. Effect of consumer reporting on signal detection: Using disproportionality analysis. Expert Opin Drug Saf. 2007;6(6):705–12.CrossRefPubMedGoogle Scholar
  21. 21.
    Almenoff JS, LaCroix KK, Yuen NA, Fram D, DuMouchel W. Comparative performance of two quantitative safety signalling methods: implications for use in a pharmacovigilance department. Drug Saf. 2006;29(10):875–87.CrossRefPubMedGoogle Scholar
  22. 22.
    Brown JS, Petronis K, Bate A, Zhang F, Dashevsky I, Kulldorff M, et al. Comparing two methods for detecting adverse event signals in observational data: empirical Bayes gamma poisson shrinker vs. tree-based scan statistic. Pharmacoepidemiol Drug Saf. 2011;20:S144.Google Scholar
  23. 23.
    Bunchuailua W, Zuckerman I, Kulsomboon V, Suwankesawong W, Singhasivanon P, Kaewkungwal J. A comparison of signal detection performance between reporting ODDS ratio and Bayesian confidence propagation neural network methods on adverse drug reaction spontaneous reporting database of the Thai FDA. Value Health. 2010;13(7):A508.CrossRefGoogle Scholar
  24. 24.
    Chen Y, Guo JJ, Steinbuch M, Lin X, Buncher CR, Patel NC. Comparison of sensitivity and timing of early signal detection of four frequently used signal detection methods: An empirical study based on the US FDA adverse event reporting system database. Pharm Med. 2008;22(6):359–65.CrossRefGoogle Scholar
  25. 25.
    Harpaz R, Dumouchel W, Lependu P, Bauer-Mehren A, Ryan P, Shah NH. Performance of pharmacovigilance signal-detection algorithms for the FDA adverse event reporting system. Clin Pharmacol Ther. 2013;93(6):539–46.CrossRefPubMedGoogle Scholar
  26. 26.
    Van Puijenbroek EP, Bate A, Leufkens HGM, Lindquist M, Orre R, Egberts ACG. A comparison of measures of disproportionality for signal detection is spontaneous reporting systems for adverse drug reactions. Pharmacoepidemiol Drug Saf. 2002;11(1):3–10.CrossRefPubMedGoogle Scholar
  27. 27.
    Caster O, Noren G, Niklas, Madigan D, Bate A. Large-scale regression-based pattern discovery: the example of screening the WHO global drug safety database. Stat Anal Data Min. 2010;3(4):197–208.Google Scholar
  28. 28.
    Tatonetti NP, Ye PP, Daneshjou R, Altman RB. Data-driven prediction of drug effects and interactions. Sci Transl Med. 2012;4(125):125ra31.CrossRefPubMedCentralPubMedGoogle Scholar
  29. 29.
    Kubota K, Koide D, Hirai T. Comparison of data mining methodologies using Japanese spontaneous reports. Pharmacoepidemiol Drug Saf. 2004;13(6):387–94.CrossRefPubMedGoogle Scholar
  30. 30.
    Kurz X, Slattery J, Addis A, Durand J, Segec A, Skibicka I, et al. The EudraVigilance database of spontaneous adverse reactions as a tool for H1N1 vaccine safety monitoring. Pharmacoepidemiol Drug Saf. 2010;19:S330–1.Google Scholar
  31. 31.
    Alvarez Y, Hidalgo A, Maignen F, Slattery J. Validation of statistical signal detection procedures in EudraVigilance post-authorization data: a retrospective evaluation of the potential for earlier signalling. Drug Saf. 2010;33(6):475–87.CrossRefPubMedGoogle Scholar
  32. 32.
    IMI PROTECT. ADR database. http://www.imi-protect.eu/methodsRep.shtml. Accessed 17 Mar 2014.
  33. 33.
    Clark JA, Klincewicz SL, Stang PE. Spontaneous adverse event signaling methods: classification and use with health care treatment products. Epidemiol Rev. 2001;23(2):191–210.CrossRefPubMedGoogle Scholar
  34. 34.
    Norén GN, Hopstadius J, Bate A. Shrinkage observed-to-expected ratios for robust and transparent large-scale pattern discovery. Stat Methods Med Res. 2013;22(1):57–69.CrossRefPubMedGoogle Scholar
  35. 35.
    DuMouchel W, Pregibon D. Empirical Bayes screening for multi-item associations. In: Proceedings of the seventh ACM SIGKDD international conference on knowledge discovery and data mining. San Francisco: ACM Press; 2001. pp. 67–76.CrossRefGoogle Scholar

Copyright information

© European Union 2015

Authors and Affiliations

  • Gianmario Candore
    • 1
  • Kristina Juhlin
    • 2
  • Katrin Manlik
    • 3
  • Bharat Thakrar
    • 4
  • Naashika Quarcoo
    • 5
  • Suzie Seabroke
    • 6
  • Antoni Wisniewski
    • 7
  • Jim Slattery
    • 1
    Email author
  1. 1.European Medicines AgencyLondonUK
  2. 2.Uppsala Monitoring CentreUppsalaSweden
  3. 3.Bayer Pharma AGBerlinGermany
  4. 4.RocheBaselswitzerland
  5. 5.GlaxoSmithKlineLondonUK
  6. 6.UK Medicines and Healthcare Products Regulatory AgencyLondonUK
  7. 7.AstraZenecaAlderley ParkUK

Personalised recommendations