Abstract
Background
Most pharmacovigilance departments maintain a system to identify adverse drug reactions (ADRs) through analysis of spontaneous reports. The signal detection algorithms (SDAs) and the nature of the reporting databases vary between operators and it is unclear whether any algorithm can be expected to provide good performance in a wide range of environments.
Objective
The objective of this study was to compare the performance of commonly used algorithms across spontaneous reporting databases operated by pharmaceutical companies and national and international pharmacovigilance organisations.
Methods
220 products were chosen and a reference set of ADRs was compiled. Within four company, one national and two international databases, 15 SDAs based on five disproportionality methods were tested. Signals of disproportionate reporting (SDRs) were calculated at monthly intervals and classified by comparison with the reference set. These results were summarised as sensitivity and precision for each algorithm in each database.
Results
Different algorithms performed differently between databases but no method dominated all others. Performance was strongly dependent on the thresholds used to define a statistical signal. However, the different disproportionality statistics did not influence the achievable performance. The relative performance of two algorithms was similar in different databases. Over the lifetime of a product there is a reduction in precision for any method.
Conclusions
In designing signal detection systems, careful consideration should be given to the criteria that are used to define an SDR. The choice of disproportionality statistic does not appreciably affect the achievable range of signal detection performance and so this can primarily be based on ease of implementation, interpretation and minimisation of computing resources. The changes in sensitivity and precision obtainable by replacing one algorithm with another are predictable. However, the absolute performance of a method is specific to the database and is best assessed directly on that database. New methods may be required to gain appreciable improvements.
This is a preview of subscription content, access via your institution.
Notes
To avoid confusion it should be noted that an SDR does not necessarily fulfil the requirements of a signal as defined in pharmacovigilance.
MedDRA® (the Medical Dictionary for Regulatory Activities) terminology is the international medical terminology developed under the auspices of the International Conference on Harmonization of Technical Requirements for Registration of Pharmaceuticals for Human Use (ICH). MedDRA® trademark is owned by the International Federation of Pharmaceutical Manufacturers and Associations (IFPMA) on behalf of ICH.
References
Bate A, Lindquist M, Edwards IR, Olsson S, Orre R, Lansner A, et al. A Bayesian neural network method for adverse drug reaction signal generation. Eur J Clin Pharmacol. 1998;54(4):315–21.
Evans SJW, Waller PC, Davis S. Use of proportional reporting ratios (PRRs) for signal generation from spontaneous adverse drug reaction reports. Pharmacoepidemiol Drug Saf. 2001;10(6):483–6.
Lindquist M, Stahl M, Bate A, Edwards IR, Meyboom RHB. A retrospective evaluation of a data mining approach to aid finding new adverse drug reaction signals in the WHO international database. Drug Saf. 2000;23(6):533–42.
Szarfman A, Machado SG, O’Neill RT. Use of screening algorithms and computer systems to efficiently signal higher-than-expected combinations of drugs and events in the US FDA’s spontaneous reports database. Drug Saf. 2002;25(6):381–92.
Van Puijenbroek EP, Diemont WL, Van Grootheest K. Application of quantitative signal detection in the Dutch spontaneous reporting system for adverse drug reactions. Drug Saf. 2003;26(5):293–301.
Almenoff J, Tonning JM, Gould AL, Szarfman A, Hauben M, Ouellet-Hellstrom R, et al. Perspectives on the use of data mining in pharmacovigilance. Drug Saf. 2005;28(11):981–1007.
Roux E, Thiessard F, Fourrier A, Begaud B, Tubert-Bitter P. Evaluation of statistical association measures for the automatic signal generation in pharmacovigilance. IEEE Trans Inf Technol Biomed. 2005;9(4):518–27.
Czarnecki A, Voss S. Safety signals using proportional reporting ratios from company and regulatory authority databases. Drug Inf J 2008. 2008;42(3):205–210.
Hauben M, Horn S, Reich L. Potential use of data-mining algorithms for the detection of ‘surprise’ adverse drug reactions. Drug Saf. 2007;30(2):143–55.
Bate A, Lindquist M, Edwards IR. The application of knowledge discovery in databases to post-marketing drug safety: example of the WHO database. Fundam Clin Pharmacol. 2008;22(2):127–40.
Banks D, Woo EJ, Burwen DR, Perucci P, Braun MM, Ball R. Comparing data mining methods on the VAERS database. Pharmacoepidemiol Drug Saf. 2005;14(9):601–9.
Gipson G. A shrinkage-based comparative assessment of observed-to-expected disproportionality measures. Pharmacoepidemiol Drug Saf. 2012;21(6):589–96.
Hauben M, Aronson JK. Defining ‘signal’ and its subtypes in pharmacovigilance based on a systematic review of previous definitions. Drug Saf. 2009;32(2):99–110.
Finney DJ. Statistical logic in the monitoring of reactions to therapeutic drugs. Methods Inf Med. 1971;10(4):237–45.
Declerck G, Bousquet C, Jaulent MC. Automatic generation of MedDRA terms groupings using an ontology. Stud Health Technol Inform. 2012;180:73–7.
Hochberg AM, Hauben M, Pearson RK, OHara DJ, Reisinger SJ, Goldsmith DI, et al. An evaluation of three signal-detection algorithms using a highly inclusive reference event database. Drug Saf. 2009;32(6):509–25.
Slattery J, Alvarez Y, Hidalgo A. Choosing thresholds for statistical signal detection with the proportional reporting ratio. Drug Saf. 2013;36(8):687–92.
Gogolak VV. The effect of backgrounds in safety analysis: The impact of comparison cases on what you see. Pharmacoepidemiol Drug Saf. 2003;12(3):249–52.
Hammond IW, Gibbs TG, Seifert HA, Rich DS. Database size and power to detect safety signals in pharmacovigilance. Expert Opin Drug Saf. 2007;6(6):713–21.
Hammond IW, Rich DS, Gibbs TG. Effect of consumer reporting on signal detection: Using disproportionality analysis. Expert Opin Drug Saf. 2007;6(6):705–12.
Almenoff JS, LaCroix KK, Yuen NA, Fram D, DuMouchel W. Comparative performance of two quantitative safety signalling methods: implications for use in a pharmacovigilance department. Drug Saf. 2006;29(10):875–87.
Brown JS, Petronis K, Bate A, Zhang F, Dashevsky I, Kulldorff M, et al. Comparing two methods for detecting adverse event signals in observational data: empirical Bayes gamma poisson shrinker vs. tree-based scan statistic. Pharmacoepidemiol Drug Saf. 2011;20:S144.
Bunchuailua W, Zuckerman I, Kulsomboon V, Suwankesawong W, Singhasivanon P, Kaewkungwal J. A comparison of signal detection performance between reporting ODDS ratio and Bayesian confidence propagation neural network methods on adverse drug reaction spontaneous reporting database of the Thai FDA. Value Health. 2010;13(7):A508.
Chen Y, Guo JJ, Steinbuch M, Lin X, Buncher CR, Patel NC. Comparison of sensitivity and timing of early signal detection of four frequently used signal detection methods: An empirical study based on the US FDA adverse event reporting system database. Pharm Med. 2008;22(6):359–65.
Harpaz R, Dumouchel W, Lependu P, Bauer-Mehren A, Ryan P, Shah NH. Performance of pharmacovigilance signal-detection algorithms for the FDA adverse event reporting system. Clin Pharmacol Ther. 2013;93(6):539–46.
Van Puijenbroek EP, Bate A, Leufkens HGM, Lindquist M, Orre R, Egberts ACG. A comparison of measures of disproportionality for signal detection is spontaneous reporting systems for adverse drug reactions. Pharmacoepidemiol Drug Saf. 2002;11(1):3–10.
Caster O, Noren G, Niklas, Madigan D, Bate A. Large-scale regression-based pattern discovery: the example of screening the WHO global drug safety database. Stat Anal Data Min. 2010;3(4):197–208.
Tatonetti NP, Ye PP, Daneshjou R, Altman RB. Data-driven prediction of drug effects and interactions. Sci Transl Med. 2012;4(125):125ra31.
Kubota K, Koide D, Hirai T. Comparison of data mining methodologies using Japanese spontaneous reports. Pharmacoepidemiol Drug Saf. 2004;13(6):387–94.
Kurz X, Slattery J, Addis A, Durand J, Segec A, Skibicka I, et al. The EudraVigilance database of spontaneous adverse reactions as a tool for H1N1 vaccine safety monitoring. Pharmacoepidemiol Drug Saf. 2010;19:S330–1.
Alvarez Y, Hidalgo A, Maignen F, Slattery J. Validation of statistical signal detection procedures in EudraVigilance post-authorization data: a retrospective evaluation of the potential for earlier signalling. Drug Saf. 2010;33(6):475–87.
IMI PROTECT. ADR database. http://www.imi-protect.eu/methodsRep.shtml. Accessed 17 Mar 2014.
Clark JA, Klincewicz SL, Stang PE. Spontaneous adverse event signaling methods: classification and use with health care treatment products. Epidemiol Rev. 2001;23(2):191–210.
Norén GN, Hopstadius J, Bate A. Shrinkage observed-to-expected ratios for robust and transparent large-scale pattern discovery. Stat Methods Med Res. 2013;22(1):57–69.
DuMouchel W, Pregibon D. Empirical Bayes screening for multi-item associations. In: Proceedings of the seventh ACM SIGKDD international conference on knowledge discovery and data mining. San Francisco: ACM Press; 2001. pp. 67–76.
Acknowledgments
We would like to thank Jenny Wong, Jeffery Painter, Ramin Arani, Niklas Noren, Phil Tregunno, Michael Kayser and Magnus Lerch for their part in the study and constructive comments. The views expressed in this paper are those of the authors only and do not reflect the official policy or position of the IMI JU (Innovative Medicines Initiative Joint Undertaking), the European Union, European Federation of Pharmaceutical Industries and Associations (EFPIA) or the Medicines and Healthcare products Regulatory Agency.
Conflicts of interest
The PROTECT project has received support from the Innovative Medicine Initiative Joint Undertaking (http://www.imi.europa.eu) under Grant Agreement No. 115004, resources of which are composed of financial contribution from the European Union’s Seventh Framework Programme (FP7/2007–2013) and EFPIA companies’ in kind contribution. Gianmario Candore, Kristina Juhlin, Katrin Manlik, Bharat Thakrar, Naashika Quarcoo, Suzie Seabroke, Antoni Wisniewski and Jim Slattery have no financial interest in any commercial signal detection software. Katrin Manlik is an employee of Bayer Pharma AG, Bharat Thakrar is an employee of Roche and holds shares in both Roche and GlaxoSmithKline, Naashika Quarcoo is an employee of and holds shares in GlaxoSmithKline, and Antoni Wisniewski is an employee of and holds shares in AstraZeneca. Products from these companies were among those used to test the methodologies in this research.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Candore, G., Juhlin, K., Manlik, K. et al. Comparison of Statistical Signal Detection Methods Within and Across Spontaneous Reporting Databases. Drug Saf 38, 577–587 (2015). https://doi.org/10.1007/s40264-015-0289-5
Published:
Issue Date:
DOI: https://doi.org/10.1007/s40264-015-0289-5
Keywords
- Reporting Odds Ratio
- Proportional Reporting Ratio
- Uppsala Monitoring Centre
- Spontaneous Reporting Database
- Disproportionate Reporting