Comparison of Statistical Signal Detection Methods Within and Across Spontaneous Reporting Databases
- 1.4k Downloads
Most pharmacovigilance departments maintain a system to identify adverse drug reactions (ADRs) through analysis of spontaneous reports. The signal detection algorithms (SDAs) and the nature of the reporting databases vary between operators and it is unclear whether any algorithm can be expected to provide good performance in a wide range of environments.
The objective of this study was to compare the performance of commonly used algorithms across spontaneous reporting databases operated by pharmaceutical companies and national and international pharmacovigilance organisations.
220 products were chosen and a reference set of ADRs was compiled. Within four company, one national and two international databases, 15 SDAs based on five disproportionality methods were tested. Signals of disproportionate reporting (SDRs) were calculated at monthly intervals and classified by comparison with the reference set. These results were summarised as sensitivity and precision for each algorithm in each database.
Different algorithms performed differently between databases but no method dominated all others. Performance was strongly dependent on the thresholds used to define a statistical signal. However, the different disproportionality statistics did not influence the achievable performance. The relative performance of two algorithms was similar in different databases. Over the lifetime of a product there is a reduction in precision for any method.
In designing signal detection systems, careful consideration should be given to the criteria that are used to define an SDR. The choice of disproportionality statistic does not appreciably affect the achievable range of signal detection performance and so this can primarily be based on ease of implementation, interpretation and minimisation of computing resources. The changes in sensitivity and precision obtainable by replacing one algorithm with another are predictable. However, the absolute performance of a method is specific to the database and is best assessed directly on that database. New methods may be required to gain appreciable improvements.
KeywordsReporting Odds Ratio Proportional Reporting Ratio Uppsala Monitoring Centre Spontaneous Reporting Database Disproportionate Reporting
We would like to thank Jenny Wong, Jeffery Painter, Ramin Arani, Niklas Noren, Phil Tregunno, Michael Kayser and Magnus Lerch for their part in the study and constructive comments. The views expressed in this paper are those of the authors only and do not reflect the official policy or position of the IMI JU (Innovative Medicines Initiative Joint Undertaking), the European Union, European Federation of Pharmaceutical Industries and Associations (EFPIA) or the Medicines and Healthcare products Regulatory Agency.
Conflicts of interest
The PROTECT project has received support from the Innovative Medicine Initiative Joint Undertaking (http://www.imi.europa.eu) under Grant Agreement No. 115004, resources of which are composed of financial contribution from the European Union’s Seventh Framework Programme (FP7/2007–2013) and EFPIA companies’ in kind contribution. Gianmario Candore, Kristina Juhlin, Katrin Manlik, Bharat Thakrar, Naashika Quarcoo, Suzie Seabroke, Antoni Wisniewski and Jim Slattery have no financial interest in any commercial signal detection software. Katrin Manlik is an employee of Bayer Pharma AG, Bharat Thakrar is an employee of Roche and holds shares in both Roche and GlaxoSmithKline, Naashika Quarcoo is an employee of and holds shares in GlaxoSmithKline, and Antoni Wisniewski is an employee of and holds shares in AstraZeneca. Products from these companies were among those used to test the methodologies in this research.
- 8.Czarnecki A, Voss S. Safety signals using proportional reporting ratios from company and regulatory authority databases. Drug Inf J 2008. 2008;42(3):205–210.Google Scholar
- 22.Brown JS, Petronis K, Bate A, Zhang F, Dashevsky I, Kulldorff M, et al. Comparing two methods for detecting adverse event signals in observational data: empirical Bayes gamma poisson shrinker vs. tree-based scan statistic. Pharmacoepidemiol Drug Saf. 2011;20:S144.Google Scholar
- 23.Bunchuailua W, Zuckerman I, Kulsomboon V, Suwankesawong W, Singhasivanon P, Kaewkungwal J. A comparison of signal detection performance between reporting ODDS ratio and Bayesian confidence propagation neural network methods on adverse drug reaction spontaneous reporting database of the Thai FDA. Value Health. 2010;13(7):A508.CrossRefGoogle Scholar
- 24.Chen Y, Guo JJ, Steinbuch M, Lin X, Buncher CR, Patel NC. Comparison of sensitivity and timing of early signal detection of four frequently used signal detection methods: An empirical study based on the US FDA adverse event reporting system database. Pharm Med. 2008;22(6):359–65.CrossRefGoogle Scholar
- 27.Caster O, Noren G, Niklas, Madigan D, Bate A. Large-scale regression-based pattern discovery: the example of screening the WHO global drug safety database. Stat Anal Data Min. 2010;3(4):197–208.Google Scholar
- 30.Kurz X, Slattery J, Addis A, Durand J, Segec A, Skibicka I, et al. The EudraVigilance database of spontaneous adverse reactions as a tool for H1N1 vaccine safety monitoring. Pharmacoepidemiol Drug Saf. 2010;19:S330–1.Google Scholar
- 32.IMI PROTECT. ADR database. http://www.imi-protect.eu/methodsRep.shtml. Accessed 17 Mar 2014.