Abstract
Background
Identification of potential drug safety problems using statistical screening algorithms in routinely collected databases of adverse drug reactions (ADRs) requires decision rules based on thresholds of the chosen parameters. Choosing higher or lower thresholds changes both the sensitivity of the screening and the number of false alarms produced, and thus has an impact on the effectiveness of the detection process.
Objective
The aim of this study was to evaluate the impact on the effectiveness of signal detection activities of choosing different warning thresholds for the proportional reporting ratio (PRR) and for the count of reports of any drug-event combination.
Methods
Signal detection methods were tested within the EudraVigilance database of suspected ADRs. Using an established set of known ADRs, the number that could be detected and the changes in time gained for earlier investigation of the signal were calculated over a range of signal detection thresholds. These figures were set against the number of false positive signals produced by the statistical signal detection algorithms.
Results
Higher thresholds for the lower confidence bound of the PRR produced fewer false positives but this benefit was offset by important losses of sensitivity in the detection of ADRs. By contrast, increases in the threshold for the count of a specific drug-event combination produced fewer false positives with little loss of either sensitivity or time gained for investigation of adverse events. A threshold of five compared with the current European Medicines Agency threshold of three gave a reduction of 25 % in false positive signals in return for a loss of 12 % in true signals detected early.
Conclusion
Changes in the standard threshold for the count of drug-event combinations can result in a substantial improvement in efficiency of the signal detection process. Initially this change might be applied only to products with a well-established safety profile.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Notes
MedDRA® terminology is the international medical terminology developed under the auspices of the International Conference on Harmonization (ICH) of Technical Requirements for Registration of Pharmaceuticals for Human Use.
References
Finney DJ. Statistical logic in the monitoring of adverse reactions to therapeutic drugs. Methods Inf Med. 1971;10:237–42.
Alvarez Y, Hidalgo A, Maignen F, Slattery J. Validation of statistical signal detection procedures in EudraVigilance post-authorisation data: a retrospective evaluation of the potential for earlier signalling. Drug Saf. 2010;33(6):475–87.
Council Regulation (EEC) No 2309/93 of 22 July 1993 laying down Community procedures for the authorization and supervision of medicinal products for human and veterinary use and establishing a European Agency for the Evaluation of Medicinal Products. http://ec.europa.eu/health/files/eudralex/vol-1/reg_1993_2309/reg_1993_2309_en.pdf. Accessed 16 May 2013.
MedDRA and MSSO. http://www.meddramsso.com. Accessed 18 Dec 2012.
European Medicines Agency. EudraVigilance. http://eudravigilance.emea.europa.eu/highres.htm. Accessed 18 Dec 2012.
Evans SJW, Waller P, Davis S. Use of proportional reporting ratios (PRR) for signal generation from spontaneous adverse drug reaction reports. Pharmacoepidemiol Drug Saf. 2001;10:483–6.
EudraVigilance Expert Working Group. Guideline on the use of statistical signal detection methods in the EudraVigilance data analysis system (EMEA/106464/2006 rev.1). http://www.emea.europa.eu/pdfs/human/phvwp/10646406enfin.pdf. Accessed 18 Dec 2012.
Pharmacoepidemiological Research on Outcomes of Therapeutics by a European Consortium. http://www.imi-protect.eu/. Accessed 18 Dec 2012.
Matsushita Y, Kuroda Y, Niwa S, Sonehara S, Hamada C, Yoshimura I. Criteria revision and performance comparison of three methods of signal detection applied to the spontaneous reporting database of a pharmaceutical manufacturer. Drug Saf. 2007;30(8):715–26.
Almenoff J, LaCroix K, Yuen N, et al. Comparative performance of two quantitative safety signalling methods. Drug Saf. 2006;29(10):875–87.
Ahmed I, Thiessard F, Miremont-Salame G, et al. Early detection of pharmacovigilance signals with automated methods based on false discovery rates. Drug Saf. 2012;35(6):495–506.
Berlin C, Blanch C, Lewis D, et al. Are all quantitative postmarketing signal detection methods equal? Performance characteristics of logistic regression and multi-item Gamma Poisson Shrinker. Pharmacoepidemiol Drug Saf. 2012;21:622–30.
Acknowledgments
The authors would like to thank Prof. H.-G. Eichler, Prof. M. Posch and F. Maignen for their constructive comments. The views expressed in this article are the personal views of the author(s) and may not be understood or quoted as being made on behalf of or reflecting the position of the EMA or one of its committees or working parties.
Funding
No sources of funding were used to conduct this study or prepare this manuscript.
Conflict of interest
Jim Slattery, Yolanda Alvarez and Ana Hidalgo have no conflicts of interest to declare that are directly relevant to the content of this study.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Slattery, J., Alvarez, Y. & Hidalgo, A. Choosing Thresholds for Statistical Signal Detection with the Proportional Reporting Ratio. Drug Saf 36, 687–692 (2013). https://doi.org/10.1007/s40264-013-0075-1
Published:
Issue Date:
DOI: https://doi.org/10.1007/s40264-013-0075-1