Choosing Thresholds for Statistical Signal Detection with the Proportional Reporting Ratio
Abstract
Background
Identification of potential drug safety problems using statistical screening algorithms in routinely collected databases of adverse drug reactions (ADRs) requires decision rules based on thresholds of the chosen parameters. Choosing higher or lower thresholds changes both the sensitivity of the screening and the number of false alarms produced, and thus has an impact on the effectiveness of the detection process.
Objective
The aim of this study was to evaluate the impact on the effectiveness of signal detection activities of choosing different warning thresholds for the proportional reporting ratio (PRR) and for the count of reports of any drug-event combination.
Methods
Signal detection methods were tested within the EudraVigilance database of suspected ADRs. Using an established set of known ADRs, the number that could be detected and the changes in time gained for earlier investigation of the signal were calculated over a range of signal detection thresholds. These figures were set against the number of false positive signals produced by the statistical signal detection algorithms.
Results
Higher thresholds for the lower confidence bound of the PRR produced fewer false positives but this benefit was offset by important losses of sensitivity in the detection of ADRs. By contrast, increases in the threshold for the count of a specific drug-event combination produced fewer false positives with little loss of either sensitivity or time gained for investigation of adverse events. A threshold of five compared with the current European Medicines Agency threshold of three gave a reduction of 25 % in false positive signals in return for a loss of 12 % in true signals detected early.
Conclusion
Changes in the standard threshold for the count of drug-event combinations can result in a substantial improvement in efficiency of the signal detection process. Initially this change might be applied only to products with a well-established safety profile.
Keywords
Signal Detection European Medicine Agency Statistical Signal Detection Proportional Reporting Ratio Marketing Authorization HolderNotes
Acknowledgments
The authors would like to thank Prof. H.-G. Eichler, Prof. M. Posch and F. Maignen for their constructive comments. The views expressed in this article are the personal views of the author(s) and may not be understood or quoted as being made on behalf of or reflecting the position of the EMA or one of its committees or working parties.
Funding
No sources of funding were used to conduct this study or prepare this manuscript.
Conflict of interest
Jim Slattery, Yolanda Alvarez and Ana Hidalgo have no conflicts of interest to declare that are directly relevant to the content of this study.
References
- 1.Finney DJ. Statistical logic in the monitoring of adverse reactions to therapeutic drugs. Methods Inf Med. 1971;10:237–42.PubMedGoogle Scholar
- 2.Alvarez Y, Hidalgo A, Maignen F, Slattery J. Validation of statistical signal detection procedures in EudraVigilance post-authorisation data: a retrospective evaluation of the potential for earlier signalling. Drug Saf. 2010;33(6):475–87.CrossRefPubMedGoogle Scholar
- 3.Council Regulation (EEC) No 2309/93 of 22 July 1993 laying down Community procedures for the authorization and supervision of medicinal products for human and veterinary use and establishing a European Agency for the Evaluation of Medicinal Products. http://ec.europa.eu/health/files/eudralex/vol-1/reg_1993_2309/reg_1993_2309_en.pdf. Accessed 16 May 2013.
- 4.MedDRA and MSSO. http://www.meddramsso.com. Accessed 18 Dec 2012.
- 5.European Medicines Agency. EudraVigilance. http://eudravigilance.emea.europa.eu/highres.htm. Accessed 18 Dec 2012.
- 6.Evans SJW, Waller P, Davis S. Use of proportional reporting ratios (PRR) for signal generation from spontaneous adverse drug reaction reports. Pharmacoepidemiol Drug Saf. 2001;10:483–6.CrossRefPubMedGoogle Scholar
- 7.EudraVigilance Expert Working Group. Guideline on the use of statistical signal detection methods in the EudraVigilance data analysis system (EMEA/106464/2006 rev.1). http://www.emea.europa.eu/pdfs/human/phvwp/10646406enfin.pdf. Accessed 18 Dec 2012.
- 8.Pharmacoepidemiological Research on Outcomes of Therapeutics by a European Consortium. http://www.imi-protect.eu/. Accessed 18 Dec 2012.
- 9.Matsushita Y, Kuroda Y, Niwa S, Sonehara S, Hamada C, Yoshimura I. Criteria revision and performance comparison of three methods of signal detection applied to the spontaneous reporting database of a pharmaceutical manufacturer. Drug Saf. 2007;30(8):715–26.CrossRefPubMedGoogle Scholar
- 10.Almenoff J, LaCroix K, Yuen N, et al. Comparative performance of two quantitative safety signalling methods. Drug Saf. 2006;29(10):875–87.CrossRefPubMedGoogle Scholar
- 11.Ahmed I, Thiessard F, Miremont-Salame G, et al. Early detection of pharmacovigilance signals with automated methods based on false discovery rates. Drug Saf. 2012;35(6):495–506.CrossRefPubMedGoogle Scholar
- 12.Berlin C, Blanch C, Lewis D, et al. Are all quantitative postmarketing signal detection methods equal? Performance characteristics of logistic regression and multi-item Gamma Poisson Shrinker. Pharmacoepidemiol Drug Saf. 2012;21:622–30.CrossRefPubMedGoogle Scholar