Abstract
Background
Completeness of adverse event (AE) reports is an important component of quality for good pharmacovigilance practices. We aimed to evaluate the impact of incorporating a measure of completeness of AE reports on quantitative signal detection.
Methods
An internal safety database from a global pharmaceutical company was used in the analysis. vigiGrade, an index score of completeness, was derived for each AE report. Data from various patient support programs (PSPs) were categorized based on average vigiGrade score per PSP. Performance of signal detection was compared between: (1) weighting and not weighting by vigiGrade score; and, (2) well documented and poorly documented PSPs using sensitivity, specificity, area under the receiver operating characteristics curve (AUC) and time-to-signal detection.
Results
The ability to detect signals did not differ significantly when weighting by vigiGrade score [sensitivity (50% vs. 45%, p = 1), specificity (82.8% vs. 82.8%, p = 1), AUC (0.66 vs. 0.63, p = 0.051) or time-to-signal detection (HR 0.81, p = 0.63)] compared to not weighting. Well documented PSPs were better at detecting signals than poorly documented PSPs (AUC 0.66 vs. 0.52; p = 0.041) but time-to-signal detection did not differ significantly (HR 1.54, p = 0.42).
Conclusion
Completeness of AE reports did not significantly impact the ability to detect signals when weighting by vigiGrade score or restricting the database based on the level of completeness. While the vigiGrade helps provide quality assessments of AE reports and prioritize cases for review, our findings indicate the tool might not be useful for quantitative signal detection when used by itself.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Hauben M, Aronson JK. Defining ‘Signal’ and its subtypes in pharmacovigilance based on a systematic review of previous definitions. Drug Saf. 2009;32(2):99–110.
Bate A, Evans SJW. Quantitative signal detection using spontaneous ADR reporting. Pharmacoepidemiol Drug Saf. 2009;18(6):427–36.
Stephenson WP, Hauben M. Data mining for signals in spontaneous reporting databases: proceed with caution. Pharmacoepidem Drug Saf. 2007;16(4):359–65.
Koutkias VG, Jaulent M-C. Computational approaches for pharmacovigilance signal detection: toward integrated and semantically-enriched frameworks. Drug Saf. 2015;38(3):219–32.
European Medicines Agency. Guideline on good pharmacovigilance practices (GVP) Module I – Pharmacovigilance systems and their quality systems [Internet]. https://www.ema.europa.eu/en/documents/scientific-guideline/guideline-good-pharmacovigilance-practices-module-i-pharmacovigilance-systems-their-quality-systems_en.pdf. Accessed 5 Nov 2015.
Bergvall T, Norén GN, Lindquist M. vigiGrade: a tool to identify well-documented individual case reports and highlight systematic data quality issues. Drug saf. 2014;37(1):65–77.
Strom BL, Kimmel SE, Hennessy S. Pharmacoepidemiology. Chapter 10 Postmarketing Spontaneous Pharmacovigilance Reporting System. Hoboken, UNITED KINGDOM: John Wiley & Sons, Incorporated; 2012.
Ribeiro A, Lima S, Zampieri M-E, Peinado M, Figueras A. Filling quality of the reports of adverse drug reactions received at the Pharmacovigilance Centre of São Paulo (Brazil): missing information hinders the analysis of suspected associations. Expert Opin Drug Saf. 2017;16(12):1329–34.
Lopez-Gonzalez E, Herdeiro MT, Figueiras A. Determinants of under-reporting of adverse drug reactions: a systematic review. Drug Saf. 2009;32(1):19–311.
Caster O, Juhlin K, Watson S, Norén GN. Improved statistical signal detection in pharmacovigilance by combining multiple strength-of-evidence aspects in vigiRank. Drug Saf. 2014;37(8):617–28.
ISMP Quarter Watch: A Critique of a Key Drug Safety Reporting System [Internet]. 2015. https://www.ismp.org/quarterwatch/drug-safety-reporting-system. Accessed 9 Mar 2018.
Portnoff JM, Lewis DJ. The enigma of pharmacovigilance of patient support programs: a survey of marketing authorization holders in Europe. Ther Innov Regul Sci. 2017;51(4):486–93.
Klein K, Scholl JHG, De Bruin ML, van Puijenbroek EP, Leufkens HGM, Stolk P. When more is less: an exploratory study of the precautionary reporting bias and its impact on safety signal detection. Clin Pharmacol Ther. 2018;103(2):296–303.
Harinstein L, Kalra D, Kortepeter CM, Muñoz MA, Wu E, Dal Pan GJ. Evaluation of postmarketing reports from industry-sponsored programs in drug safety surveillance. Drug Saf. 2019;42(5):649–55.
Poluzzi E, Raschi E, Piccinni C, De Ponti F. Data mining techniques in pharmacovigilance: analysis of the publicly accessible FDA adverse event reporting system (AERS). In: Data mining applications in engineering and medicine. InTech; 2012.
Harpaz R, DuMouchel W, LePendu P, Bauer-Mehren A, Ryan P, Shah NH. Performance of pharmacovigilance signal detection algorithms for the FDA adverse event reporting system. Clin Pharmacol Ther. 2013. https://doi.org/10.1038/clpt.2013.24.
Henderson M. Disproportionality measures [Internet]. GitHub. https://github.com/statmike/Disproportionality-Measures-with-SAS. Accessed 10 Oct 2019.
Cutts M. Oxford guide to plain english [Internet]. OUP Oxford; 2013. https://books.google.com/books?id=xd11AQAAQBAJ.
Palmer DD. A trainable rule-based algorithm for word segmentation. In: Proceedings of the 35th Annual Meeting of the Association for Computational Linguistics and Eighth Conference of the European Chapter of the Association for Computational Linguistics. Madrid, Spain: Association for Computational Linguistics; 1997. p. 321–8.
Pierce JR. An introduction to information theory: symbols, signals and noise [Internet]. Dover Publications; 2012. (Dover Books on Mathematics). https://books.google.com/books?id=eKvhiI2ogwEC.
Barnard G. Statistical calculation of word entropies for four Western languages. IRE Trans Inf Theory. 1955;1(1):49–53.
Lee I, Lee TA, Crawford SY, Kilpatrick RD, Calip GS, Jokinen JD. Impact of adverse event reports from marketing authorization holder-sponsored patient support programs on the performance of signal detection in pharmacovigilance. Expert Opin Drug Saf. 2020. https://doi.org/10.1080/14740338.2020.1792883.
UMC | vigiMethods [Internet]. https://www.who-umc.org/vigibase/vigilyze/vigimethods/. Accessed 3 Nov 2019.
Funding
No source of funding was used to assist in the preparation of this study.
Author information
Authors and Affiliations
Contributions
All authors contributed to the study conception and design. Material preparation and data analysis were performed by IL. The first draft of the manuscript was written by IL and all authors commented on previous versions of the manuscript. All authors read and approved the final manuscript.
Corresponding author
Ethics declarations
Conflict of interest
Inyoung Lee was supported by the UIC-AbbVie Fellowship in Pharmacovigilance and Patient Safety. Jeremy D. Jokinen was employed by AbbVie, Inc. and owned stocks of the company while the submitted work was conducted; and currently is an employee and shareholder of Bristol-Myers Squibb Company. Gregory S. Calip received grants from AbbVie, Inc. and Pfizer, Inc. outside the submitted work; and at the time of publication, reports current employment with Flatiron Health, Inc., which is an independent subsidiary of the Roche group. Ryan D. Kilpatrick is an employee of AbbVie, Inc. and may own stocks of the company. All other authors have no conflicts of interest to disclose.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Lee, I., Jokinen, J.D., Crawford, S.Y. et al. Exploring Completeness of Adverse Event Reports as a Tool for Signal Detection in Pharmacovigilance. Ther Innov Regul Sci 55, 142–151 (2021). https://doi.org/10.1007/s43441-020-00199-z
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s43441-020-00199-z