Skip to main content
SpringerLink
Log in

An Exploratory Study of the Impact of COVID-19 Vaccine Spontaneous Reporting on Masking Signal Detection in EudraVigilance

  • Original Research Article
  • Published:
Drug Safety Aims and scope Submit manuscript

Abstract

Introduction

During the signal detection process, statistical methods are used to identify drug–event combinations (DECs) which are disproportionately reported when compared with other drugs and events in the entire database. We hypothesise that the high volume of COVID-19 vaccine adverse drug reaction (ADR) reports transmitted to EudraVigilance may have affected the performance of disproportionality statistics used in routine signal detection, potentially resulting in signals either being masked, or false associations being flagged as potential signals.

Objective

Our aim was to study the impact of COVID-19 vaccine spontaneous reporting on statistical signal detection in EudraVigilance.

Methods

We recalculated the reporting odds ratio (ROR) for signals that were previously discussed at the level of the Pharmacovigilance Risk Assessment Committee, or signals that were retrieved from EudraVigilance, by omitting COVID-19 vaccine reports from the standard ROR calculation and then comparing the lower confidence interval (LCI) of the recalculated ROR to the LCI of the actual ROR in EudraVigilance.

Results

In total, 52 signals for 38 active substances were reviewed. For 35 signals, the LCI of the recalculated ROR value was lower than the LCI of the actual ROR (suggesting that COVID-19 vaccine ADR reporting had a positive effect on the strength of the signal) while for 15 signals the LCI of the recalculated ROR value was higher than the LCI of the actual ROR (suggesting that COVID-19 vaccine ADR reporting had an attenuating effect on the strength of the signal). For two signals, no change in the ROR was observed. In our analysis, six significant results were found. Five DECs were found to be masked: bleomycin and immune thrombocytopenia (actual ROR LCI = 0.94, recalculated ROR LCI = 1.02), vortioxetine and heavy menstrual bleeding (actual ROR LCI = 0.3, recalculated ROR LCI = 1.06), caplacizumab and heavy menstrual bleeding (actual ROR LCI = 0.98, recalculated ROR LCI = 3.47), ziprasidone and amenorrhoea (actual ROR LCI = 0.84, recalculated ROR LCI = 1.67), and azacitidine and pericarditis (actual ROR LCI = 0.81, recalculated ROR LCI = 2.01). For the DEC of adalimumab and immune reconstitution inflammatory syndrome, the LCI of the actual ROR value was 1.14 and removing COVID-19 vaccine reporting resulted in an LCI of the recalculated ROR value of 0.94 (below threshold).

Conclusions

We demonstrated five cases of masking and one case of false-positive association due to the influence of COVID-19 vaccine spontaneous reporting on the ROR. This suggests that the high number of adverse drug reaction reports for COVID-19 vaccines in EudraVigilance has the potential to affect routine statistical signal detection activities. The impact of COVID-19 vaccine ADR reports on current signal detection practices requires further evaluation and solutions to tackle masking issues in EudraVigilance may need to be developed.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

References

  1. Sienkiewicz K, Burzyńska M, Rydlewska-Liszkowska I, Sienkiewicz J, Gaszyńska E. The importance of direct patient reporting of adverse drug reactions in the safety monitoring process. Int J Environ Res Public Health. 2021;19:413. https://doi.org/10.3390/ijerph19010413.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  2. Zuccarelli M, Micallef B, Butler D, Serracino-Inglott A, Borg J-J. Improving the data quality of spontaneous ADR reports: a practical example from Malta. Expert Opin Drug Saf. 2022;21:253–68. https://doi.org/10.1080/14740338.2022.1993820.

    Article  PubMed  Google Scholar 

  3. European Commission. Commission implementing regulation (EU) No 520/2012 of 19 June 2012 on the performance of pharmacovigilance activities provided for in regulation (EC) No 726/2004 of the European parliament and of the council and directive 2001/83/ EC of the European parliament and of the council. OJ. 2012; L159:5–25.

  4. European Medicines Agency. Comirnaty and Spikevax: possible link to very rare cases of myocarditis and pericarditis [Internet]. Amsterdam (NL): European Medicines Agency; 2021 July 09 [cited 2022 Dec 28]. https://www.ema.europa.eu/en/news/comirnaty-spikevax-possible-link-very-rare-cases-myocarditis-pericarditis.

  5. Pharmacovigilance Risk Assessment Committee. Signal assessment report on Myocarditis, pericarditis with Tozinameran (COVID-19 mRNA vaccine (nucleoside-modified)—COMIRNATY)—EMA/PRAC/575791/2021 [Internet]. Amsterdam (NL): European Medicines Agency; 2021 Dec 02 [cited 2022 Dec 28]. https://www.ema.europa.eu/en/documents/prac-recommendation/signal-assessment-report-myocarditis-pericarditis-tozinameran-COVID-19-mrna-vaccine_en.pdf.

  6. Pharmacovigilance Risk Assessment Committee. Signal assessment report on myocarditis and pericarditis with Spikevax—COVID-19 mRNA vaccine (nucleoside-modified)—EMA/PRAC/759618/2021 [Internet]. Amsterdam (NL): European Medicines Agency; 2021 Dec 02 [cited 2022 Dec 28]. Available from: https://www.ema.europa.eu/en/documents/prac-recommendation/signal-assessment-report-myocarditis-pericarditis-spikevax-previously-COVID-19-vaccine-moderna-covid_en.pdf.

  7. European Medicines Agency. PRAC recommendations on safety signals [Internet]. Amsterdam (NL): European Medicines Agency; 2022 [cited 2022 Dec 28]. https://www.ema.europa.eu/en/human-regulatory/post-authorisation/pharmacovigilance/signal-management/prac-recommendations-safety-signals.

  8. Ibrahim H, Abdo A, El Kerdawy AM, Eldin AS. Signal detection in pharmacovigilance: a review of informatics-driven approaches for the discovery of drug-drug interaction signals in different data sources. Artif Intell Life Sci. 2021;1: 100005. https://doi.org/10.1016/j.ailsci.2021.100005.

    Article  CAS  Google Scholar 

  9. van Puijenbroek EP, Bate A, Leufkens HG, Lindquist M, Orre R, Egberts AC. A comparison of measures of disproportionality for signal detection in spontaneous reporting systems for adverse drug reactions. Pharmacoepidemiol Drug Saf. 2002;11:3–10. https://doi.org/10.1002/pds.668.

    Article  CAS  PubMed  Google Scholar 

  10. Grundmark B, Holmberg L, Garmo H, Zethelius B. Reducing the noise in signal detection of adverse drug reactions by standardizing the background: a pilot study on analyses of proportional reporting ratios-by-therapeutic area. Eur J Clin Pharmacol. 2014;70:627–35. https://doi.org/10.1007/s00228-014-1658-1.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  11. Candore G, Juhlin K, Manlik K, Thakrar B, Quarcoo N, Seabroke S, Wisniewski A, Slattery J. Comparison of statistical signal detection methods within and across spontaneous reporting databases. Drug Saf. 2015;38:577–87. https://doi.org/10.1007/s40264-015-0289-5.

    Article  CAS  PubMed  Google Scholar 

  12. Inspections, Human Medicines, Pharmacovigilance and Committees Division. Screening for adverse reactions in EudraVigilance—EMA/849944/2016 [Internet]. Amsterdam (NL): European Medicines Agency; 2016 Dec 19 [cited 2022 Dec 28]. https://www.ema.europa.eu/en/documents/other/screening-adverse-reactions-eudravigilance_en.pdf.

  13. Böhm R, von Hehn L, Herdegen T, Klein HJ, Bruhn O, Petri H, Höcker J. OpenVigil FDA—inspection of U.S. American adverse drug events pharmacovigilance data and novel clinical applications. PLoS ONE. 2016;11: e0157753. https://doi.org/10.1371/journal.pone.0157753.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  14. Sardella M, Lungu C. Evaluation of quantitative signal detection in EudraVigilance for orphan drugs: possible risk of false negatives. Ther Adv Drug Saf. 2019;10:2042098619882819. https://doi.org/10.1177/2042098619882819.

    Article  PubMed  PubMed Central  Google Scholar 

  15. Pharmacovigilance Office, Quality and Safety of Medicines Department. Designated Medical Event (DME) list—EMA/326038/2020 [Internet]. Amsterdam (NL): European Medicines Agency; 2020 Jun 15 (cited 2022 Dec 28). https://www.ema.europa.eu/en/documents/other/designated-medical-event-dme-list_en.xlsx.

  16. Kumar A, Singh R, Kaur J, Pandey S, Sharma V, Thakur L, et al. Wuhan to world: the COVID-19 pandemic. Front Cell Infect Microbiol. 2021;11: 596201. https://doi.org/10.3389/fcimb.2021.596201.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  17. Brown P, Waite F, Larkin M, Lambe S, McShane H, Pollard AJ, et al. “It seems impossible that it’s been made so quickly”: a qualitative investigation of concerns about the speed of COVID-19 vaccine development and how these may be overcome. Hum Vaccin Immunother. 2022;18:2004808. https://doi.org/10.1080/21645515.2021.2004808.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  18. European Medicines Agency. EMA Human medicines database [Database on the internet]. Amsterdam (NL): European Medicines Agency; 2022 [Accessed 2022 Dec 28]. www.ema.europa.eu/en/medicines.

  19. Organisation for Economic Co-operation and Development. Access to COVID-19 vaccines: Global approaches in a global crisis [Internet]. Paris (FR): OECD; 2021 Mar 18 [cited 2022 Dec 28]. https://www.oecd.org/coronavirus/policy-responses/access-to-COVID-19-vaccines-global-approaches-in-a-global-crisis-c6a18370/.

  20. Kant A, van Hunsel F, van Puijenbroek E. Numbers of spontaneous reports: How to use and interpret? Br J Clin Pharmacol. 2022;88:1365–8. https://doi.org/10.1111/bcp.15024.

    Article  CAS  PubMed  Google Scholar 

  21. Human Medicines Division. 2022 Annual Report on EudraVigilance for the European Parliament, the Council and the Commission (Reporting period: 1 January to 31 December 2022)—EMA/900566/2022 [Internet]. Amsterdam (NL): European Medicines Agency; 2023 Mar 20 [Cited 2023 April 11]. https://www.ema.europa.eu/en/documents/report/2022-annual-report-eudravigilance-european-parliament-council-commission_en.pdf.

  22. Maignen F, Hauben M, Dogné JM. A mathematical framework to quantify the masking effect associated with the confidence intervals of measures of disproportionality. Ther Adv Drug Saf. 2017;8:231–44. https://doi.org/10.1177/2042098617704143.

    Article  PubMed  PubMed Central  Google Scholar 

  23. Juhlin K, Ye X, Star K, Norén GN. Outlier removal to uncover patterns in adverse drug reaction surveillance—a simple unmasking strategy. Pharmacoepidemiol Drug Saf. 2013;22:1119–29. https://doi.org/10.1002/pds.3474.

    Article  PubMed  Google Scholar 

  24. Maignen F, Hauben M, Hung E, Van Holle L, Dogne JM. Assessing the extent and impact of the masking effect of disproportionality analyses on two spontaneous reporting systems databases. Pharmacoepidemiol Drug Saf. 2014;23:195–207. https://doi.org/10.1002/pds.3529.

    Article  PubMed  Google Scholar 

  25. Montastruc F, Salvo F, Arnaud M, Bégaud B, Pariente A. Signal of gastrointestinal congenital malformations with antipsychotics after minimising competition bias: a disproportionality analysis using data from Vigibase(®). Drug Saf. 2016;39:689–96. https://doi.org/10.1007/s40264-016-0413-1.

    Article  CAS  PubMed  Google Scholar 

  26. Pharmacovigilance Risk Assessment Committee. List of signals discussed at PRAC since September 2012—EMA/PRAC/530804/2013 [Internet]. Amsterdam (NL): European Medicines Agency; 2022 May 05 [Accessed 2022 May 15]. https://www.ema.europa.eu/en/human-regulatory/post-authorisation/pharmacovigilance/signal-management/prac-recommendations-safety-signals.

  27. European Medicines Agency and Heads of Medicines Agencies. Guideline on good pharmacovigilance practices (GVP) Module IX – Signal management (Rev 1)—EMA/827661/2011 [Internet]. Amsterdam (NL): European Medicines Agency; 2017 Oct 09 [cited 2022 Dec 28]. https://www.ema.europa.eu/en/documents/scientific-guideline/guideline-good-pharmacovigilance-practices-gvp-module-ix-signal-management-rev-1_en.pdf.

  28. Pariente A, Avillach P, Salvo F, Thiessard F, Miremont-Salamé G, Fourrier-Reglat A, et al. Effect of competition bias in safety signal generation: analysis of a research database of spontaneous reports in France. Drug Saf. 2012;35:855–64. https://doi.org/10.1007/BF03261981.

    Article  PubMed  Google Scholar 

  29. Candore G, Monzon S, Slattery J, Piccolo L, Postigo R, Xurz X, et al. The impact of mandatory reporting of non-serious safety reports to eudravigilance on the detection of adverse reactions. Drug Saf. 2022;45:83–95. https://doi.org/10.1007/s40264-021-01137-0.

    Article  PubMed  Google Scholar 

  30. Harpaz R, DuMouchel W, Van Manen R, Nip A, Bright S, Szarfman A, et al. Signaling COVID-19 vaccine adverse events. Drug Saf. 2022;45:765–80. https://doi.org/10.1007/s40264-022-01186-z.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  31. Vidlin S. Unmasking data in the COVID-19 vaccine era [Internet]. Uppsala (SE): Uppsala Reports; 2022 Nov 25 [cited 2022 Dec 28]. https://www.uppsalareports.org/articles/unmasking-data-in-the-COVID-19-vaccine-era/.

  32. Montastruc JL, Sommet A, Bagheri H, Lapeyre-Mestre M. Benefits and strengths of the disproportionality analysis for identification of adverse drug reactions in a pharmacovigilance database. Br J Clin Pharmacol. 2011;72:905–8. https://doi.org/10.1111/j.1365-2125.2011.04037.x.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  33. Borg JJ, Aislaitner G, Pirozynski M, Mifsud S. Strengthening and rationalizing pharmacovigilance in the EU: where is Europe heading to? A review of the new EU legislation on pharmacovigilance. Drug Saf. 2011;34:187–97. https://doi.org/10.2165/11586620-000000000-00000.

    Article  PubMed  Google Scholar 

  34. Butler D, Vucic K, Straus S, Cupelli A, Micallef B, Serracino-Inglott A, et al. Regulatory experience of handling Risk Management Plans (RMPs) for medicinal products in the EU. Expert Opin Drug Saf. 2021;20:815–26. https://doi.org/10.1080/14740338.2021.1909569.

    Article  PubMed  Google Scholar 

  35. European Medicines Agency. New measures to minimise risk of meningioma with medicines containing nomegestrol or chlormadinone—EMA/874908/2022 [Internet]. Amsterdam (NL): European Medicines Agency; 2022 Nov 28 (cited 2022 Dec 28). https://www.ema.europa.eu/en/documents/referral/new-measures-minimise-risk-meningioma-medicines-containing-nomegestrol-chlormadinone_en.pdf.

  36. Nguyen P, Hoisnard L, Neumann A, Zureik M, Weill A. Utilisation prolongée de l’acétate de chlormadinone et risque de méningiome intracrânien: une étude de cohorte à partir des données du SND. Saint-Denis (FR): EPI-PHARE; 2021 April [cited 2022 Dec 28]. https://www.epi-phare.fr/app/uploads/2021/04/epi-phare_rapport_acetate_chlormadinone_avril-2021-1.pdf.

  37. Nguyen P, Hoisnard L, Neumann A, Zureik M, Weill A. Utilisation prolongée de l’acétate de nomégestrol et risque de méningiome intracrânien: une étude de cohorte à partir des données du SNDS. Saint-Denis (FR): EPI-PHARE; 2021 April [cited 2022 Dec 28]. https://www.epi-phare.fr/app/uploads/2021/04/epi-phare_rapport_acetate_nomegetrol_avril-2021.pdf.

  38. Durand J, Dogné JM, Cohet C, Browne K, Gordillo-Marañón M, Piccolo L, et al. Safety monitoring of COVID-19 vaccines: perspective from the European Medicines Agency. Clin Pharmacol Ther. 2023;113:1223–34. https://doi.org/10.1002/cpt.2828.

    Article  PubMed  Google Scholar 

  39. European Medicines Agency and Heads of Medicines Agencies. Guideline on good pharmacovigilance practices (GVP) Module IX Addendum I—methodological aspects of signal detection from spontaneous reports of suspected adverse reaction—EMA/209012/2015 [Internet]. Amsterdam (NL): European Medicines Agency; 2017 Oct 09 [cited 2022 Dec 28]. https://www.ema.europa.eu/en/documents/scientific-guideline/guideline-good-pharmacovigilance-practices-gvp-module-ix-addendum-i-methodological-aspects-signal_en.pdf.

  40. Maignen F, Hauben M, Hung E, Holle LV, Dogne JM. A conceptual approach to the masking effect of measures of disproportionality. Pharmacoepidemiol Drug Saf. 2014;23:208–17. https://doi.org/10.1002/pds.3530.

    Article  PubMed  Google Scholar 

  41. Bate A, Hornbuckle K, Juhaeri J, Motsko SP, Reynolds RF. Hypothesis-free signal detection in healthcare databases: finding its value for pharmacovigilance. Ther Adv Drug Saf. 2019;10:2042098619864744. https://doi.org/10.1177/2042098619864744.

    Article  PubMed  PubMed Central  Google Scholar 

  42. Lai LY, Arshad F, Areia C, Alshammari TM, Alghoul H, Casajust P, et al. Current approaches to vaccine safety using observational data: a rationale for the EUMAEUS (Evaluating Use of Methods for Adverse Events Under Surveillance-for Vaccines) study design. Front Pharmacol. 2022;13: 837632. https://doi.org/10.3389/fphar.2022.837632.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  43. Lavertu A, Vora B, Giacomini KM, Altman R, Rensi S. A new era in pharmacovigilance: toward real-world data and digital monitoring. Clin Pharmacol Ther. 2021;109:1197–202. https://doi.org/10.1002/cpt.2172.

    Article  PubMed  PubMed Central  Google Scholar 

  44. Pappa D, Stergioulas LK. Harnessing social media data for pharmacovigilance: a review of current state of the art, challenges and future directions. Int J Data Sci Anal. 2019;8:113–35. https://doi.org/10.1007/s41060-019-00175-3.

    Article  Google Scholar 

  45. Flynn R, Plueschke K, Quinten C, Strassmann V, Duijnhoven RG, Gordillo-Marañon M, et al. Marketing authorization applications made to the European Medicines Agency in 2018–2019: what was the contribution of real-world evidence? Clin Pharmacol Ther. 2022;111:90–7. https://doi.org/10.1002/cpt.2461.

    Article  PubMed  Google Scholar 

  46. Arlett P, Kjaer J, Broich K, Cooke E. Real-world evidence in EU medicines regulation: enabling use and establishing value. Clin Pharmacol Ther. 2022;111:21–3. https://doi.org/10.1002/cpt.2479.

    Article  PubMed  Google Scholar 

  47. European Medicines Agency and Heads of Medicines Agencies. European medicines agencies network strategy to 2025—EMA/85501/2020 [Internet]. Amsterdam (NL): European Medicines Agency; 2020 (cited 2022 Dec 28). Available from: https://www.ema.europa.eu/en/documents/report/european-union-medicines-agencies-network-strategy-2025-protecting-public-health-time-rapid-change_en.pdf.

Download references

Author information

Authors and Affiliations

  1. Medicines Authority, Sir Temi Żammit Buildings, Malta Life Sciences Park, San Ġwann, SĠN 3000, Malta

    Benjamin Micallef, Anthony Serracino-Inglott & John-Joseph Borg

  2. University of Namur, Rue de Bruxelles 61, 5000, Namur, Belgium

    Jean-Michel Dogné

  3. College of Medicine and Health, University of Exeter, Exeter, UK

    Janet Sultana

  4. Medicines Evaluation Board, Utrecht, The Netherlands

    Sabine M. J. M. Straus

  5. School of Pharmacy, Department of Biology, University of Tor Vergata, Rome, Italy

    Robert Nisticò & John-Joseph Borg

  6. Department of Pharmacy, University of Malta, Msida, Malta

    Anthony Serracino-Inglott

Authors
  1. Benjamin Micallef
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jean-Michel Dogné
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Janet Sultana
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Sabine M. J. M. Straus
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Robert Nisticò
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Anthony Serracino-Inglott
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. John-Joseph Borg
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to John-Joseph Borg.

Ethics declarations

Consent to participate

Not applicable.

Funding

This study was not funded.

Conflict of interest

All authors declare that they have no conflicts of interest. The views expressed in this article are the personal views of the authors and may not be understood or quoted as being made on behalf of or reflecting the position of the regulatory agency/agencies or organisations with which the authors are employed/affiliated.

Availability of data and material

No data are available.

Ethics approval

Not applicable.

Consent for publication

Not applicable.

Code availability

Not applicable.

Author contributions

J.J.B contributed to study conceptualisation, methodology and data interpretation. B.M carried out the data collection, data analysis and data interpretation. All authors contributed to the writing and review of the manuscript. All authors read and approved the final version of the manuscript.

Supplementary Information

Below is the link to the electronic supplementary material.

Supplementary file1 (PDF 395 KB)

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Micallef, B., Dogné, JM., Sultana, J. et al. An Exploratory Study of the Impact of COVID-19 Vaccine Spontaneous Reporting on Masking Signal Detection in EudraVigilance. Drug Saf 46, 1089–1103 (2023). https://doi.org/10.1007/s40264-023-01346-9

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s40264-023-01346-9

Search

Navigation