Dear Editor,

The article of Willame et al. [1] intended to estimate background incidence rates of certain immunological diseases typically attributed to vaccine adverse effects, to contribute to benefit and risk calculations of vaccines, particularly in terms of conducting observed-to-expected analyses [2]. As a conclusion the authors wrote: “This study demonstrated that the European ADVANCE system can identify specific autoimmune events, that age-,sex- and time-specific rates can be generated based on available tools, and that the background incidence rates are mostly consistent across selected European healthcare databases” [1].

This sounds very promising, and thus, in this sense, the author of this letter was interested in the data provided regarding the incidence rates of acute disseminated encephalomyelitis (ADEM). But, what he found was exactly what the authors of Willame et al. [1] stated in the discussion: “However, our pooled crude rates should be interpreted with caution because they were not adjusted for any relevant covariates, nor were they weighted by the data sources with the largest person-time contribution, and should only be used in the context of each individual DAP’s [Data Access Provider’s] results” [1].

This means, methodically, Willame et al. [1] merged a number of selected databases to create a new statistical population. According to recent definitions [1], this has to be named a meta-analysis, and thus, all the limitations given for such an approach have to be considered. Further, it is of particular importance to compare the results with earlier literature. But here, this was not done. Finally, an own comparison of the estimates calculated by Willame et al. [1] for ADEM with earlier published data revealed that the ADEM incidences of Willame et al. [1] differ substantially, namely by factors of at least one order of magnitude, from estimates reported earlier in literature, but surprisingly also from the figures provided by a later publication by Willame et al., which has been released in 2023 [4] (Table 1).

Table 1 ADEM background rates (according to Law/Brighton collaboration [5] modified)
Full size table

Due to the assignment of the publication to the ADVANCE project, I expected that the ADVANCE code of conduct [13] was strictly followed by the authors. From this point of view, the ADEM data of the publication suggest that this was not consistently the case because of a lack of methodical accuracy. For instance, there is no formulation of the research question, which should be the basis for ADVANCE research work, according to the applicable conventions [13].

However, the most serious finding is that heterogeneity has not been adequately considered by Willame et al. 2021 [1], although this is a major issue in any meta-analysis. In fact, they provided apparent markers of heterogeneity quite extensively in the supplementary material, but did not stress the fact that they found substantial heterogeneity between the databases (Fig. S1 and Table S2 [1]). Eighteen out of twenty I2 estimates were above 50%, which is commonly the threshold for substantial heterogeneity [14], whereby the I2 for ADEM was 96.8% (general practitioner data) and 98.3% (hospital data).

In this context, the incidence rate for ADEM provided in abstract and the main table (Table 3) of Willame et al. [1] should be considered to be misleading, particularly because the calculation of confidence intervals for the merged data was done without any adjustment for heterogeneity. As a result, the very large numbers generated extremely small confidence intervals, and this signals a precision that is not reflecting the actual situation. This becomes apparent with a look at the ADEM incidence rate, which is specified with 5.3 per 100,000 patient years and a 95% confidence interval of 5.2–5.3, whereas for the large databases selected for this meta-analysis the ADEM incidence rates range in their span from 0.95 per 100,000 patient years (UK THIN) to 11.8 per 100,000 patient years (Italy ARS).

A consequence of this particular provision of data became apparent by a statement of the EU’s Pharmacovigilance Risk Assessment Committee (PRAC), in the minutes of the PRAC meeting held on 05–08 July 2021 concerning the Pfizer/BioNTech COVID-19 mRNA-vaccine BNT 162b2: “Finally, the MAH [Marketing Authorisation Holder] should use ACCESS background rates for the analysis of cases of acute disseminated encephalomyelitis (ADEM)” [15]. However, at that time, there were no other ACCESS-related background rates for ADEM other than the figures provided by Willame et al. [1, 4]. This means that the PRAC requested concrete safety evaluations based on a source that was considered by the authors of Willame et al. [1] themselves as not suitable for such an approach, as explained above.

Thus, the above cited PRAC advice possibly affected related PRAC pharmacovigilance decisions to the detriment of patients’ safety, because in fact, Pfizer/BioNTech used the ADEM data published by Willame et al. [1] already in 2021 for observed-to-expected analyses concerning ADEM reports regarding their COVID-19 mRNA-vaccine BNT 162b2 [16]. However, on this basis, it is no surprise that the incidence of ADEM adverse event reports for BNT 162b2 in terms of per 100,000 patient years was below the background figure used. However, the same calculation based on the ADEM background incidence figures provided by Willame et al. [4] leads to the conclusion of a pharmacovigilance signal according to relevant pharmacovigilance guidelines, which was denied when based on Willame et al. [1].