Figures 2, 3, and 4 provide an overview of the percentage of overlap and recall between EVDAS and FAERS/VigiBase® data sources for scenario 1, i.e., application of identical signal detection methods in all three databases. Table 3 summarizes the descriptive statistics. Appendix A in the ESM lists all values of overlap/recall for the individual products.
Table 3 Summary of recall and overlap for scenario 1 for all 100 substances Both overlap and recall increase with the level of the MedDRA® hierarchy (note, while we are not suggesting that signal detection is done at HLT or SMQ level, medical review of a PT-level safety signal would normally find related medical concepts, i.e., at the HLT and/or SMQ level). At the HLT or SMQ level overlap and recall indicate little differentiation between the signal information generated from each of the data sources. Distributions are skewed left, with skewness more pronounced at the higher MedDRA® levels. This implies that most products have high overlap/recall values, with the exception of a small number of outliers (e.g., see Fig. 4, where this is most prominent). Figures 2, 3, 4, 8, 9, 10, 11, 12, and 13 reflect the results for all 100 substances.
In order to illustrate how the signals in EVDAS are coincident with signals from the FAERS and VigiBase® sources, three examples are presented.
The first example (Fig. 5) shows fluticasone and salmeterol, a prescription medication indicated for use in asthma and chronic obstructive pulmonary disease, with almost identical signals emerging from the various data sources. At the HLT or SMQ level (where similar terms are often grouped for medical review), both overlap and recall approach 100% (i.e., recall of 97.9% and overlap of 94.2%), indicating an interchangeable representation of the medical concepts.
Nearly all signals found in EVDAS are also found in FAERS/VigiBase® (high overlap). This can be inspected in the CPL through the presence of very few light-blue items, which represent the EVDAS-only signals
Very few signals are unique to EVDAS (high recall). This can be inspected in the CPL through the near absence of red items, which represent signals not found in EVDAS.
A second example is shown in Fig. 6. This CPL shows benzonatate (a cough medicine), which has 100% overlap and low recall (45.8% at the HLT or SMQ level). The CPL plot shows that all EVDAS signals are also found in FAERS/VigiBase® (100% overlap), but there are a significant number of FAERS/VigiBase® signals not found in EVDAS, represented by the red items.
Finally, Fig. 7 shows a “typical” substance, i.e., a substance for which the recall is close to the median recall of 87.9% and the overlap is close to the median overlap of 97.7% at the HLT or SMQ level. One such “typical” substance is insulin aspart, with HLT or SMQ recall of 89.1% and HLT overlap of 98.0%.
As can be seen, there are a small number of medical concepts unique to EVDAS, usually reflecting small case counts (the item sizes are small), while very few signals are present in FAERS/VigiBase® that are not captured in EVDAS (red items). These “missing” signals are also reflecting smaller case counts.
Figures 8, 9, and 10 provide an overview of the percentage of overlap and recall between EVDAS and FAERS/VigiBase® data sources for scenario 2, i.e., application of the “standard” signal detection methods in the respective three databases. Table 4 summarizes the descriptive statistics. Appendix B in the ESM lists all values of overlap/recall for the individual products.
Table 4 Summary of recall and overlap for scenario 2 for all 100 substances Comparison of Tables 2 and 3 shows uniformly higher values for recall under scenario 2 relative to scenario 1, while the overlap values are consistently lower.
Regression Analysis
In order to characterize beyond the individual product level which product characteristics are affecting the level of overlap and recall, several covariates were investigated at the HLT level, including time-on-market (ToM), the difference in time-on-market between first approval in the EU versus the USA (Diff ToM), the number of EudraVigilance cases reported in the EU as a proportion of the total number of spontaneous cases reported in EudraVigilance for a product (EVDAS EU proportion), and the ratio of the number of cases reported in EudraVigilance versus FAERS (EVDAS FAERS ratio). Simple linear regression models were fitted to the data. Table 5 summarizes the results.
Table 5 Summary of results of overlap and recall linear regression models No specific covariates can be identified that systematically affect the EVDAS recall. The two variables that partially determine the overlap are the relative number of EU cases in EudraVigilance and the ratio of EVDAS cases and FAERS cases, presumably due to the differences in marketing authorizations, or market penetration in different regions. Overlap increases with those two ratios. See Fig. 11 for the fitted line graph.
Note that, not unexpectedly, the two variables EVDAS EU proportion and EVDAS FAERS ratio are correlated, as shown in Fig. 12 (P value = 0.000).