FormalPara Key Points

There was no consistent pattern in the directionality of disproportionality analysis estimates with specific comparators.

Calendar time-based restrictions anchored on market approval led to stronger false-positive signals, while anchoring on drug safety warnings led to weaker false-positive signals.

1 Introduction

Pharmacovigilance analyses using spontaneous reporting systems (SRS) are an important tool for the early detection of safety signals. Via assessing disproportionalities in the reporting of specific drug-event combinations (DECs) compared to the reporting of all other drugs in the SRS, pharmacovigilance analyses considerably improved our understanding of adverse drug reactions in recent decades [1, 2]. SRS have important advantages such as their potential to cover the whole patient population, and the fact that they are inexpensive, which makes their establishment and operation feasible even in low-income settings.

SRS-based pharmacovigilance analyses also have several inherent limitations. For example, reports often do not provide sufficient information on comorbidities and comedications to confirm that a drug caused an event. Hence, confounding by indication constitutes a major challenge. To reduce this bias, there has been a trend towards the use of restricted reference sets such as active comparators, a concept borrowed from pharmacoepidemiology [3, 4]. However, the impact of using restricted reference sets instead of the full-data reference set as comparators on other biases that are specific to SRS-based signal detection is not clear [5].

These biases include, but are not limited to, signal leakage (or confounding by association), competition bias (or masking), and different reporting biases such as under-reporting or stimulated reporting [6,7,8]. To address this knowledge gap, we used the unrestricted full-data reference set and two types of restricted reference sets based on active comparators for signal detection in two well-studied DECs with a high propensity for different sources of bias. We applied an approach based on similar principles as the quantitative bias analysis (QBA) or quasi-QBA for brevity by conducting comparisons between unrestricted and restricted reference sets to observe the relative direction of bias [9]. This was done both during the entire study period and after the application of calendar time-based restrictions anchored on external events potentially affecting reporting behavior. Our overall aim was to assess how different comparators influence the directionality of bias in pharmacovigilance analyses and also whether a single comparator could be recommended for SRS-based signal detection.

2 Methods

2.1 Data Source

For this study, we used the US Food and Drug Administration (FDA) Adverse Event Reporting System (FAERS) database [10]. The FAERS contains reports of suspected adverse drug reactions based on voluntary submissions by healthcare providers and patients and mandatory submissions by pharmaceutical companies. Information on these reports is publicly available in seven different data files linkable via a unique identifying number: (i) a demographic file with information about the patient, including age, sex, and weight; (ii) a drug file with the role of the drug in the adverse event (primary suspect drug, secondary suspect drug, concomitant drug, or interacting drug), the active ingredient and/or brand name(s); (iii) a reaction file with clinical information about the adverse event (coded via ‘preferred terms’ from the Medical Dictionary for Regulatory Activities [11]); (iv) an outcome file with information about the severity of the adverse event and potential complications; (v) a report source file; (vi) a therapy dates file with start and end dates for the reported drugs; and (vii) an indications for use file with information about the underlying indications for the reported drugs (coded using preferred terms). The design and the analyses described below were based on a protocol that was finalized prior to the initiation of the study.

2.2 Rationale for the Choice of DECs

The two DECs used for the purposes of this study were (i) the direct oral anticoagulant (DOAC) rivaroxaban and hepatotoxicity and (ii) the sodium-glucose cotransporter 2 (SGLT2) inhibitor canagliflozin and acute kidney injury (AKI). These DECs were chosen because of their high propensity for bias, specifically for false-positive signals because of the potential for different types of stimulated reporting.

2.2.1 Rivaroxaban and Hepatotoxicity

Rivaroxaban was one of the first two DOACs to be approved for the prevention of ischemic stroke among patients with non-valvular atrial fibrillation (the other one was dabigatran). Because ximelagatran, the first DOAC to be studied in a phase III trial, was not approved (or was removed from the market shortly after approval) because of its hepatotoxic risk [12], there were initial concerns with respect to the hepatic safety of DOACs.

The Randomized Evaluation of Long-Term Anticoagulation Therapy (RE-LY) and the Rivaroxaban Once Daily Oral Direct Factor Xa Inhibition Compared with Vitamin K Antagonism for Prevention of Stroke and Embolism Trial in Atrial Fibrillation (ROCKET AF) randomized controlled trials did not yield any hepatotoxic signals for dabigatran or rivaroxaban [13, 14]. Moreover, several subsequent pharmacoepidemiologic studies corroborated the absence of a hepatotoxic risk for dabigatran and rivaroxaban when compared with warfarin, a vitamin K antagonist (VKA) [15,16,17]. However, an early pharmacovigilance study by Raschi and colleagues showed signals for liver injury and liver failure for rivaroxaban (not for dabigatran) [18]. Therefore, we hypothesized these signals to be false positives resulting from notoriety bias, a bias known to occur when safety concerns such as those mentioned above exist.

2.2.2 Canagliflozin and AKI

Canagliflozin was approved in 2013 as the first SGLT2 inhibitor, a class of oral antidiabetic drugs used in the management of type 2 diabetes mellitus. Canagliflozin was followed by dapagliflozin and empagliflozin in 2014. By interfering with the co-uptake of glucose and sodium in the proximal nephron, SGLT2 inhibitors increase sodium delivery to the distal nephron, resulting in afferent arteriole vasoconstriction and a decreased glomerular filtration rate. Thus, potential nephrotoxicity of these medications was an initial safety concern.

Indeed, in June 2016, there was a safety warning issued by the FDA on the use of SGLT2 inhibitors and the risk of AKI [19]. The safety warning mentioned that between March 2013 and October 2015, FAERS documented 101 cases of AKI with a temporal relationship mostly with canagliflozin (73 patients) but also with dapagliflozin (28 patients). Despite these concerns, randomized controlled trials on the efficacy and safety of SGLT2 inhibitors in patients with chronic kidney disease (with or without type 2 diabetes) observed a lower risk of kidney failure and other serious renal events when compared with placebo [20,21,22]. Moreover, several pharmacoepidemiologic studies did not show an increased risk of AKI or serious adverse events pertaining to the urogenital system in general associated with the use of SGLT2 inhibitors, when compared with the use of dipeptidyl-peptidase-4 (DPP-4) inhibitors [23,24,25]. Despite the probable absence of an excess nephrotoxic risk with SGLT2 inhibitors overall and with canagliflozin in particular, we hypothesized that the initial safety concerns and the publication of the regulatory safety warning could lead to stimulated reporting and false-positive signals.

2.3 Study Period

For the rivaroxaban and hepatotoxicity DEC, we extracted and linked the quarterly raw data submitted to FAERS between 1 July, 2011 (the date of rivaroxaban approval by the FDA) and 30 June, 2023. For the canagliflozin and AKI DEC, the same process was repeated for the time period between 1 April, 2013 (the date of canagliflozin approval by the FDA) and 30 June, 2023. We utilized the most recent updates for each report assigned to the quarter in which the report was first made available by the FDA.

Rigorous data cleaning included the correction of misspelled drug names, the removal of duplicate reports identified by merging active ingredient and trade medication names, and the flagging of reports with the same reporting country, sex, event date, age, adverse events, and drugs. We only included the most recently updated data at the time of the first report and also removed reports with insufficient or inconsistent information. Finally, we considered only reports where the study drug was listed as a primary suspect drug or secondary suspect drug.

2.4 Outcome Definition

We aimed to maximize the specificity of our outcome definitions. Hence, we used preferred terms explicitly referring to the adverse events of interest. For the rivaroxaban and hepatoxicity DEC, we defined hepatotoxicity using the preferred terms “acute yellow liver atrophy,” “allergic hepatitis,” “alloimmune hepatitis,” “autoimmune hepatitis,” “cholestatic liver injury,” “chronic hepatitis,” “drug-induced liver injury,” “hepatic cytolysis,” “hepatic infiltration eosinophilic,” “hepatic necrosis,” “hepatic steatofibrosis,” “hepatic steatosis,” “hepatitis,” “hepatitis acute,” “hepatitis chronic active,” “hepatitis chronic persistent,” “hepatitis fulminant,” “hepatitis toxic,” “hepatocellular injury,” “hepatotoxicity,” “immune-mediated hepatic disorder,” “immune-mediated hepatitis,” “liver injury,” “mixed liver injury,” “non-alcoholic steatohepatitis,” “non-alcoholic fatty liver disease,” and “steatohepatitis”. To emulate the study by Raschi and colleagues [18], we also assessed liver failure, a severe form of hepatotoxicity, which was defined using the preferred terms “acquired hepatocerebral degeneration,” “acute hepatic failure,” “acute on chronic liver failure,” “chronic hepatic failure,” “coma hepatic,” “hepatic encephalopathy,” “hepatic failure,” “hepatogenous diabetes,” “hepatorenal failure,” “hepatorenal syndrome,” and “subacute hepatic failure”. For the canagliflozin and AKI DEC, we defined AKI using the preferred term “acute kidney injury”.

2.5 Comparators

We considered the influence of three different reference sets or comparators. The first comparator was the ‘full-data reference set’ that contained all other drugs in FAERS except for the study drug with no restrictions. This comparator acted as our benchmark for relative comparison against the two comparators that implemented restrictions. The second comparator was the ‘active-comparator reference set’ that included only reports with at least one drug with the same indication as the study drug. Hence, this comparator included both the drugs from the active comparator class (VKAs in the case of rivaroxaban and DPP-4 inhibitors in the case of canagliflozin), but also all other drugs from the same class as the study drug (non-rivaroxaban DOACs in the case of rivaroxaban and non-canagliflozin SGLT2 inhibitors in the case of canagliflozin). Reports that contained both the study drug and another drug with the same indication were excluded from the analysis. Finally, the third comparator was the ‘active-comparator class-exclusion reference set,’ a combination set that included only drugs from the active comparator class, while removing all other drugs from the same class as the study drug. Here, rivaroxaban was compared to VKAs and canagliflozin was compared to DPP-4 inhibitors. Reports containing both the study drug and a drug from the class of active comparators were excluded. An overview of the different comparators can be found in Table 1.

Table 1 Description of the three different comparators for the two drug-event combinations
Full size table

For both DECs, we hypothesized that the use of restricted comparators would decrease confounding by indication but increase the potential for reporting biases. For the rivaroxaban and hepatotoxicity DEC, we expected the impact of confounding by indication to be minor given the unclear association between thrombosis (the main indication for DOACs) and hepatotoxicity. We also expected the impact of reporting biases such as notoriety bias to be major, especially in the early years given the initial safety concerns. For the canagliflozin and AKI DEC, we expected the impact of confounding by indication to be strong given the well-established association between type 2 diabetes (the main indication for SGLT2 inhibitors) and kidney disease. We also expected the impact of reporting biases such as notoriety bias to be moderate and to vary over time depending on the timepoint of the FDA safety warning.

2.6 Calendar Time Restrictions

We also restricted the analysis time periods to consider meaningful events that may have influenced reporting behavior, allowing a further elucidation of the impact of different biases over time. For both DECs, the follow-up in the primary analyses ended 30 June, 2023, which was the latest date of data availability. For the rivaroxaban and hepatotoxicity DEC, we restricted the follow-up in a sensitivity analysis to September 2013, thereby mimicking the pharmacovigilance study by Raschi and colleagues that detected safety signals for rivaroxaban [18]. The rationale was that newly marketed drugs should have a boost in reporting shortly after market entry [26]. This early boost could be even more pronounced with rivaroxaban given the initial concerns over potential hepatotoxicity with DOACs. For the canagliflozin and AKI DEC, we restricted the follow-up in a sensitivity analysis to June 2016, that is until the publication of the safety warning by the FDA. The rationale was that considering only the time until the safety warning should minimize stimulated reporting.

2.7 Disproportionality Analysis Methods

To empirically study the influence of comparator selection on signal detection, we derived estimates of disproportionate reporting for each DEC with each of the comparators and within each calendar time interval, as defined above. These estimates were computed with commonly used disproportionality analysis methods including the proportional reporting ratio (PRR), the reporting odds ratio (ROR), and the relative reporting rate, the target parameter for the Bayesian approaches known as the Bayesian Confidence Propagation Neural Network (BCPNN) and the Multi-item Gamma Poisson Shrinker (MGPS) [27].

The PRR is a reporting-based analog of the relative risk, which compares reporting rates of a specific adverse event in those who reported the drug of interest with the drugs in the chosen reference set. The ROR is a reporting-based analog of the odds ratio, which compares the odds of reporting of an adverse event with the drug of interest to the odds of the same adverse event with the drugs in the reference set. The relative reporting rate compares the rate of reporting the ADR of interest against what would be expected under the assumption of independent reporting.

Both the PRR and the ROR tend to produce false-positive signals in settings of sparse data. In such settings, Bayesian implementations of the relative reporting rate such as BCPNN and MGPS can shrink the estimates because they allow for the specification of a priori knowledge, or beliefs, about a DEC. This then enables the calculation of estimates with reasonable properties even when only small amounts of data are available [28]. Of note, as more data are collected, these estimates become less dependent on the prior information and more reflective of the information housed in the data; as a result, when sample sizes become large enough, they will resemble their frequentist counterparts. Hence, Bayesian estimators possess favorable properties in the context of detecting drug safety signals for rare events, or early on in the marketed period of a new medication.

A criticism of these estimators is the potential for specifying prior subjective beliefs that may make a drug appear more or less safe. However, in our analyses, and in pharmacovigilance in general, we used the BCPNN that specifies priors based on conservative assumptions about the reporting relationship [29]. The method incorporates an independence assumption that mitigates concerns surrounding false-positive signals [29]. Of note, both DECs used in our analyses were not expected to produce small enough counts to produce major discrepancies in the signals detected between the PRR, ROR, and BCPNN algorithms.

The MGPS is empirical Bayes, that is, the a priori information is not provided by the investigator as with the BCPNN but is estimated directly from the data. The portion of the MGPS algorithm that attempts to introduce the prior information is conducted on the analytic dataset, which makes convergence of optimization routines unlikely in small subsets of a spontaneous reporting database; such as in the case of restricted reference sets. Hence, we excluded this algorithm from our analyses given that our purpose was to ascertain the influence of reference set selection on relative reporting bias. Indeed, the potentially limited performance of the numerical procedure would make an evaluation of the impact of bias on the disproportionality analysis estimates with the MGPS challenging.

2.8 Quasi-QBA Analysis

We quantified the absolute bias comparing PRR, ROR, and IC between each of the two restricted comparators and the unrestricted comparator, which acted as a benchmark, for both DECs. To this end, we converted PRR and ROR to a logarithmic scale to enhance comparability with IC, which is already logarithmically converted. Then, we used the formula DPRES−DPUNRES, where DPRES is the disproportionality analysis estimate in the restricted comparators and DPUNRES is the disproportionality analysis estimate in the unrestricted comparator. We also quantified the absolute bias between the calendar time-restricted estimates.

In general, in order to make a generalizable recommendation for the selection of a comparator, we would expect to see similar relative relationships in the directionality of the estimates between the benchmark analysis and each of the restricted comparators for both DECs. Of note, for the two calendar time intervals, we expected to see stimulated reporting immediately after the noted external events and described the impact of each comparator on this bias by comparing between the time intervals for each DEC. All analyses were conducted using R (R Foundation, Vienna, Austria).

3 Results

3.1 Rivaroxaban and Hepatotoxicity

We did not observe a safety signal for hepatotoxicity or liver failure with rivaroxaban across comparators and disproportionality analysis methods (Table 2). However, we did observe safety signals both for hepatotoxicity and liver failure with rivaroxaban when restricting the study period until September 2013, the initial post-approval period for DOACs (Table 2). For hepatotoxicity, the rivaroxaban signal was overall stronger with the restricted comparators across disproportionality analysis methods (e.g., active-comparator reference set: PRR, 2.00: 95% confidence interval [CI] 1.55–2.58 vs full-data reference set: PRR, 1.29; 95% CI 1.06–1.58). For liver failure, the strength of the rivaroxaban signal was similar across comparators and disproportionality analysis methods.

Table 2 Disproportionality analysis estimates for rivaroxaban and hepatotoxicity and liver failure across comparators and study periods
Full size table

3.2 Canagliflozin and AKI

We observed a safety signal for AKI with canagliflozin across comparators and disproportionality analysis methods (Table 3). The canagliflozin signal was stronger with the unrestricted comparator (e.g., active-comparator reference set: PRR, 2.71; 95% CI 2.56–2.87 vs full-data reference set: PRR, 8.90; 95% CI 8.51–9.30). When restricting the study period until June 2016, that is prior to the safety warning by the FDA, the canagliflozin signal persisted (Table 3). Among the two comparators yielding positive signals, the signal was stronger with the unrestricted comparator (e.g., active-comparator reference set: PRR, 1.52; 95% CI 1.21–1.90 vs full-data reference set: PRR, 5.38; 95% CI 4.75–6.10).

Table 3 Disproportionality analysis estimates for canagliflozin and acute kidney injury across comparators and study periods
Full size table

3.3 Quasi-QBA Results

The results of a quasi-QBA with the unrestricted comparator in the entire study period as the reference group (‘benchmark’) are presented in Tables 4, 5, 6. For the rivaroxaban and hepatotoxicity DEC, we observed positive values indicating an increasing shift in all estimates compared to the reference group, with the calendar time restrictions increasing the magnitude of the shift by multiples (Table 4). For the rivaroxaban and liver failure DEC, we observed similar results; an exception were the estimates with the active-comparator/class-exclusion comparator in the entire study period, which pull in a negative direction from the reference group (Table 5). For the canagliflozin and AKI DEC, the estimates were all negative; this highlights a reduction in the estimates compared with the reference group where the magnitude of the shift is of the same order between calendar time intervals (Table 6). Overall, these observations failed to demonstrate consistency in the relative behavior of the estimates (directionality) against the benchmark when using similar reference set definitions.

Table 4 Absolute differencea in disproportionality analysis estimates between the two restricted comparators and the unrestricted comparator for rivaroxaban and hepatotoxicity
Full size table
Table 5 Absolute differencea in disproportionality analysis estimates between the two restricted comparators and the unrestricted comparator for rivaroxaban and liver failure
Full size table
Table 6 Absolute differencea in disproportionality analysis estimates between the two restricted comparators and the unrestricted comparator for canagliflozin and acute kidney injury
Full size table

We also illustrated the potential influence of the chosen comparator (and disproportionality analysis methods) on bias, thereby focusing on a subset of the analyses. Figure 1A–C show that the rivaroxaban signal for hepatotoxicity in the analyses restricted to the initial post-approval period was augmented with the restricted comparators when using PRR or ROR (not when using BCPNN in the case of the active-comparator class-exclusion comparator). Figure 2A–C show that the rivaroxaban signal for liver failure in the same time interval remained unaffected by the choice of comparator or disproportionality analysis method. Figure 3A–C show that the canagliflozin signal for AKI was weakened when using restricted comparators across disproportionality analysis methods in the time period prior to the FDA safety warning (but also in the entire study period).

Fig. 1
figure 1

Forest plots summarizing the analyses on the rivaroxaban signal for hepatotoxicity during the initial post-approval period (2011–2013). A proportional rate ratio (PRR); B reporting odds ratio (ROR); C information component (IC). CI confidence interval, CrI credible interval

Full size image
Fig. 2
figure 2

Forest plots summarizing the analyses on the rivaroxaban signal for liver failure during the initial post-approval period (2011–2013). A proportional rate ratio (PRR); B reporting odds ratio (ROR); C information component (IC). CI confidence interval, CrI credible interval

Full size image
Fig. 3
figure 3

Forest plots summarizing the analyses on the canagliflozin signal for acute kidney injury until prior to the US Food and Drug Administration safety warning (2013–2016). A proportional rate ratio (PRR); B reporting odds ratio (ROR); C information component (IC). CI confidence interval, CrI credible interval

Full size image

4 Discussion

In this paper, we applied a quasi-QBA approach using two DECs with a high propensity for reporting biases and false-positive signals. Specifically, we conducted a relative comparison of different restricted comparators against the unrestricted comparator of the full-data reference set across disproportionality analysis methods, while also exploring potential effects of calendar time. Overall, we could not identify a predictable pattern in the direction or the magnitude of disproportionality analysis estimates when using specific comparators, while the heterogeneity among different disproportionality analysis estimates was low. However, calendar time-based restrictions anchored on relevant external events seemed to have a considerable impact.

In the presence of residual bias, it is often recommended to make assumptions about its directionality in order to interpret a decision made from a classical hypothesis test as being robust to the bias. However, these assumptions should not lead to heuristics and blanket approaches. A helpful example in this regard can be derived from pharmacoepidemiology, an area adjacent to pharmacovigilance. Non-differential misclassification, a type of information bias, is often assumed to merely attenuate effect sizes, which supports the notion that the presence of this bias could be ignored when a statistically significant association is observed. However, recent methodological work showed that this assumption has several potential exceptions, which necessitate more critical approaches to the evaluation of the bias [30].

In a recent paper, it was argued that active comparators may decrease bias in signal detection studies and attenuate disproportionality analysis estimates in the context of SGLT2 inhibitors [31]. We considered whether this observation might generalize using a quasi-QBA and could not confirm this: while active comparator-based approaches performed best for the DEC involving the SGLT2 inhibitor canagliflozin and AKI, this was not the case for the DEC involving the DOAC rivaroxaban and hepatotoxicity or liver failure. Indeed, during the initial post-approval period of DOACs, that is in a time period where stimulated reporting is to be expected, all restricted comparators yielded false-positive signals stronger or of equal magnitude than the unrestricted comparator, depending on the specific outcome and disproportionality analysis method used.

Our results indicate that the influence of bias will not consistently decrease when using certain comparators. Rather, it seems that active comparators perform better than the unrestricted full-data reference set in situations where confounding by indication is a major concern (i.e., the canagliflozin AKI DEC), but not necessarily in others (i.e., the rivaroxaban and hepatotoxicity DEC). In fact, the use of restricted comparators for the rivaroxaban and hepatotoxicity DEC could even introduce or augment competition bias given the large number of hepatotoxic medications.

Our results also showed that the use of different disproportionality analysis methods, either frequentist (PRR, ROR) or Bayesian (BCPNN) does not seem to have a major impact on effect estimates. Heterogeneity among disproportionality analysis methods was only observed for the rivaroxaban signal for hepatotoxicity in the initial post-approval period of the medication. In that analysis, the signal was stronger with the restricted comparators when using PRR or ROR but not when using BCPNN for the active-comparator class-exclusion reference set. The consistency among disproportionality analysis methods has also been described elsewhere [8] and was expected given the similarity in the signal detection performance of the different methods when sufficient data are available.

For each comparator, we also explored the impact of bias through additional calendar time-based restrictions. For the rivaroxaban and hepatotoxicity DEC, restricting the study period to the first few years post-drug approval enhanced bias. An initial rise in the number of reports for a given drug in the first years of availability followed by a decline despite subsequent increases in prescribing rates is known in the signal detection literature as the Weber effect, a type of stimulated reporting [32]. As mentioned above, active-comparator-based approaches performed either worse or similar to the unrestricted approach in this setting.

For the canagliflozin and AKI DEC, calendar time-based restriction was anchored on the issuance of a safety warning by the FDA [19]. Previous work by our group and others on the potential effects of safety warnings, alerts, and drug label changes by regulatory authorities as well as media coverage, major events such as the COVID-19 pandemic, or prominently published case reports has suggested that DECs can be affected by stimulated reporting [33,34,35,36,37]. Here, when restricting the study period until before the safety warning, there was a decrease in the strength of the signal for canagliflozin across comparators, with active comparators performing best.

Our study has some strengths. First, the use of a very large database such as FAERS allowed the calculation of precise effect estimates for both DECs, even after restricting the study periods, thus facilitating our quasi-QBA approach. Second, the use of two different DECs from separate areas of clinical pharmacology, three different comparators (restricted and unrestricted), and three different disproportionality analysis estimates (frequentist and Bayesian) yielded a comprehensive overview regarding the shift of estimates of signal detection under different conditions. Thus, our study presents an addition to previous methodological work in pharmacovigilance on the use of restricted comparators [38, 39] and the interpretation of different disproportionality analysis estimates [40].

Our study also has limitations. First, the presented analyses are not able to safely disentangle the different sources of bias, as departures from the expected ‘null’ effect are probably an amalgam of various reporting biases and confounding. That being said, given that spontaneous reporting data are known to be influenced by multiple forms of unobservable biases, which can occur both at the reporter level and also because of external factors, an attempt to isolate the effect of a particular biasing factor would not be realistic. Rather, our interest lied in whether the restriction of the data to a defined comparator is preferable for the identification of signals. Second, confounding is possible. Given that the probability of missing values for demographic characteristics in spontaneous reports occurring in a ‘random’ manner is low, we decided against the adjustment for age and sex in our analyses because this restriction could introduce selection bias. Finally, our use of the unrestricted reference set as a benchmark will not allow for a direct comparison between DEC as the relative relationships will vary.

Summing up, our quasi-QBA approach showed that restricted comparators such as active comparators seem to perform better than unrestricted comparators in settings with strong confounding by indication. Our analysis also showed that restrictions based on calendar time and anchored on relevant external events can have a considerable impact. Future studies testing the use of restrictions in the definition of comparators and of the study period for more DECs in different therapeutic areas could provide additional evidence. That being said, we think that some preliminary recommendations can be made based on our findings. First, active comparators should not be considered a panacea in pharmacovigilance; its use should be reserved for DECs with a high propensity for confounding by indication. Second, calendar time-based restrictions should be incorporated in the analytical plan when there is a well-justified rationale that external events (e.g., regulatory safety warnings or drug label changes, media coverage, pandemics, prominently published case reports) may affect reporting behavior. Considering the totality of generated results, investigators could then deliver the best possible evidence to triage detected signals, and inform subsequent pharmacoepidemiologic studies.