The Development and Evaluation of Triage Algorithms for Early Discovery of Adverse Drug Interactions

Abstract

Background

Around 20 % of all adverse drug reactions (ADRs) are due to drug interactions. Some of these will only be detected in the postmarketing setting. Effective screening in large collections of individual case safety reports (ICSRs) requires automated triages to identify signals of adverse drug interactions. Research so far has focused on statistical measures, but clinical information and pharmacological characteristics are essential in the clinical assessment and may be of great value in first-pass filtering of potential adverse drug interaction signals.

Objective

The aim of this study was to develop triages for adverse drug interaction surveillance, and to evaluate these prospectively relative to clinical assessment.

Methods

A broad set of variables were considered for inclusion in the triages, including cytochrome P450 (CYP) activity, explicit suspicions of drug interactions as noted by the reporter, dose and treatment overlap, and a measure of interaction disproportionality. Their unique contributions in predicting signals of adverse drug interactions were determined through logistic regression. This was based on the reporting in the WHO global ICSR database, VigiBase™, for a set of known adverse drug interactions and corresponding negative controls. Three triages were developed, each producing an estimated probability that a given drug–drug–ADR triplet constitutes an adverse drug interaction signal. The triages were evaluated against two separate benchmarks derived from expert clinical assessment: adverse drug interactions known in the literature and prospective adverse drug interaction signals. For reference, the triages were compared with disproportionality analysis alone using the same benchmarks.

Results

The following were identified as valuable predictors of adverse drug interaction signals: plausible CYP metabolism; notes of suspected interaction by the reporter; and reports of unexpected therapeutic response, altered therapeutic effect with dose information and altered therapeutic effect when only two drugs had been used. The new triages identified reporting patterns corresponding to both prospective signals of adverse drug interactions and already established ones. They perform better than disproportionality analysis alone relative to both benchmarks.

Conclusions

A range of predictors for adverse drug interaction signals have been identified. They substantially improve signal detection capacity compared with disproportionality analysis alone. The value of incorporating clinical and pharmacological information in first-pass screening is clear.

Appendices

Appendix 1: Extraction of Known Adverse Drug Interactions and Drug–Drug–Adverse Drug Reactions (DDAs) not Known to Interact

We have previously described in detail how a reference set of known adverse drug interactions and DDAs not known to interact was constructed based on Stockley’s Interaction Alerts [9]. The main steps are repeated here.

Stockley’s Interaction Alerts [11] is an electronic quick ready-reference with information on more than 40,000 drug–drug, drug–alcohol and drug–food pairs. It is one component of Stockley’s Drug Interactions [16], a well-renowned international drug interaction reference. Among other things, its textual descriptions list adverse drug reactions (ADRs) that might result from the drug interactions.

For drug pairs entered between the first quarter of 2007 and the third quarter of 2009, the drug names were matched to substances in the WHO Drug Dictionary Enhanced [13], and pairs where either drug was ethanol or nicotine were excluded. From each free text description, WHO Adverse Reaction Terminology (WHO-ART) preferred terms [13] were extracted using a customized algorithm [9, 26], and were then combined with the drug pair to form DDAs. For each DDA, a note was made of the quarter when it was first included into Stockley’s Interaction Alerts.

Any DDA was excluded if its ADR was part of a variable to be tested for inclusion into the triages (see the definitions of variables 1, 2, and 6 in Table 1). Furthermore, any DDA was excluded that did not fulfil basic reporting requirements into the WHO global individual case safety report database, VigiBase™ [13], which is the main data source in this study: After removal of suspected duplicates [14], at least three reports were required where the two drugs were listed as suspected, interacting, or concomitant, entered between 1 January 1990 and the quarter prior to inclusion of the DDA into Stockley’s Interaction Alerts; and at least one report was required with both drugs listed as suspected or interacting, in that same time period. Finally, for the remaining DDAs, the extracted ADRs were manually validated against the source text, and false hits were removed.

This yielded 324 known adverse drug interactions. The previously reported number, 322 [9], excludes two DDAs that fulfilled the reporting requirements in the quarter prior to inclusion into Stockley’s Interaction Alerts, but not at the database end date.

The reference set was completed by including a comparison group of DDAs not known to interact. All drugs in Stockley’s Interaction Alerts were paired, and those pairs that were actually listed were excluded. Thereafter the remaining pairs were combined with all ADR terms extracted from Stockley’s Interaction Alerts in the identification of known adverse drug interactions, thus forming a new group of DDAs. From this group, DDAs with certain drugs or ADRs were excluded, as described above for the known adverse drug interactions. For the remaining DDAs, it was identified for what quarters they fulfilled the VigiBase™ reporting requirements used for the known adverse drug interactions. Finally, 20 DDAs not known to interact were randomly chosen for each known adverse drug interaction, matched on quarter of data extraction. (The same DDA could not be selected for multiple quarters.)

Appendix 2: Regression Modelling

For the purpose of empirically deriving triages, lasso logistic regression was used. The lasso shrinkage restricts the coefficients so that the sum of their absolute values cannot exceed a predefined limit [27]. This reduces the risk of over-fitting to the available dataset, which in turn should yield better predictive performance on new data. At the same time, the lasso performs model selection by setting some coefficients to exactly zero, i.e. it allows only the best predictive variables into the model. Shrinkage regression of this type has grown in usage over recent years, and has been repeatedly and successfully applied in the medical domain [28, 29]. We used the implementation of lasso logistic regression provided by Friedman et al. [30].

Prior to model fitting, all variables were standardized to have unit variance. However, all coefficients presented (see Tables 2, 3) have been transformed back, and should be applied to the variables’ respective original scales.

The standard approach of five-fold cross-validation was used to select the appropriate level of shrinkage [31]. Predictive performance was measured in terms of mean squared error [32]. Once the optimal amount of shrinkage had been determined, the entire data set was used to fit a single model.

Appendix 3: Construction of the Evaluation Set

Because clinical assessment is tedious, the size of the evaluation set was limited to 100 DDAs. To make the evaluation as informative as possible, it was important to ensure coverage of likely interaction signals as well as unlikely ones. To this end, the random selection of DDAs into the evaluation set was stratified according to the predicted probabilities of the two lean triages. Creating strata by simply dividing the 0–1 interval into equally spaced pieces would yield disproportionately many DDAs with high probabilities, since the distribution of DDAs is heavily skewed towards zero. At the same time, to stratify according to the cumulative distribution of predicted probabilities would yield unsatisfactorily many DDAs with the lowest possible probability, since those DDAs amount to more than 70 % of all included DDAs. We used an intermediate alternative, whereby each stratum was created by incrementally increasing its upper limit. A stratum was considered complete as soon as it was filled up by more DDAs than its specific target size, which was computed as the ratio between the remaining number of DDAs and the remaining number of strata.