FormalPara Key Points

Overrides of high dose-range alerts are common and often appropriately done.

Harm from these overrides was rare, although inappropriately overridden alerts were more associated with harm.

Increasing the positive predictive value of these alerts may reduce provider alert fatigue.

1 Introduction

Adverse drug events (ADEs) have been associated with an increase in patient length of hospital stay, and patient morbidity and mortality [1,2,3]. Medication-related clinical decision support (CDS) is a potential way to reduce the occurrence of ADEs [4, 5]. Despite these studied benefits, CDS alerts are overridden frequently, sometimes inappropriately [6,7,8,9]. Override rates are variable across institutions and dependent on the type of alert evaluated.

A patient population that is particularly susceptible to ADEs includes those in the intensive care unit (ICU), due to an increased number of high-risk medications, altered pharmacokinetics, and susceptible organ systems (e.g., organ failure) [10, 11]. Research regarding CDS alert overrides in the ICU is limited, but suggests that up to 90% of these alerts are overridden, often appropriately [8, 9]. However, when CDS alerts are overridden inappropriately, there is up to a sixfold increase in ADEs compared to appropriately overridden alerts [9]. However, studies evaluating the risk of inappropriate overrides focused on common types of CDS alerts (allergy, drug–drug interaction, geriatric, renal), but had limited characterization of dose-range alerts, which detect potential underdosing and overdosing of prescribed medications.

More broadly, few data are available regarding the impact of dosing CDS [12,13,14,15]. In one study evaluating medication prescriptions, one in 40 prescriptions were identified as having a dosing error with significant potential for patient harm, as the dose exceeded the maximum daily dose approved by the Summary of Product Characteristics [13]. Data indicates that tailoring of CDS to consider patient-specific factors reduces the prescription of excessive doses of medications [14].

Patients in the ICU tend to be especially ill and have an especially high risk of ADEs. Dosing of medications in the ICU is complex, and we evaluated the clinical significance of high dose-range CDS alerts, which identify doses prescribed over the maximum that are likely to cause patient harm. To do this, we performed a prospective study of the frequency and appropriateness of high dose-range alerts in the ICU and evaluated the clinical consequences of those which were overridden. By assessing how often these alerts appear, how providers respond to them, and identifying the extent of harm associated with these high dose-range CDS alert overrides, we hope to identify ways to improve the current CDS alert systems.

2 Methods

We performed a prospective, observational study of high dose-range medication-related CDS alert overrides by all types of providers (anesthesiologist, fellow, nurse practitioner, physician, physician assistant, resident). Although these alerts are presented to other clinicians (e.g., pharmacists, nurses), this study only focuses on prescribers. Alert overrides were generated between September 2016 and April 2017 for patients admitted to one of the following adult ICUs at Brigham and Women’s Hospital: two medical, two neurology, and two surgical. Targeted alerts were triggered by the prescription of doses greater than or equal to 5% over the maximum daily dose recommended by our medication knowledge base (First DataBank, South San Francisco, CA, USA) and any high dose of high-risk medications, as identified by the Institute for Safe Medication Practices [16]. This knowledge base was not tailored by our institution and is not ICU-specific. Dose-range alerts were dependent on the daily dose, which accounted for factors such as renal function and weight, and included single-dose and infusion dose alerts if the maximum daily dose was above the recommended daily threshold. This study was approved by the Partners HealthCare Institutional Review Board.

2.1 Appropriateness Evaluation

Criteria for appropriateness of overrides were created using a similar approach to that described in our previously published data; part of this process drew on both guidelines and clinical experience of a multidisciplinary group [9, 17]. Criteria were specific for alert categories and modified until a consensus was reached for the criteria specific to each alert. Appropriateness was independently evaluated for all overridden alerts by two reviewers, one clinical pharmacist (AW) and one research assistant (CR). The inter-rater agreement for appropriateness was determined with a κ statistic. Disagreements were resolved by discussions between the two independent reviewers, with a third experienced reviewer consulted when necessary. The κ for the criteria agreement of appropriateness was 0.87 [95% confidence interval (CI) 0.85–0.90], with an actual agreement of 88.0%.

2.2 ADE Evaluation

We performed patient chart reviews for all patients whose high dose-range alerts were overridden to see if this resulted in them experiencing an ADE. We considered the time period from which the alert was overridden to the point at which the order was stopped. ADEs evaluated in this study were specific to the overridden alert (e.g., digoxin alert → symptomatic bradycardia). Data relevant to an ADE, such as laboratory reports, medication orders and patient notes documented by nurses or providers, were abstracted and summarized by one reviewer. These data were then forwarded to two independent reviewers to determine the likelihood (no ADE, potential ADE, definite ADE) and severity of the ADE (significant, serious, life-threatening). These two reviewers were blinded as to whether the alert was appropriately overridden or not by the clinician. If consensus was not achieved, a third experienced reviewer was consulted. The κ for ADE determination was 0.83 (95% CI 0.80–0.86), indicating almost perfect agreement, with an actual agreement of 85%. Study personnel had undergone training based on guidance developed by the Center for Excellence for Patient Safety Research at Brigham and Women’s Hospital, which has been used in previous studies and has been previously described [17].

2.3 Statistical Analysis

Descriptive statistics were used to summarize patient characteristics. Fisher’s exact test was used to compare the difference in rate of ADEs by override appropriateness. A p value of < 0.05 was considered significant. Statistical analysis was completed using R V.3.3.3 (R Core Team, Vienna, Austria).

3 Results

The override rate of high dose-range alerts was 93.0% (total n = 1525) during the study period. We identified a total of 1418 dose-range overridden alerts from 755 patients admitted to the ICUs during the study period. Patient demographics are detailed in Table 1. Patients were generally middle-aged and emergently admitted to the hospital (i.e., non-scheduled admissions) as a medical admission.

Table 1 Patient demographics (n = 755)
Full size table

3.1 Medications Associated with Overrides

A total of 139 medications resulted in the 1418 alerts. The five most overridden medications were insulin regular (n = 262, 18.5%), glycopyrrolate (n = 79, 5.6%), lorazepam (n = 70, 4.9%), heparin (n = 60, 4.2%), and calcium gluconate (n = 56, 3.9%). In grouping by medication class, insulin was associated with the most overrides (n = 262, 18.5%), followed by antibiotics (n = 175, 12.3%), and benzodiazepines (n = 138, 9.7%). There were 429 (30.3%) overrides that were associated with a continuous infusion of a medication. There were 15 overrides (1.1%) which were apparent computer entry errors (e.g., selecting dose by weight instead of straight dose, extra zero at end of dose) (n = 8: vancomycin; n = 2: phenytoin; n = 1: acetaminophen, methylprednisolone, morphine, pentobarbital, sodium bicarbonate oral). Of all overrides, 325 (22.9%) were associated with an override reason. “Will monitor” was the most frequently used override reason (n = 177, 54.5%), followed by “Benefit outweighs risk” (n = 101, 31.1%), and “Patient tolerated before” (n = 20, 6.2%). There were three overrides that had an override reason of “Inaccurate warning” (n = 2: sodium bicarbonate bolus; n = 1: cefazolin).

3.2 Appropriateness of Overrides

The appropriateness rate of overrides was 88.8% (appropriate: n = 1259; inappropriate: n = 159). The medications associated with more than 1% of alert overrides and their rates of appropriateness are detailed in Table 2.

Table 2 Medications accounting for > 1% of total sample and rate of appropriateness
Full size table

Of the continuous infusion overrides, the vast majority were appropriately overridden (n = 426, 99.3%). All of the “Inaccurate warning” overrides were appropriate overrides. There were 89 medications (64.0%) that were only associated with an appropriate override. These included the commonly used medications in the ICU of fentanyl (n = 39), haloperidol (n = 31), morphine (n = 15), and epinephrine infusion (n = 12). There were a total of 17 medications that were only associated with an inappropriate override. The highest number of these overrides was two each for atovaquone, ceftazidime, ciprofloxacin, and olanzapine.

3.3 Evaluation of ADEs

A total of 791 appropriately overridden (62.9%) and 20 inappropriately overridden alerts (12.6%) resulted in medication administration to the patient. The rate of ADEs for the appropriately and inappropriately overridden alerts per 100 overridden alerts was 1.3 and 5.0, respectively (p < 0.001). Appropriate overrides resulted in exclusively non-preventable ADEs. Details of all the ADEs (n = 11) found in this study are discussed in Table 3, including the medication and dose ordered, appropriateness of the override and the type and severity of the ADE. There was no patient mortality related to the ADEs identified in this study.

Table 3 Details of ADEs found in study
Full size table

4 Discussion

We evaluated the appropriateness of high dose-range CDS overrides and the ADEs associated with these overrides in the ICU. We found that a high proportion of overrides were overridden and most of these overrides were appropriate. However, inappropriate overrides were associated with a fourfold increased risk of ADEs, compared to appropriately overridden alerts. These data suggest that to improve safety, we need to both turn off more of the unnecessary warnings and also underscore the small number of truly important ones.

Much of the published data on high dose-range CDS alerts are focused on the pediatric patient population, because of the complex parameters that need to be factored in routinely, especially age and weight [18,19,20,21,22]. Recommendations from this population include the combination of dose-rounding guidelines, in addition to incorporation of cognitive processes (i.e., human factors) into the creation of CDS alerts. Studies in adult patient populations follow the same recommendations, although with much more lenient dose recommendations [12,13,14]. One study created an algorithm for dose-range checking, which was then tested in their institution’s electronic health record (EHR) and led to subsequent refinement of their algorithm [12]. They found 11 factors that could be incorporated into CDS alerts to improve their performance, such as differentiation by dosage form (e.g., oral vs. intravenous), incorporation of therapy indication, and incorporation of age into dosing. A main recommendation that is consistent is to evaluate the dose-range CDS alerts that exist within an institution.

Some of the medications associated with overrides were surprising because of how commonly these medications are used in the ICU. A small proportion of the alerts (1.1%) were apparently obvious deviations from standard dosing, likely occurring because of a computer input key error. One example is for the ordering of vancomycin at our institution, which can be dosed by the provider for the final dose, or calculated by the computer by providing a dose by patient weight. This differs by how the vancomycin is ordered (i.e., stand-alone or via order set), leading to potential confusion dependent on what the provider might be used to. Standardization could help to prevent these errors from occurring. Many overrides were due to continuous infusions that actually followed our institutional guidelines, which illustrates one area for improvement in the decision support. The appropriateness rate of overrides is consistent with published literature [8, 9, 23]. The most significant finding in our appropriateness evaluation was that 89 medications (64.0%) and their associated CDS alerts were exclusively appropriately overridden. Primary efforts to improve CDS alerts should target these medications to improve the clinical significance of high dose-range CDS alerts.

In the ADE evaluation, we found a significant increase in ADEs as a consequence of inappropriately overridden alerts, compared to appropriately overridden alerts. An interesting finding was the difference in the rate of medication administration by the appropriateness of the override, with appropriately overridden alerts approximately five times more likely to be administered to the patient. This may be due to identifying dose errors after initial prescription of the medication (e.g., pharmacist verification, nursing administration, subsequent provider discontinuation). These potential interventions may have contributed to the small number of ADEs (n = 1) that we found in the inappropriately overridden alerts. Additionally, there were five ADEs that were a result of continuation of a patient’s home medication. Standard of care at our institution is to determine a patient’s home medication list within days of a patient’s admission; however, whether it is truly accurate or not is unknown as often ICU patients may not be able to verbally provide correct information. Adverse events may have occurred due to the alterations in organ function related to critical illness [10, 11]. These events are concerning as they are difficult to account for with dose-range alerts, as long as the patient was not admitted for a reason related to their home medications.

4.1 Recommendations to Improve Clinical Decision Support

Based on the findings of our study, one method to improve our institution’s high dose-range CDS alerts would be to remove the alerts that coincide with our institutional dose guidelines, namely for continuous infusion medications. As shown by our results, all but three continuous infusion alerts were overridden appropriately according to institutional guidelines. Removal of these alerts, which accounted for approximately one-third of alerts during our study period, would significantly reduce the number of alerts that providers may encounter.

An additional method that has been shown to be successful is the use of a statistical analysis approach to define sensible dosing limits [24]. Investigators identified the 100 most commonly prescribed drugs at their institution and used two physicians to create theoretical “warning” (highest single dose that would be reasonably prescribed) and “disallow” (maximum dose allowable in any circumstance) limits. These limits were then compared to the prescriptions within their institution over a 1-year period. They found an area under the curve for their predictive model of 99.4% and 99.7% for the “warning” and “disallow” limits, respectively.

Continually monitoring alerts that have been identified by providers as clinically irrelevant (e.g., “Inaccurate warning”) is another potential method to improve CDS [25]. We found that the three alerts that were overridden with this reason were appropriately overridden. Acknowledging these overrides and educating providers that this is a way to tailor an institution’s CDS alerts may help to increase buy-in of this technology to improve patient care.

Finally, incorporation of additional patient factors will help to increase the positive predictive value of these alerts [13, 14, 26]. One approach includes the adaptation of a pharmacokinetic approach to dosing for patients with renal impairment, due to the flaws associated with current systems [14]. In addition, this study incorporated patient information (e.g., patient age, other ordered medications) from the EHR into dose recommendations. This improved algorithm resulted in a 20% decrease in excessive ordered doses compared to the baseline CDS (p < 0.001). Other potential patient factors that can be included into these alerts are the degree of monitoring the patient has, the ordered monitoring that is available, and a patient’s past exposure to drugs and the associated doses with each episode.

In summary, we believe that alerts in general should account for as many possible patient and institution characteristics as possible. This would include factoring in patient labs (e.g., electrolytes), renal function/replacement therapies, and institutional guidelines. Therefore, the “out-of-the-box” approach for commercial EHRs should be carefully evaluated at each institution to ensure that alerts are not overwhelming to front-line clinicians.

4.2 Limitations

Our study had several limitations. First, this study is based on only one commercial EHR/database and therefore, may not necessarily be applicable to other institutions. This included accounting for factors such as renal function, which may not be included at other institutions. Second, we may not have factored in all possible aspects of why a provider may have overrode an alert. However, we believe that we made considerable efforts to include as many different aspects in determining appropriateness criteria, including the formulation of criteria using a multidisciplinary expert team and the independent adjudicators. Finally, we were dependent on clinical documentation for the determination of ADEs, which might have limited our findings. The small number of inappropriate overrides makes it difficult to compare the rate of ADEs between the appropriately and inappropriately overridden alerts. However, our results are consistent with research our study team has previously completed.

5 Conclusion

Approximately nine out of ten identified dose-range CDS overrides were appropriately overridden. Inappropriate overrides were four times as likely to be associated with an ADE compared to appropriate overrides, confirming that decision support can identify clinically important situations and suggesting that better approaches are needed to make it clear to clinicians which decision support is particularly important. Further efforts should be targeted at improving the positive predictive value of CDS in a number of ways, including the incorporation of more patient-specific factors.