Medication-related clinical decision support (CDS) alerts have been shown to be effective at reducing adverse drug events (ADEs). However, these alerts are frequently overridden, with limited data linking these overrides to harm. Dose-range checking alerts are a type of CDS alert that could have a significant impact on morbidity and mortality, especially in the intensive care unit (ICU) setting.
We performed a single-center, prospective, observational study of adult ICUs from September 2016 to April 2017. Targeted overridden alerts were triggered when doses greater than or equal to 5% over the maximum dose were prescribed. The primary outcome was the appropriateness of the override, determined by two independent reviewers, using pre-specified criteria formulated by a multidisciplinary group. Overrides which resulted in medication administration were then evaluated for ADEs by chart review.
The override rate of high dose-range alerts in the ICU was 93.0% (total n = 1525) during the study period. A total of 1418 overridden alerts from 755 unique patients were evaluated for appropriateness (appropriateness rate 88.8%). The most common medication associated with high dose-range alerts was insulin regular infusion (n = 262, 18.5%). The rates of ADEs for the appropriately and inappropriately overridden alerts per 100 overridden alerts were 1.3 and 5.0, respectively (p < 0.001).
Overriding high dose-range CDS alerts was found to be common and often appropriate, suggesting that more intelligent dose checking is needed. Some alerts were clearly inappropriately presented to the provider. Inappropriate overrides were associated with an increased risk of ADEs, compared to appropriately overridden alerts.
|Overrides of high dose-range alerts are common and often appropriately done.|
|Harm from these overrides was rare, although inappropriately overridden alerts were more associated with harm.|
|Increasing the positive predictive value of these alerts may reduce provider alert fatigue.|
Adverse drug events (ADEs) have been associated with an increase in patient length of hospital stay, and patient morbidity and mortality [1,2,3]. Medication-related clinical decision support (CDS) is a potential way to reduce the occurrence of ADEs [4, 5]. Despite these studied benefits, CDS alerts are overridden frequently, sometimes inappropriately [6,7,8,9]. Override rates are variable across institutions and dependent on the type of alert evaluated.
A patient population that is particularly susceptible to ADEs includes those in the intensive care unit (ICU), due to an increased number of high-risk medications, altered pharmacokinetics, and susceptible organ systems (e.g., organ failure) [10, 11]. Research regarding CDS alert overrides in the ICU is limited, but suggests that up to 90% of these alerts are overridden, often appropriately [8, 9]. However, when CDS alerts are overridden inappropriately, there is up to a sixfold increase in ADEs compared to appropriately overridden alerts . However, studies evaluating the risk of inappropriate overrides focused on common types of CDS alerts (allergy, drug–drug interaction, geriatric, renal), but had limited characterization of dose-range alerts, which detect potential underdosing and overdosing of prescribed medications.
More broadly, few data are available regarding the impact of dosing CDS [12,13,14,15]. In one study evaluating medication prescriptions, one in 40 prescriptions were identified as having a dosing error with significant potential for patient harm, as the dose exceeded the maximum daily dose approved by the Summary of Product Characteristics . Data indicates that tailoring of CDS to consider patient-specific factors reduces the prescription of excessive doses of medications .
Patients in the ICU tend to be especially ill and have an especially high risk of ADEs. Dosing of medications in the ICU is complex, and we evaluated the clinical significance of high dose-range CDS alerts, which identify doses prescribed over the maximum that are likely to cause patient harm. To do this, we performed a prospective study of the frequency and appropriateness of high dose-range alerts in the ICU and evaluated the clinical consequences of those which were overridden. By assessing how often these alerts appear, how providers respond to them, and identifying the extent of harm associated with these high dose-range CDS alert overrides, we hope to identify ways to improve the current CDS alert systems.
We performed a prospective, observational study of high dose-range medication-related CDS alert overrides by all types of providers (anesthesiologist, fellow, nurse practitioner, physician, physician assistant, resident). Although these alerts are presented to other clinicians (e.g., pharmacists, nurses), this study only focuses on prescribers. Alert overrides were generated between September 2016 and April 2017 for patients admitted to one of the following adult ICUs at Brigham and Women’s Hospital: two medical, two neurology, and two surgical. Targeted alerts were triggered by the prescription of doses greater than or equal to 5% over the maximum daily dose recommended by our medication knowledge base (First DataBank, South San Francisco, CA, USA) and any high dose of high-risk medications, as identified by the Institute for Safe Medication Practices . This knowledge base was not tailored by our institution and is not ICU-specific. Dose-range alerts were dependent on the daily dose, which accounted for factors such as renal function and weight, and included single-dose and infusion dose alerts if the maximum daily dose was above the recommended daily threshold. This study was approved by the Partners HealthCare Institutional Review Board.
Criteria for appropriateness of overrides were created using a similar approach to that described in our previously published data; part of this process drew on both guidelines and clinical experience of a multidisciplinary group [9, 17]. Criteria were specific for alert categories and modified until a consensus was reached for the criteria specific to each alert. Appropriateness was independently evaluated for all overridden alerts by two reviewers, one clinical pharmacist (AW) and one research assistant (CR). The inter-rater agreement for appropriateness was determined with a κ statistic. Disagreements were resolved by discussions between the two independent reviewers, with a third experienced reviewer consulted when necessary. The κ for the criteria agreement of appropriateness was 0.87 [95% confidence interval (CI) 0.85–0.90], with an actual agreement of 88.0%.
We performed patient chart reviews for all patients whose high dose-range alerts were overridden to see if this resulted in them experiencing an ADE. We considered the time period from which the alert was overridden to the point at which the order was stopped. ADEs evaluated in this study were specific to the overridden alert (e.g., digoxin alert → symptomatic bradycardia). Data relevant to an ADE, such as laboratory reports, medication orders and patient notes documented by nurses or providers, were abstracted and summarized by one reviewer. These data were then forwarded to two independent reviewers to determine the likelihood (no ADE, potential ADE, definite ADE) and severity of the ADE (significant, serious, life-threatening). These two reviewers were blinded as to whether the alert was appropriately overridden or not by the clinician. If consensus was not achieved, a third experienced reviewer was consulted. The κ for ADE determination was 0.83 (95% CI 0.80–0.86), indicating almost perfect agreement, with an actual agreement of 85%. Study personnel had undergone training based on guidance developed by the Center for Excellence for Patient Safety Research at Brigham and Women’s Hospital, which has been used in previous studies and has been previously described .
Descriptive statistics were used to summarize patient characteristics. Fisher’s exact test was used to compare the difference in rate of ADEs by override appropriateness. A p value of < 0.05 was considered significant. Statistical analysis was completed using R V.3.3.3 (R Core Team, Vienna, Austria).
The override rate of high dose-range alerts was 93.0% (total n = 1525) during the study period. We identified a total of 1418 dose-range overridden alerts from 755 patients admitted to the ICUs during the study period. Patient demographics are detailed in Table 1. Patients were generally middle-aged and emergently admitted to the hospital (i.e., non-scheduled admissions) as a medical admission.
Medications Associated with Overrides
A total of 139 medications resulted in the 1418 alerts. The five most overridden medications were insulin regular (n = 262, 18.5%), glycopyrrolate (n = 79, 5.6%), lorazepam (n = 70, 4.9%), heparin (n = 60, 4.2%), and calcium gluconate (n = 56, 3.9%). In grouping by medication class, insulin was associated with the most overrides (n = 262, 18.5%), followed by antibiotics (n = 175, 12.3%), and benzodiazepines (n = 138, 9.7%). There were 429 (30.3%) overrides that were associated with a continuous infusion of a medication. There were 15 overrides (1.1%) which were apparent computer entry errors (e.g., selecting dose by weight instead of straight dose, extra zero at end of dose) (n = 8: vancomycin; n = 2: phenytoin; n = 1: acetaminophen, methylprednisolone, morphine, pentobarbital, sodium bicarbonate oral). Of all overrides, 325 (22.9%) were associated with an override reason. “Will monitor” was the most frequently used override reason (n = 177, 54.5%), followed by “Benefit outweighs risk” (n = 101, 31.1%), and “Patient tolerated before” (n = 20, 6.2%). There were three overrides that had an override reason of “Inaccurate warning” (n = 2: sodium bicarbonate bolus; n = 1: cefazolin).
Appropriateness of Overrides
The appropriateness rate of overrides was 88.8% (appropriate: n = 1259; inappropriate: n = 159). The medications associated with more than 1% of alert overrides and their rates of appropriateness are detailed in Table 2.
Of the continuous infusion overrides, the vast majority were appropriately overridden (n = 426, 99.3%). All of the “Inaccurate warning” overrides were appropriate overrides. There were 89 medications (64.0%) that were only associated with an appropriate override. These included the commonly used medications in the ICU of fentanyl (n = 39), haloperidol (n = 31), morphine (n = 15), and epinephrine infusion (n = 12). There were a total of 17 medications that were only associated with an inappropriate override. The highest number of these overrides was two each for atovaquone, ceftazidime, ciprofloxacin, and olanzapine.
Evaluation of ADEs
A total of 791 appropriately overridden (62.9%) and 20 inappropriately overridden alerts (12.6%) resulted in medication administration to the patient. The rate of ADEs for the appropriately and inappropriately overridden alerts per 100 overridden alerts was 1.3 and 5.0, respectively (p < 0.001). Appropriate overrides resulted in exclusively non-preventable ADEs. Details of all the ADEs (n = 11) found in this study are discussed in Table 3, including the medication and dose ordered, appropriateness of the override and the type and severity of the ADE. There was no patient mortality related to the ADEs identified in this study.
We evaluated the appropriateness of high dose-range CDS overrides and the ADEs associated with these overrides in the ICU. We found that a high proportion of overrides were overridden and most of these overrides were appropriate. However, inappropriate overrides were associated with a fourfold increased risk of ADEs, compared to appropriately overridden alerts. These data suggest that to improve safety, we need to both turn off more of the unnecessary warnings and also underscore the small number of truly important ones.
Much of the published data on high dose-range CDS alerts are focused on the pediatric patient population, because of the complex parameters that need to be factored in routinely, especially age and weight [18,19,20,21,22]. Recommendations from this population include the combination of dose-rounding guidelines, in addition to incorporation of cognitive processes (i.e., human factors) into the creation of CDS alerts. Studies in adult patient populations follow the same recommendations, although with much more lenient dose recommendations [12,13,14]. One study created an algorithm for dose-range checking, which was then tested in their institution’s electronic health record (EHR) and led to subsequent refinement of their algorithm . They found 11 factors that could be incorporated into CDS alerts to improve their performance, such as differentiation by dosage form (e.g., oral vs. intravenous), incorporation of therapy indication, and incorporation of age into dosing. A main recommendation that is consistent is to evaluate the dose-range CDS alerts that exist within an institution.
Some of the medications associated with overrides were surprising because of how commonly these medications are used in the ICU. A small proportion of the alerts (1.1%) were apparently obvious deviations from standard dosing, likely occurring because of a computer input key error. One example is for the ordering of vancomycin at our institution, which can be dosed by the provider for the final dose, or calculated by the computer by providing a dose by patient weight. This differs by how the vancomycin is ordered (i.e., stand-alone or via order set), leading to potential confusion dependent on what the provider might be used to. Standardization could help to prevent these errors from occurring. Many overrides were due to continuous infusions that actually followed our institutional guidelines, which illustrates one area for improvement in the decision support. The appropriateness rate of overrides is consistent with published literature [8, 9, 23]. The most significant finding in our appropriateness evaluation was that 89 medications (64.0%) and their associated CDS alerts were exclusively appropriately overridden. Primary efforts to improve CDS alerts should target these medications to improve the clinical significance of high dose-range CDS alerts.
In the ADE evaluation, we found a significant increase in ADEs as a consequence of inappropriately overridden alerts, compared to appropriately overridden alerts. An interesting finding was the difference in the rate of medication administration by the appropriateness of the override, with appropriately overridden alerts approximately five times more likely to be administered to the patient. This may be due to identifying dose errors after initial prescription of the medication (e.g., pharmacist verification, nursing administration, subsequent provider discontinuation). These potential interventions may have contributed to the small number of ADEs (n = 1) that we found in the inappropriately overridden alerts. Additionally, there were five ADEs that were a result of continuation of a patient’s home medication. Standard of care at our institution is to determine a patient’s home medication list within days of a patient’s admission; however, whether it is truly accurate or not is unknown as often ICU patients may not be able to verbally provide correct information. Adverse events may have occurred due to the alterations in organ function related to critical illness [10, 11]. These events are concerning as they are difficult to account for with dose-range alerts, as long as the patient was not admitted for a reason related to their home medications.
Recommendations to Improve Clinical Decision Support
Based on the findings of our study, one method to improve our institution’s high dose-range CDS alerts would be to remove the alerts that coincide with our institutional dose guidelines, namely for continuous infusion medications. As shown by our results, all but three continuous infusion alerts were overridden appropriately according to institutional guidelines. Removal of these alerts, which accounted for approximately one-third of alerts during our study period, would significantly reduce the number of alerts that providers may encounter.
An additional method that has been shown to be successful is the use of a statistical analysis approach to define sensible dosing limits . Investigators identified the 100 most commonly prescribed drugs at their institution and used two physicians to create theoretical “warning” (highest single dose that would be reasonably prescribed) and “disallow” (maximum dose allowable in any circumstance) limits. These limits were then compared to the prescriptions within their institution over a 1-year period. They found an area under the curve for their predictive model of 99.4% and 99.7% for the “warning” and “disallow” limits, respectively.
Continually monitoring alerts that have been identified by providers as clinically irrelevant (e.g., “Inaccurate warning”) is another potential method to improve CDS . We found that the three alerts that were overridden with this reason were appropriately overridden. Acknowledging these overrides and educating providers that this is a way to tailor an institution’s CDS alerts may help to increase buy-in of this technology to improve patient care.
Finally, incorporation of additional patient factors will help to increase the positive predictive value of these alerts [13, 14, 26]. One approach includes the adaptation of a pharmacokinetic approach to dosing for patients with renal impairment, due to the flaws associated with current systems . In addition, this study incorporated patient information (e.g., patient age, other ordered medications) from the EHR into dose recommendations. This improved algorithm resulted in a 20% decrease in excessive ordered doses compared to the baseline CDS (p < 0.001). Other potential patient factors that can be included into these alerts are the degree of monitoring the patient has, the ordered monitoring that is available, and a patient’s past exposure to drugs and the associated doses with each episode.
In summary, we believe that alerts in general should account for as many possible patient and institution characteristics as possible. This would include factoring in patient labs (e.g., electrolytes), renal function/replacement therapies, and institutional guidelines. Therefore, the “out-of-the-box” approach for commercial EHRs should be carefully evaluated at each institution to ensure that alerts are not overwhelming to front-line clinicians.
Our study had several limitations. First, this study is based on only one commercial EHR/database and therefore, may not necessarily be applicable to other institutions. This included accounting for factors such as renal function, which may not be included at other institutions. Second, we may not have factored in all possible aspects of why a provider may have overrode an alert. However, we believe that we made considerable efforts to include as many different aspects in determining appropriateness criteria, including the formulation of criteria using a multidisciplinary expert team and the independent adjudicators. Finally, we were dependent on clinical documentation for the determination of ADEs, which might have limited our findings. The small number of inappropriate overrides makes it difficult to compare the rate of ADEs between the appropriately and inappropriately overridden alerts. However, our results are consistent with research our study team has previously completed.
Approximately nine out of ten identified dose-range CDS overrides were appropriately overridden. Inappropriate overrides were four times as likely to be associated with an ADE compared to appropriate overrides, confirming that decision support can identify clinically important situations and suggesting that better approaches are needed to make it clear to clinicians which decision support is particularly important. Further efforts should be targeted at improving the positive predictive value of CDS in a number of ways, including the incorporation of more patient-specific factors.
Leape LL, Brennan TA, Laird N, Lawthers AG, Locallo AR, Barnes BA, et al. The nature of adverse events in hospitalized patients. Results of the Harvard Medical Practice Study II. N Engl J Med. 1991;324:377–84.
Classen DC, Pestotnik Evans RS, Lloyd JF, Burke JP. Adverse drug events in hospitalized patients. Excess length of stay, extra costs, and attributable mortality. JAMA. 1997;277:301–6.
Bates DW, Spell N, Cullen DJ, Burdick E, Laird N, Petersen LA, et al. The costs of adverse drug events in hospitalized patients. JAMA. 1997;277:307–11.
Bates DW, Leape LL, Cullen DJ, Laird N, Petersen LA, Teich JM, et al. Effect of computerized physician order entry and a team intervention on prevention of serious medication errors. JAMA. 1998;280:1311–6.
Bates DW, Teich JM, Lee J, Seger D, Kuperman GJ, Ma’Luf N, et al. The impact of computerized physician order entry on medication error prevention. J Am Med Inform Assoc. 1999;6:313–21.
Lin CP, Payne TH, Nichol P, Hoey PJ, Anderson CL, Gennari JH. Evaluating clinical decision support systems: monitoring CPOE order check override rates in the Department of Veterans Affairs’ Computerized Patient Record System. J Am Med Inform Assoc. 2008;15:620–6.
Nanji KC, Slight SP, Seger DL, Cho I, Fiskio JM, Redden LM, et al. Overrides of medication-related clinical decision support alerts in outpatients. J Am Med Inform Assoc. 2014;21:487–91.
Wong A, Amato MG, Seger DL, Slight SP, Beeler PE, Dykes PC, et al. Evaluation of medication-related clinical decision support overrides in the intensive care unit. J Crit Care. 2017;39:156–61.
Wong A, Amato MG, Seger DL, Rehr C, Wright A, Slight SP, et al. Prospective evaluation of medication-related clinical decision support overrides in the intensive care unit. BMJ Qual Saf. 2018;27:718–24.
Cullen DJ, Sweitzer BJ, Bates DW, Burdick E, Edmondson A, Leape LL. Preventable adverse drug events in hospitalized patients: a comparative study of intensive care and general care units. Crit Care Med. 1997;25:1289–97.
Rothschild JM, Landrigan CP, Cronin JW, Kaushal R, Lockley SW, Burdick E, et al. The Critical Care Safety Study: the incidence and nature of adverse events and serious medical errors in intensive care. Crit Care Med. 2005;33:1694–700.
Coleman JJ, Nwulu U, Ferner RE. Decision support for sensible dosing in electronic prescribing systems. J Clin Pharm Ther. 2012;37:415–9.
Seidling HM, Barmawi AA, Kaltschmidt J, Bertsche T, Pruszcdlo MG, Haefeli WE. Detection and prevention of prescriptions with excessive doses in electronic prescribing systems. Eur J Clin Pharmacol. 2007;63:1185–92.
Seidling HM, Schmitt SP, Bruckner T, Kaltschmidt J, Pruszydlo MG, Senger C, et al. Patient-specific electronic clinical decision support reduces prescription of excessive doses. Qual Saf Health Care. 2010;19:e15.
Winterstein AG, Johns TE, Rosenberg EI, Hatton RC, Gonzalez-Rothi R, Kanjanarat P. Nature and causes of clinically significant medication errors in a tertiary care hospital. Am J Health Syst Pharm. 2004;61:1908–16.
ISMP list of high-alert medications in acute care settings. Institute for Safe Medication Practices website. https://www.ismp.org/tools/institutionalhighAlert.asp. Updated 2014. Accessed 20 July 2018.
Morimoto T, Gandhi TK, Seger AC, Hsieh TC, Bates DW. Adverse drug events and medication errors: detection and classification methods. Qual Saf Health Care. 2004;13:306–14.
Lesar TS, Briceland LL, Delcoure K, Parmalee JC, Masta-Gornic V, Pohl H. Medication prescribing errors in a teaching hospital. JAMA. 1990;263:2329–34.
Kaushal R, Barker KN, Bates DW. How can information technology improve patient safety and reduce medication errors in children’s health care? Arch Pediatr Adolesc Med. 2001;155:1002–7.
Fortescue EB, Kaushal R, Landrigan CP, McKenna KJ, Clapp MD, Federico F, et al. Prioritizing strategies for preventing medication errors and adverse drug events in pediatric inpatients. Pediatrics. 2003;111:722–9.
Potts AL, Barr FE, Gregory DF, Wright L, Patel NR. Computerized physician order entry and medication errors in a pediatric critical care unit. Pediatrics. 2004;113:59–63.
Johnson KB, Lee CKK, Spooner A, Davison CL, Helmke JS, Weinberg ST. Automated dose-rounding recommendations for pediatric medications. Pediatrics. 2011;128:e4220–8.
Nanji KC, Seger DL, Slight SP, Amato MG, Beeler PE, Her QL, et al. Medication-related clinical decision support alert overrides in inpatients. J Am Med Inform Assoc. 2018;25:476–81.
Coleman JJ, Hodson J, Ferner RE. Deriving dose limits for warnings in electronic prescribing systems: statistical analysis of prescription data at University Hospital Birmingham. UK. Drug Saf. 2013;35:291–8.
Rehr CA, Wong A, Seger DL, Bates DW. Determining inappropriate medication alerts from “inaccurate warning” overrides in the intensive care unit. Appl Clin Inform. 2018;9:268–74.
Eschmann E, Beeler PE, Schneemann M, Blaser J. Developing strategies for predicting hyperkalemia in potassium-increasing drug-drug interactions. J Am Med Inform Assoc. 2017;24:60–6.
This study was supported by a grant from the American College of Clinical Pharmacy Critical Care Practice and Research Network.
Conflict of interest
Dr. David Bates reports receiving equity from Intensix, which makes software to support clinical decision making in intensive care; being named as co-inventor on Patent Number 6029138 held by Brigham and Women’s Hospital on the use of decision support software for medical management, licensed to Medicalis, and holding a minority equity position in Medicalis, which develops web-based decision support for radiology test ordering; consulting for Early Sense, which makes patient safety monitoring systems; receiving equity and cash compensation from QPID, a company focused on intelligence systems for electronic health records; receiving cash compensation from CDI (Negev), which is a not-for profit incubator for health IT start-ups; and receiving equity from Enelgy, which makes software to support evidence-based clinical decisions, from Ethosmart, which makes software to help patients with chronic diseases, and from MDClone, which takes clinical data and produces de-identified versions of it. Dr. Patrick Beeler was supported by the Swiss National Science Foundation. Dr. Adrian Wong, Ms. Christine Rehr, Ms. Diane Seger, Dr. Mary Amato, Dr. Sarah Slight and Dr. Adam Wright have no conflicts of interest to disclose.
This study was approved by the Partners HealthCare Institutional Review Board.
About this article
Cite this article
Wong, A., Rehr, C., Seger, D.L. et al. Evaluation of Harm Associated with High Dose-Range Clinical Decision Support Overrides in the Intensive Care Unit. Drug Saf 42, 573–579 (2019). https://doi.org/10.1007/s40264-018-0756-x