Medication organizations across the USA have adopted electronic health records, and one of the most anticipated benefits of these was improved medication safety, but alert fatigue has been a major issue.
We compared the appropriateness of medication-related clinical decision support alerts triggered by two commercial applications: EPIC and Seegnal’s platform.
This was a retrospective comparison of two commercial applications. We provided Seegnal with deidentified inpatient, outpatient, and inpatient genetic electronic medical record (EMR)-extracted datasets for 657, 2731, and 413 patients, respectively. Seegnal then provided the alerts that would have triggered, which we compared with those triggered by EPIC in clinical care. A random sample of the alerts triggered were reviewed for appropriateness, and the positive predictive value (PPV) and negative predictive value (NPV) were calculated. We also reviewed all the inpatient and outpatient charts for patients within our cohort who were receiving ten or more concomitant medications with alerts we found to be appropriate to assess whether any adverse events had occurred and whether Seegnal’s platform could have prevented them.
Results from EPIC and the Seegnal platform were compared based on alert load, PPV, NPV, and potential adverse events. Overall, compared with EPIC, the Seegnal platform triggered fewer alerts in the inpatient (1697 vs. 27,540), outpatient (2341 vs. 35,134), and inpatient genetic (1493 vs. 20,975) cohorts. The Seegnal platform had higher specificity in the inpatient (99 vs. 0.3%; p < 0.0001), outpatient (99 vs. 0.3%; p < 0.0001), and inpatient genetic (97.9 vs. 1.2%; p < 0.0001) groups and higher sensitivity in the inpatient (100 vs. 68.8%; p < 0.0001) and outpatient (88.6 vs.78.3%; p < 0.0001) groups but not in the inpatient genetic cohort (81 vs. 78.5%; p = 0.11). We identified 16 adverse events that occurred in the inpatient setting, 11 (69%) of which potentially could have been prevented with the Seegnal platform.
Overall, the Seegnal platform triggered 94% fewer alerts than EPIC in the inpatient setting and 93% fewer in the outpatient setting, with much higher sensitivity and specificity. This application could substantially reduce alert fatigue and improve medication safety at the same time.
|The Seegnal platform triggered fewer alerts in all settings.|
|Lower alert rates would reduce alert burden and alert fatigue and improve safety.|
Adverse drug events (ADEs) represent a substantial patient safety issue and occur in approximately 1.5 million inpatients every year in the USA . Some studies suggest ADEs account for up to 5–17% of hospital admissions [2,3,4,5]. Of the 1.5 million ADEs occurring in the USA annually, approximately 400,000 are considered preventable. Medical errors are among the top ten causes of death in the USA, cause 6–10% of hospital admissions, and result in costs estimated to be more than $US1000 per patient per year . ADEs are defined as untoward medical occurrences associated with the use of drugs that may or may not have been preventable, whereas medical errors are defined as preventable adverse events. More than 20% of patients treated with multiple medications (polypharmacy) experience major drug-related problems such as adverse side effects, lack of efficacy, and unnecessary hospitalizations . One way to reduce the frequency of ADEs is to utilize clinical decision support (CDS) systems.
Overall, CDS systems have been shown to reduce medication errors by 81% in the inpatient setting, although these data came from internally developed applications with highly tuned decision support [6, 7]. Even though nearly all electronic health records (EHRs) at most hospitals include some form of CDS, nearly all this CDS is vendor developed and often does not address patient-specific factors. CDS represents an effective way to reduce errors and ADEs [8,9,10]. However, this impact may be decreased or even extinguished if too many clinically inappropriate alerts are given [11,12,13]. One study reported that approximately 60% of overrides of alerts were appropriate and that override rates varied based on type. For example, override rates were 98% for duplicate medication alerts, 96.5% for drug allergy alerts, 82.5% for nonformulary medication alerts, 26.4% for age-based medication substitution alerts, and just 2.2% for renal alerts . Many other studies have found even higher override rates [15, 16].
Alert fatigue introduces the risk of missing critical alerts that may compromise patient safety [17,18,19,20]. A study performed at our institution using a legacy home-grown EHR system found that inappropriately overridden CDS alerts were associated with an increased risk of ADEs . Nearly all major US healthcare organizations have now switched to commercial EHRs, with EPIC and Cerner being the market leaders, and organizations generally contract for medication decision support with a third-party company, with three companies controlling nearly all of the market: FDB, Medispan, and Multum (which is owned by Cerner). In such cases, the EHR imposes some logic on top of the linked database, and the combined capacities compose the CDS solution.
In this study, we calculated the positive predictive value (PPV), negative predictive value (NPV), sensitivity, and specificity, and overall alert load generated by the Seegnal platform compared with those generated by EPIC as implemented within our healthcare system.
Seegnal (Tel Aviv, Israel) has developed an application called the Seegnal platform. This platform is based on a multitier mechanism that relies on proprietary, real-time algorithm-derived alert personalization, Hansten & Horn’s knowledge-based clinical drug interaction evaluations, the University of Washington’s pharmacokinetic and pharmacogenetic databases, and FDB data. The Seegnal platform assesses a wide scope of relevant drug-related problem data including drug–drug interactions, drug allergy, drug dosing, drugs in pregnancy/lactation, drugs and smoking, drug–kidney/liver function interactions, pharmacokinetic drug interactions, drug–food/herbals/vitamin interactions, and pharmacogenetic interactions. The Seegnal platform simultaneously assesses multiple patient-specific parameters (e.g., drug dose/administration route, renal function, laboratory values, electrocardiogram, smoking), and medication-specific parameters (e.g., potency, dose–response curves, time to steady state, accumulative dosing, combined effects of multiple [more than two] medications) in real time to determine the need to trigger an alert or bypass the potential alert for the specific patient. The Seegnal platform also has an innovative user interface that enables the provider at the point of care to detect, prioritize, and choose an appropriate alternative. The Seegnal platform’s user interface was not assessed in this evaluation. To date, one study has been conducted using patient-specific parameters to alert on drug interactions compared with a customized vendor-based CDS database . No study has directly evaluated the alert load produced by the Seegnal platform compared with those of other vendor-based CDS.
Study Site and Data
This study was performed at Brigham and Women’s Hospital (BWH), Boston, MA, USA, a large urban academic medical center that uses a leading vendor EHR system, EPIC (Verona, WI, USA). BWH provided Seegnal with deidentified inpatient, outpatient, and inpatient genetic electronic medical record-extracted datasets of 657, 2731, and 413 patients, respectively. The dataset contained 56 predefined structured (coded) parameters that were potentially documented in the patients’ EHR during their hospitalization period and/or outpatient visits between 3 June 2018 and 9 June 2018. The dataset for inpatients with genetic data contained all data files during any inpatient admission between 1 January 2017 and 31 December 2019 as only a small portion of patients had genetic data available. The inpatient genetic records were identified by Partners Healthcare Biobank. The purpose of the Partners Biobank is to help researchers understand how health is affected by genes, lifestyle, and environment. Patients who consent to participate in the Partners Biobank contribute a DNA sample for genetic testing, including pharmacogenomics testing. During our data pull, we identified patients with mutations at cytochrome P450 (CYP)-3A5. Examples of parameters extracted were patients’ medication list with start and end dates, dose, frequency, route of administration, comments if any, active problem list, smoking status, pregnancy status, and various laboratory results such as potassium, creatinine, liver function tests, and so on. Specific patient modifiers/factors/data elements extracted from the EHR are listed in the electronic supplementary material.
Seegnal then ran the retrospective dataset against the Seegnal platform. The output from the Seegnal platform provided the alerts that would have been triggered in only the Seegnal platform, in both the Seegnal platform and EPIC, and only in EPIC as utilized within BWH. The alerts were then classified into predefined categories and reviewed for appropriateness to the specific patients. Two pharmacists independently reviewed a random sample of up to 100 alerts per each of the predefined categories (977 inpatient alerts, 808 inpatient genetic alerts, and 938 outpatient alerts, for a sum of 2723 alerts) but were blinded as to whether the alert originated from only the Seegnal platform, both the Seegnal platform and EPIC, or only EPIC.
The types of alerts reviewed included disease–drug contraindication, dosing, drug–drug interaction, duplicate therapy, geriatric, most probable side effects, renal dosing, pregnancy, pharmacokinetic drug interactions, and pharmacokinetic drug interactions with the patient parameter of smoking. The pharmacists (SS, DS) then compared their review; for any differences, two additional reviewers (DB, a doctor with extensive experience in patient safety, and JH, a pharmacist with extensive experience with drug–drug interactions) made the final decision as to whether or not the alert was appropriate.
Once the random sample alerts were evaluated, we then calculated the PPV (true positive/(true positive + false positive)), NPV (true negative/(true negative + false negative)), specificity (true negative/(true negative + false positive)), sensitivity (true positive/(true positive + false negative)), and overall interruptive alert load generated by the Seegnal platform compared with those of EPIC as it functions at BWH. True positives were alerts considered appropriate for the patient, false positives were alerts that were not appropriate for the patient, true negatives were alerts we agreed should not be triggered, and false negatives were alerts that should have triggered but did not. Examples of each are provided in Table 1. Analysis was performed in SAS v. 9.4 (SAS Institute, Cary, NC, USA), and significance was set using a two-sided p value of < 0.05.
We excluded evaluation of the appropriateness of the genetic alerts returned from the Seegnal platform from our final inpatient and genetic sample because we only had data for one pharmacogenetic marker (CYP3A5), so very few alerts fired. We evaluated all the other alerts for the inpatient genetic cohort. We also reviewed all the inpatient/inpatient genetic (n = 69) and outpatient (n = 65) charts for patients within our cohort who were receiving ten or more concomitant medications with alerts we found to be appropriate to assess whether any adverse events had occurred and whether the Seegnal platform could have prevented them.
The inpatient cohort consisted of 657 patients with 665 admissions; eight patients had two admissions. The inpatient genetic cohort consisted of 413 patients with 413 visits. The outpatient cohort consisted of 2731 patients with 2749 visits; 18 patients had two visits during the study period. Demographics of all patients can be found in Table 2.
Summary of Main Outcomes
In the inpatient setting, the Seegnal platform triggered 1697 alerts, which was 94% less than the EPIC platform, which triggered 27,540 alerts (Table 3). Sensitivity was 100% for the Seegnal platform and 68.8% for EPIC. In the inpatient genetic group (Table 3), the Seegnal platform triggered 1493 inpatient alerts and EPIC triggered 20,975. The sensitivities were similar: 81% for the Seegnal platform and 78.5% for EPIC. In the outpatient setting (Table 3), the Seegnal platform triggered 93% fewer alerts and sensitivity was 88.6%, whereas sensitivity was 78.3% for EPIC.
Inpatient and Outpatient Chart Review
We identified 16 adverse events that occurred in the inpatient setting. Of these, the Seegnal platform potentially would have prevented 11 (68.8%). Examples included patients who developed hypotension, bradycardia, QTc prolongation, increased combined side effects, and renal dysfunction. One adverse event involved a patient with a history of nausea and vomiting from oral potassium chloride, which might have been prevented if it had been documented in a structured format under allergies/intolerances instead of being included in the free-text chart notes. Overall, reviewers judged that four (25.0%) adverse events were nonpreventable, so no CDS could have prevented them, and one adverse event was possibly preventable. Of the 11 preventable adverse events that did occur, nine were judged serious and two were considered significant (Table 4). We found no adverse events in the outpatient setting.
We evaluated the alerting performance of the Seegnal platform compared with that of the EPIC combination and found that the Seegnal platform would have triggered nearly twentyfold fewer alerts, with higher sensitivity and higher specificity in both the inpatient and the outpatient setting. At the same time, the Seegnal platform potentially could have prevented a small number of adverse events associated with issues for which EPIC did not alert.
The high alert burden was consistent with other recent literature reporting very high override rates, mainly because of the inappropriate triggering of alerts [15, 19,20,21]. Not only did the Seegnal platform have a much lower alert load but it also had higher specificity, higher sensitivity, better PPV, and better NPV than EPIC in both the inpatient and the outpatient setting. It achieved this by utilizing patient-specific parameters that can reduce inappropriate alerting and diminish alert fatigue.
EHRs linked with CDS have been demonstrated to prevent many types of medication errors in an internally developed application . However, nearly all systems implemented in the USA are now commercial, and other studies suggest these are not yet as effective as some of the earlier implementations . The potential benefit of CDS is diminished by inappropriate alerting [11,12,13]. Studies have also shown that CDS can have unintended consequences. One study found that CDS contained incorrect or misleading CDS content or out-of-date content . Another study also found alert fatigue and user unfriendly systems with limited ability to customize databases or output to be major problems . The frequency of inappropriate alerts leads clinicians to ignore nearly all alerts, even those that can lead to patient harm. The Seegnal platform employs a CDS knowledge base that is patient specific and flexible and can be tailored to an institution’s specific requirements. This reduces the inappropriate alert load and alert fatigue. Overall, the Seegnal platform outperformed our current CDS system.
Other countries have taken a different approach for medication-related decision support. Notably, the Dutch use a database called G-Standaard, which is issued monthly by Z-Index, an organization owned by the Royal Dutch Association for the Advancement of Pharmacy . This database has been refined to reduce some of the false-positive alerts but is not used outside the Netherlands .
Many studies have found that, when alerts are overridden inappropriately, the chance of harm to a patient increases significantly [17,18,19,20,21]. This study identified 16 adverse events that occurred in the inpatient setting. A retrospective chart review found that 11 of the 16 alerts would have potentially been prevented by the Seegnal platform, whereas the current system failed to catch these.
This study has several limitations. Because the design was retrospective, many of the features of the Seegnal platform, for example using electronic medication administration record data to assess the number of medication doses a patient had received, could not be assessed. The tool might perform better in real time with access to these additional features. The Seegnal platform has not been subjected to the customization/configuration of a healthcare facility, which could also improve or worsen performance. As the genetic information available within our biobank was limited at the time of data extraction, we were unable to fully evaluate the pharmacogenetic decision support. We did not measure how clinicians would respond to suggested alternatives to the identified drug-related problems; that will need to be assessed prospectively. Specifically, the platform should be tested in real time to assess the alert load reduction, user interface, and performance characteristics.
We evaluated two commercial applications regarding their performance around medication-related CDS and found that the Seegnal platform substantially outperformed EPIC in alert load, specificity, sensitivity, PPV, and NPV in both the inpatient and the outpatient setting. It would reduce the alert load by nearly twentyfold. The performance of the Seegnal platform is attributed mostly to its ability to personalize alerting because of its real-time consideration of up to hundreds of patient-specific and medication-specific factors. It could also help notify physicians of a small number of new potential ADEs, and—based on the chart review—69% of these potentially could be prevented. This application should be further evaluated in real time, and further evaluation of pharmacogenetic alerts would also be beneficial.
Institute of Medicine. Evidence-based medicine and the changing nature of health care: 2007 IOM annual meeting summary. Washington, DC: National Academies Press; 2008.
Beijer HJ, de Blaey CJ. Hospitalizations caused by adverse drug reactions (ADR): a meta-analysis of observational studies. Pharm World Sci. 2002;24(2):46–54.
Lazarou J, Pomeranz BH, Corey PN. Incidence of adverse drug reactions in hospitalized patients: a meta-analysis of prospective studies. JAMA. 1998;279:1200–5.
Budnitz DS, Pollock DA, Weidenbach KN, et al. National surveillance of emergency department visits for outpatient adverse drug events. JAMA. 2006;296(15):1858–66. https://doi.org/10.1001/jama.296.15.1858.
Makary MA, Daniel M. Medical error- the third leading cause of death in the US. BMJ. 2016;353:i2139.
Corsonello A, Pedone C, Corica F, et al. Concealed renal insufficiency and adverse drug reactions in elderly hospitalized patients. Arch Intern Med. 2005;165:790–5.
Bates DW, Leape LL, Cullen DJ, et al. Effect of computerized physician order entry and a team intervention on prevention of serious medication errors. JAMA. 1998;280:1311–6.
Kaushal R, Shojania KG, Bates DW. Effects of computerized physician order entry and clinical decision support systems on medication safety: a systematic review. Arch Intern Med. 2003;163(12):1409–16.
Nuckols TK, Smith-Spangler C, Morton SC, et al. The effectiveness of computerized order entry at reducing preventable adverse drug events and medication errors in hospital settings: a systematic review and meta-analysis. Syst Rev. 2014;3:56. https://doi.org/10.1186/2046-4053-3-56.
Shamliyan TA, Duval S, Du J, Kane RL. Just what the doctor ordered. Review of the evidence of the impact of computerized physician order entry system on medication errors. Health Serv Res. 2007;43:32–5.
Wolfstadt JI, Gurwitz JH, Field TS, et al. The effect of computerized physician order entry with clinical decision support on the rates of adverse drug events: a systematic review. J Gen Intern Med. 2008;23(4):451–8.
Bates DW, Teich JM, Lee J, et al. The impact of computerized physician order entry on medication error prevention. J Am Med Inform Assoc. 1999;6:313–21.
Lin CP, Payne TH, Nichol P, Hoey PJ, Anderson CL, Gennari JH. Evaluating clinical decision support systems: monitoring CPOE order check override rates in the Department of Veterans Affairs’ Computerized Patient Record System. J Am Med Inform Assoc. 2008;15:620–6.
Nanji KC, Slight SP, Seger DL, et al. Overrides of medication-related clinical decision support alerts in inpatients. J Am Med Inform Assoc. 2018;25(5):476–81. https://doi.org/10.1093/jamia/ocx15.
Nanji KC, Slight SP, Seger DL, et al. Overrides of medication-related clinical decision support alerts in outpatients. J Am Med Inform Assoc. 2014;21(03):487–91.
Kuperman GJ, Bobb A, Payne TH, et al. Medication-related clinical decision support in computerized provider order entry systems: a review. J Am Med Inform Assoc. 2007;14(01):29–40.
Payne TH, Nichol WP, Hoey P, Savarino J. Characteristics and override rates of order checks in a practitioner order entry system. Proc AMIA Symp. 2002;2002:602–6.
Ancker JS, Edwards A, Nosal S, Hauser D, Mauer E, Kaushal R, with the HITEC Investigators. Effects of workload, work complexity, and repeated alerts on alert fatigue in a clinical decision support system. BMC Med Inform Decis Mak. 2017;17:36.
Weingart SN, Toth M, Sands DZ, Aronson MD, Davis RB, Phillips RS. Physicians’ decisions to override computerized drug alerts in primary care. Arch Intern Med. 2003;163(21):2625–31.
Hsieh TC, Kuperman GJ, Jaggi T, et al. Characteristics and consequences of drug allergy alert overrides in a computerized physician order entry system. J Am Med Inform Assoc. 2004;11(06):482–91.
Wong A, Amato MG, Seger DL, et al. Evaluation of medication-related clinical decision support alert overrides in the intensive care unit. J Crit Care. 2017;39:156–61.
Horn J, Ueng S. The effect of patient-specific drug-drug interaction alerting on the frequency of alerts: a pilot study. Ann Pharmacother. 2019;53(11):1087–92. https://doi.org/10.1177/1060028019863419.
Bates DW, Leape LL, Cullen DJ, et al. Effect of computerized physician order entry and a team intervention on prevention of serious medication errors. JAMA. 1998;280(15):1311–6. https://doi.org/10.1001/jama.280.15.1311.
Ash JS, Sitting DF, Campbell EM, Guappone KP, Dykstra RH. Some unintended consequences of clinical decision support systems. AMIA Annu Symp Proc. 2007;2008:26–30.
The G-Standaard: the medicines standard in healthcare. KNMP.nl apothekersorganisatie. https://www.knmp.nl/producten/gebruiksrecht-g-standaard/informatie-over-de-g-standaard/the-g-standaard-the-medicines-standard-in-healthcare. Accessed 20 Dec 2020.
Coleman JJ, van der Sijs H, Haefeli WE, et al. On the alert: future priorities for alerts in clinical decision support for computerized physician order entry identified from a European workshop [published correction appears in BMC Med Inform Decis Mak. 2013;13:122]. BMC Med Inform Decis Mak. 2013;13:111. https://doi.org/10.1186/1472-6947-13-111
This research was supported by Seegnal (Sayvon, Israel), which had no role in the development of the manuscript.
Conflict of interest
John Horn is a consultant to Seegnal US, Inc. and Urovant Sciences, Inc. and is co-author and publisher of The Top 100 Drug Interactions: A Guide to Patient Management. Dr. Bates consults for EarlySense, which makes patient safety monitoring systems; receives cash compensation from CDI (Negev), Ltd, which is a not-for-profit incubator for health information technology startups; receives equity from ValeraHealth, which makes software to help patients with chronic diseases; receives equity from Clew, which makes software to support clinical decision making in intensive care; receives equity from MDClone, which takes clinical data and produces deidentified versions of it; receives equity from AESOP, which makes software to reduce medication error rates; and receives research funding from IBM Watson Health. SN Shah, DL Seger, and JM Fiskio have no conflicts of interest that are directly relevant to the content of this article.
This was approved by Partners Healthcare IRB.
Consent to Participate
Consent for Publication
Availability of Data and Material
Below is the link to the electronic supplementary material.
About this article
Cite this article
Shah, S.N., Seger, D.L., Fiskio, J.M. et al. Comparison of Medication Alerts from Two Commercial Applications in the USA. Drug Saf (2021). https://doi.org/10.1007/s40264-021-01048-0