FormalPara Key Points

The Seegnal platform triggered fewer alerts in all settings.

Lower alert rates would reduce alert burden and alert fatigue and improve safety.

1 Introduction

Adverse drug events (ADEs) represent a substantial patient safety issue and occur in approximately 1.5 million inpatients every year in the USA [1]. Some studies suggest ADEs account for up to 5–17% of hospital admissions [2,3,4,5]. Of the 1.5 million ADEs occurring in the USA annually, approximately 400,000 are considered preventable. Medical errors are among the top ten causes of death in the USA, cause 6–10% of hospital admissions, and result in costs estimated to be more than $US1000 per patient per year [5]. ADEs are defined as untoward medical occurrences associated with the use of drugs that may or may not have been preventable, whereas medical errors are defined as preventable adverse events. More than 20% of patients treated with multiple medications (polypharmacy) experience major drug-related problems such as adverse side effects, lack of efficacy, and unnecessary hospitalizations [6]. One way to reduce the frequency of ADEs is to utilize clinical decision support (CDS) systems.

Overall, CDS systems have been shown to reduce medication errors by 81% in the inpatient setting, although these data came from internally developed applications with highly tuned decision support [6, 7]. Even though nearly all electronic health records (EHRs) at most hospitals include some form of CDS, nearly all this CDS is vendor developed and often does not address patient-specific factors. CDS represents an effective way to reduce errors and ADEs [8,9,10]. However, this impact may be decreased or even extinguished if too many clinically inappropriate alerts are given [11,12,13]. One study reported that approximately 60% of overrides of alerts were appropriate and that override rates varied based on type. For example, override rates were 98% for duplicate medication alerts, 96.5% for drug allergy alerts, 82.5% for nonformulary medication alerts, 26.4% for age-based medication substitution alerts, and just 2.2% for renal alerts [14]. Many other studies have found even higher override rates [15, 16].

Alert fatigue introduces the risk of missing critical alerts that may compromise patient safety [17,18,19,20]. A study performed at our institution using a legacy home-grown EHR system found that inappropriately overridden CDS alerts were associated with an increased risk of ADEs [21]. Nearly all major US healthcare organizations have now switched to commercial EHRs, with EPIC and Cerner being the market leaders, and organizations generally contract for medication decision support with a third-party company, with three companies controlling nearly all of the market: FDB, Medispan, and Multum (which is owned by Cerner). In such cases, the EHR imposes some logic on top of the linked database, and the combined capacities compose the CDS solution.

In this study, we calculated the positive predictive value (PPV), negative predictive value (NPV), sensitivity, and specificity, and overall alert load generated by the Seegnal platform compared with those generated by EPIC as implemented within our healthcare system.

2 Methods

2.1 Technology Assessed

Seegnal (Tel Aviv, Israel) has developed an application called the Seegnal platform. This platform is based on a multitier mechanism that relies on proprietary, real-time algorithm-derived alert personalization, Hansten & Horn’s knowledge-based clinical drug interaction evaluations, the University of Washington’s pharmacokinetic and pharmacogenetic databases, and FDB data. The Seegnal platform assesses a wide scope of relevant drug-related problem data including drug–drug interactions, drug allergy, drug dosing, drugs in pregnancy/lactation, drugs and smoking, drug–kidney/liver function interactions, pharmacokinetic drug interactions, drug–food/herbals/vitamin interactions, and pharmacogenetic interactions. The Seegnal platform simultaneously assesses multiple patient-specific parameters (e.g., drug dose/administration route, renal function, laboratory values, electrocardiogram, smoking), and medication-specific parameters (e.g., potency, dose–response curves, time to steady state, accumulative dosing, combined effects of multiple [more than two] medications) in real time to determine the need to trigger an alert or bypass the potential alert for the specific patient. The Seegnal platform also has an innovative user interface that enables the provider at the point of care to detect, prioritize, and choose an appropriate alternative. The Seegnal platform’s user interface was not assessed in this evaluation. To date, one study has been conducted using patient-specific parameters to alert on drug interactions compared with a customized vendor-based CDS database [22]. No study has directly evaluated the alert load produced by the Seegnal platform compared with those of other vendor-based CDS.

2.2 Study Site and Data

This study was performed at Brigham and Women’s Hospital (BWH), Boston, MA, USA, a large urban academic medical center that uses a leading vendor EHR system, EPIC (Verona, WI, USA). BWH provided Seegnal with deidentified inpatient, outpatient, and inpatient genetic electronic medical record-extracted datasets of 657, 2731, and 413 patients, respectively. The dataset contained 56 predefined structured (coded) parameters that were potentially documented in the patients’ EHR during their hospitalization period and/or outpatient visits between 3 June 2018 and 9 June 2018. The dataset for inpatients with genetic data contained all data files during any inpatient admission between 1 January 2017 and 31 December 2019 as only a small portion of patients had genetic data available. The inpatient genetic records were identified by Partners Healthcare Biobank. The purpose of the Partners Biobank is to help researchers understand how health is affected by genes, lifestyle, and environment. Patients who consent to participate in the Partners Biobank contribute a DNA sample for genetic testing, including pharmacogenomics testing. During our data pull, we identified patients with mutations at cytochrome P450 (CYP)-3A5. Examples of parameters extracted were patients’ medication list with start and end dates, dose, frequency, route of administration, comments if any, active problem list, smoking status, pregnancy status, and various laboratory results such as potassium, creatinine, liver function tests, and so on. Specific patient modifiers/factors/data elements extracted from the EHR are listed in the electronic supplementary material.

Seegnal then ran the retrospective dataset against the Seegnal platform. The output from the Seegnal platform provided the alerts that would have been triggered in only the Seegnal platform, in both the Seegnal platform and EPIC, and only in EPIC as utilized within BWH. The alerts were then classified into predefined categories and reviewed for appropriateness to the specific patients. Two pharmacists independently reviewed a random sample of up to 100 alerts per each of the predefined categories (977 inpatient alerts, 808 inpatient genetic alerts, and 938 outpatient alerts, for a sum of 2723 alerts) but were blinded as to whether the alert originated from only the Seegnal platform, both the Seegnal platform and EPIC, or only EPIC.

The types of alerts reviewed included disease–drug contraindication, dosing, drug–drug interaction, duplicate therapy, geriatric, most probable side effects, renal dosing, pregnancy, pharmacokinetic drug interactions, and pharmacokinetic drug interactions with the patient parameter of smoking. The pharmacists (SS, DS) then compared their review; for any differences, two additional reviewers (DB, a doctor with extensive experience in patient safety, and JH, a pharmacist with extensive experience with drug–drug interactions) made the final decision as to whether or not the alert was appropriate.

2.3 Analysis

Once the random sample alerts were evaluated, we then calculated the PPV (true positive/(true positive + false positive)), NPV (true negative/(true negative + false negative)), specificity (true negative/(true negative + false positive)), sensitivity (true positive/(true positive + false negative)), and overall interruptive alert load generated by the Seegnal platform compared with those of EPIC as it functions at BWH. True positives were alerts considered appropriate for the patient, false positives were alerts that were not appropriate for the patient, true negatives were alerts we agreed should not be triggered, and false negatives were alerts that should have triggered but did not. Examples of each are provided in Table 1. Analysis was performed in SAS v. 9.4 (SAS Institute, Cary, NC, USA), and significance was set using a two-sided p value of < 0.05.

Table 1 Examples of true positive, false positive, true negative, and false negative alerts
Full size table

We excluded evaluation of the appropriateness of the genetic alerts returned from the Seegnal platform from our final inpatient and genetic sample because we only had data for one pharmacogenetic marker (CYP3A5), so very few alerts fired. We evaluated all the other alerts for the inpatient genetic cohort. We also reviewed all the inpatient/inpatient genetic (n = 69) and outpatient (n = 65) charts for patients within our cohort who were receiving ten or more concomitant medications with alerts we found to be appropriate to assess whether any adverse events had occurred and whether the Seegnal platform could have prevented them.

3 Results

3.1 Patient Demographics

The inpatient cohort consisted of 657 patients with 665 admissions; eight patients had two admissions. The inpatient genetic cohort consisted of 413 patients with 413 visits. The outpatient cohort consisted of 2731 patients with 2749 visits; 18 patients had two visits during the study period. Demographics of all patients can be found in Table 2.

Table 2 Patient demographics
Full size table

3.1.1 Summary of Main Outcomes

In the inpatient setting, the Seegnal platform triggered 1697 alerts, which was 94% less than the EPIC platform, which triggered 27,540 alerts (Table 3). Sensitivity was 100% for the Seegnal platform and 68.8% for EPIC. In the inpatient genetic group (Table 3), the Seegnal platform triggered 1493 inpatient alerts and EPIC triggered 20,975. The sensitivities were similar: 81% for the Seegnal platform and 78.5% for EPIC. In the outpatient setting (Table 3), the Seegnal platform triggered 93% fewer alerts and sensitivity was 88.6%, whereas sensitivity was 78.3% for EPIC.

Table 3 Inpatient alerts, inpatient genetic alerts, and outpatient alerts
Full size table

3.2 Inpatient and Outpatient Chart Review

We identified 16 adverse events that occurred in the inpatient setting. Of these, the Seegnal platform potentially would have prevented 11 (68.8%). Examples included patients who developed hypotension, bradycardia, QTc prolongation, increased combined side effects, and renal dysfunction. One adverse event involved a patient with a history of nausea and vomiting from oral potassium chloride, which might have been prevented if it had been documented in a structured format under allergies/intolerances instead of being included in the free-text chart notes. Overall, reviewers judged that four (25.0%) adverse events were nonpreventable, so no CDS could have prevented them, and one adverse event was possibly preventable. Of the 11 preventable adverse events that did occur, nine were judged serious and two were considered significant (Table 4). We found no adverse events in the outpatient setting.

Table 4 Description of adverse events identified by chart review
Full size table

4 Discussion

We evaluated the alerting performance of the Seegnal platform compared with that of the EPIC combination and found that the Seegnal platform would have triggered nearly twentyfold fewer alerts, with higher sensitivity and higher specificity in both the inpatient and the outpatient setting. At the same time, the Seegnal platform potentially could have prevented a small number of adverse events associated with issues for which EPIC did not alert.

The high alert burden was consistent with other recent literature reporting very high override rates, mainly because of the inappropriate triggering of alerts [15, 19,20,21]. Not only did the Seegnal platform have a much lower alert load but it also had higher specificity, higher sensitivity, better PPV, and better NPV than EPIC in both the inpatient and the outpatient setting. It achieved this by utilizing patient-specific parameters that can reduce inappropriate alerting and diminish alert fatigue.

EHRs linked with CDS have been demonstrated to prevent many types of medication errors in an internally developed application [23]. However, nearly all systems implemented in the USA are now commercial, and other studies suggest these are not yet as effective as some of the earlier implementations [21]. The potential benefit of CDS is diminished by inappropriate alerting [11,12,13]. Studies have also shown that CDS can have unintended consequences. One study found that CDS contained incorrect or misleading CDS content or out-of-date content [23]. Another study also found alert fatigue and user unfriendly systems with limited ability to customize databases or output to be major problems [24]. The frequency of inappropriate alerts leads clinicians to ignore nearly all alerts, even those that can lead to patient harm. The Seegnal platform employs a CDS knowledge base that is patient specific and flexible and can be tailored to an institution’s specific requirements. This reduces the inappropriate alert load and alert fatigue. Overall, the Seegnal platform outperformed our current CDS system.

Other countries have taken a different approach for medication-related decision support. Notably, the Dutch use a database called G-Standaard, which is issued monthly by Z-Index, an organization owned by the Royal Dutch Association for the Advancement of Pharmacy [25]. This database has been refined to reduce some of the false-positive alerts but is not used outside the Netherlands [26].

Many studies have found that, when alerts are overridden inappropriately, the chance of harm to a patient increases significantly [17,18,19,20,21]. This study identified 16 adverse events that occurred in the inpatient setting. A retrospective chart review found that 11 of the 16 alerts would have potentially been prevented by the Seegnal platform, whereas the current system failed to catch these.

This study has several limitations. Because the design was retrospective, many of the features of the Seegnal platform, for example using electronic medication administration record data to assess the number of medication doses a patient had received, could not be assessed. The tool might perform better in real time with access to these additional features. The Seegnal platform has not been subjected to the customization/configuration of a healthcare facility, which could also improve or worsen performance. As the genetic information available within our biobank was limited at the time of data extraction, we were unable to fully evaluate the pharmacogenetic decision support. We did not measure how clinicians would respond to suggested alternatives to the identified drug-related problems; that will need to be assessed prospectively. Specifically, the platform should be tested in real time to assess the alert load reduction, user interface, and performance characteristics.

5 Conclusion

We evaluated two commercial applications regarding their performance around medication-related CDS and found that the Seegnal platform substantially outperformed EPIC in alert load, specificity, sensitivity, PPV, and NPV in both the inpatient and the outpatient setting. It would reduce the alert load by nearly twentyfold. The performance of the Seegnal platform is attributed mostly to its ability to personalize alerting because of its real-time consideration of up to hundreds of patient-specific and medication-specific factors. It could also help notify physicians of a small number of new potential ADEs, and—based on the chart review—69% of these potentially could be prevented. This application should be further evaluated in real time, and further evaluation of pharmacogenetic alerts would also be beneficial.