FormalPara Key Points

It is important to investigate the real-life pattern and natural course of clinical alerts of Clinical Decision Support Systems (CDSSs).

Our data may contribute to the current literature, as they allow researchers to more carefully interpret negative study results of clinical trials investigating CDSSs.

Further research is needed to completely understand the effectiveness of CDSS interventions and our proposed ‘natural course of clinical alerts’ may be an important parameter in assessing the quality of clinical rules and thereby a potential target to optimize CDSSs.

1 Introduction

The population is ageing rapidly and as a result multimorbidity and associated polypharmacy is an increasing health risk leading to considerable mortality and morbidity [1,2,3]. As such, 30% of emergency department visits and hospital admissions in older age are attributable to side effects and inappropriate prescription of medicine, since the risk of inappropriate medication use, adverse drug events (ADE) and medication-related problems (MRPs) is increasing [4,5,6]. These risks increase even further when older patients are hospitalized, at least in part due to the fact that acutely ill older patients are often exposed to new prescriptions by multiple prescribers [7, 8]. Thus, a variety of interventions to reduce MRPs in older hospitalized individuals has been studied with widely varying impact [3, 9]. Most of these interventions studied the participation of pharmacists to ideally prevent, and otherwise reduce the impact of MRPs, although in recent years the involvement of a clinical decision support system (CDSS) has emerged, demonstrating the possibility of reducing potentially inappropriate prescribing in hospitalized older patients [7, 9,10,11].

Recently, two large clinical trials, the OPERAM (OPtimising thERapy to prevent Avoidable hospital admissions in Multimorbid older people) trial and the SENATOR (Software ENgine for the Assessment and optimization of drug and non-drug Therapy in Older peRsons) trial investigated medication optimization supported by a CDSS including the Screening Tool of Older Persons’ Prescriptions (STOPP) and Screening Tool to Alert to Right Treatment (START) recommendations [3, 12]. However, both studies found no significant differences in incidence of adverse drug reactions, drug-related hospital admissions and mortality between the intervention and control arms, and only a low proportion of the recommendations accepted by the pharmacotherapy team or attending physicians [12, 13]. Potential explanations for these low acceptance rates were the fact that recommendations were deemed of low clinical relevance by clinicians as well as variable attitudes to the intervention and/or participation in clinical trials and patient-specific factors [14, 15].

The ultimate goal of a CDSS is to improve patient-related outcomes, however, as long as only low proportions of recommendations are accepted and rules with low clinical relevance are used in CDSSs, it is unlikely this goal will be achieved [13, 16, 17]. These specific factors, presumably leading to low response rates and alert fatigue, have not been recorded in the initial trials evaluating CDSSs. Therefore, the purpose of the present study was to report on existing knowledge gaps in a real-life clinical setting of hospitalized older patients, in order to describe, evaluate and optimize the use of the CDSS in clinical practice. To this end, we investigated the real-life pattern of an in-hospital implemented CDSS, including (i) whether generated alerts were resolved; (ii) whether a recorded action by the pharmacist led to an improved number of resolved alerts; and (iii) description of the natural course of generated alerts, in particular of those in the non-intervention group, as these data are largely lacking in current studies.

2 Methods

2.1 System Details, Description of CDSS System, Clinical Rule Reporter (CRR)

The CDSS used in Zuyderland Medical Centre, a large teaching hospital in the Netherlands, is the Clinical Rule Reporter (CRR). The CRR has been implemented in daily practice since 2016 and is mainly based on guidance for medication-related laboratory testing, dosing support in patients with renal impairment and guidance for optimal use for antibiotics and anticoagulation [18, 19]. It is used for medication surveillance using demographic, medication and laboratory data from admitted patients and contains several clinical rules [19]. The CDSS in this study consisted of 80 rules (supplementary data table 1, see electronic supplementary material [ESM]).

The CDSS analyses the medication from all admitted patients on a daily basis. The clinical pharmacist receives (per patient) a report in which all alerts are given. The clinical pharmacist then assesses whether further action is indicated according to a distinct rule. For this study, we defined this action as ‘an action by the pharmacist’ when a rule was discussed extensively (i.e. consultation between pharmacist and physician) or when an intervention was performed by the pharmacist or by the physician (after consultation).

2.2 Patients

We included patients aged 60 years and older, hospitalized in 2018, from January 01, 2018 to December 31, 2018, in Zuyderland Medical Centre, the Netherlands. Patients admitted to rehabilitation wards and short stay departments were excluded. Demographic data (age and sex) were collected. The evaluation of the CRR was investigated in a retrospective study using a database used for standard care, which is why this study did not require ethical approval.

2.3 Data Collection

The data of the generated alerts for the included patients and their management were extracted to Qlik Sense version September 2020 SR1. Qlik Sense is a tool which visualizes data in an interactive way. All rules were categorized as one of the following: renal function, potassium, antibiotics (intravenous [IV] to oral), antibiotics (long use), opioids/laxatives, anticoagulant therapy and unknown lab value. Alongside the number of unique patients, the following data was collected in order to evaluate the use of the CRR: (i) the total number of alerts, (ii) the number of alerts ‘handled’ by the pharmacist, (iii) the number of alerts resulting in an action of the pharmacist and (iv) the outcome of the alert—described as ‘green’ (resolved), ‘red’ (unresolved) and ‘unknown’. We also calculated the percentage of resolved and unresolved alerts, after excluding the ‘unknown’ alerts. The latter were excluded because it is unknown whether an ‘unknown’ alert was actually resolved by discontinuing the medication (and thus the rule that generated the alert did not apply anymore) or because the patient was discharged.

2.4 Statistical Analyses

Descriptive statistics were used to describe the study population and were presented as mean (SD) or median (IQR) as appropriate. To test the differences in percentages in the groups with or without an action by the pharmacist an unpaired student’s t-test was used. Statistical analyses were performed using SPSS Statistics v25 (IBM, Armonk, NY, USA).

3 Results

3.1 Baseline Characteristics

In 2018, anonymized data from 3574 unique hospitalized patients were included for the current analyses. The mean age was 76.7 (SD 8.3) years and 53% were female. From these patients, in total 8073 alerts were generated, of which 7907 (97.9% of total) were handled by the pharmacist (day 1). The patient characteristics and subdivision of the clinical alerts per category and whether the pharmacist performed or did not perform an action are described in Table 1. The percentages of actions performed varied between the different rule categories from 33.8% (opioids/laxatives) to 94% (anticoagulant therapy), respectively (Table 1).

Table 1 Baseline characteristics
Full size table

In 4083 (51.6%) of the handled alerts, an action was performed by the pharmacist, resulting in 1297 (36.1% after excluding ‘unknown’) resolved alerts after day 1, while when the pharmacist did not perform an action, which was the case in 3824 alerts, 915 (27.3% after excluding ‘unknown’) alerts were resolved (36.1% vs 27.3%, p < 0.001). These results are schematically summarized in Fig. 1.

Fig. 1
figure 1

Schematic overview of alert trajectories and outcome of the alerts (day 1)

Full size image

For the day 3 evaluation, 5750 handled alerts were analysed. In 3025 (52.6%) alerts, an action by the pharmacist was performed, resulting in 1242 (62.4% after excluding ‘unknown’) resolved alerts after day 3, while when no action was performed, which was the case in 2725 alerts, 772 (48.0% after excluding ‘unknown’) alerts were resolved (62.4% vs 48.0%, p < 0.001). These results are schematically summarized in Fig. 2.

Fig. 2
figure 2

Schematic overview of alert trajectories and outcome of the alerts (day 3)

Full size image

3.2 Rule Categories

When we analysed the different rule categories separately, we found that only in the category ‘renal function’ did the percentages differ significantly between an action versus no action of the pharmacist (16.6% vs 10.6%, p < 0.001). In other rule categories, we observed small positive effects when an action was performed by the pharmacist, that is, in the categories potassium, antibiotics (both IV to oral, and long use) and anticoagulant therapy; however, these differences did not differ statistically significant (Fig. 3).

Fig. 3
figure 3

Number of ‘positive’ outcome alerts per clinical rule category divided per action by pharmacist day 1

Full size image

On day 3, similar results were observed, with a significant difference between those with and without an action of the pharmacist only found in the category ‘renal function’ (29.8% vs 19.4%, p < 0.001) (Fig. 4).

Fig. 4
figure 4

Number of ‘positive’ outcome alerts per clinical rule category divided per action by pharmacist day 3

Full size image

4 Discussion

This study demonstrates the real-life pattern and natural course of clinical alerts of an in-hospital implemented CDSS. As such, we have shown that when the pharmacist did not perform an action, 27% and 48% of the generated alerts are resolved after 1 and 3 days, respectively. These percentages improved to 36% and 62% on day 1 and 3 when an action by the pharmacist was performed. Also, we found that the percentages of resolved alerts varied widely between different rule categories.

The finding that actions taken by pharmacists in response to the generated alerts were beneficial and led to an improved total number of resolved alerts was not surprising, since this has been studied extensively in the past [20, 21]. However, this study reveals that many alerts are resolved without a specific intervention by the pharmacist. Our study therefore may provide important insight into the null effect of CDSS in the OPERAM and SENATOR trials, since alerts that are resolved without intervention are also addressed per usual care and therefore are unlikely to improve clinical outcomes [3, 12]. We speculate that these ‘false positive’ alerts may distract from potentially more clinically relevant alerts and therefore lead to a reduced effect of CDSS interventions. We believe that the CDSS may be particularly useful for rare MRPs with, for instance, uncommon high-risk medications, and that therefore the initial clinical trials may have been underpowered to detect the impact of a CDSS on these events.

Our observational study shows some reassuring findings, given the total number of resolved alerts on one hand, but also the total number of resolved alerts in specific categories on the other hand, which are generally considered as more clinically relevant. Most notable are the number of resolved ‘anticoagulant therapy’ and ‘potassium’ alerts, which may indicate that clinicians themselves are perfectly able to determine the clinical relevance and also prioritization of these alerts. Moreover, this is substantiated by the low number of resolved ‘opioid + laxative’ alerts, an otherwise clinically relevant recommendation, which is likely to have been deemed of lower priority in the acute setting.

The discrepancy in acceptance of recommendations between our real-life clinical data and data from recent trials may be explained by several factors. First, in the two largest clinical trials to date, START/STOPP recommendations were used and therefore alert fatigue may have occurred due to lower prioritization of otherwise clinically relevant recommendations [3, 12, 14, 15]. This observation contrasts with our set of clinical rules that are largely directly clinically relevant and also of greater priority in the hospital setting, although our figures also show that clinicians seem to be perfectly able to prioritize this themselves. Second, variable attitudes towards the intervention itself (both OPERAM and SENATOR used software-generated recommendations) or participation in trials by investigators or participants might have played a role, while these factors play no significant role when implemented in clinical practice. Third, in the SENATOR trial, specific exclusion criteria, such as admission to a geriatric ward, were applied that not only limited the generalizability of the findings, but also excluded a large proportion of the population concerned, namely the geriatric population [12, 22]. Nevertheless, since OPERAM had only minimal exclusion criteria, it is unlikely this explains this discrepancy completely [3, 23].

We have shown that different rule categories show varying percentages of resolved alerts. When investigating the impact of computerized interventions, acceptance rates (or agreement with recommendations) vary widely between different studies and interventions [7]. Recommendations may not be accepted by clinicians as the recommendation might be deemed of low clinical relevance, patients may insist on medication continuation, or possible interactions may be monitored instead of direct medication discontinuation [7, 24,25,26].

Our study has several strengths and limitations. Strengths include the large sample size of 3754 unique patients and the real-life clinical setting of an already implemented CDSS. Our study also has some limitations. First, our definition of ‘action taken by the pharmacist’ is broad, including when a rule was discussed extensively (i.e. consultation between pharmacist and physician) or when an intervention was performed by the pharmacist or the physician (after consultation). This might increase the effect of the pharmacist because some of the ‘pharmacist interventions’ were performed by the physician. Second, our study is limited by its retrospective and observational design. Third, in 2018 our CDSS only consisted of 80 clinical rules. As such, clinically relevant, but also well -known rules (such as START/STOPP) have not been included in this version of the CDSS, making this study difficult to interpret in the current field of studies investigating generic CDSSs. Fourth, although this is a relatively large study, we only had access to a limited set of clinical data and were therefore unable to investigate the impact of clinical predictors on the percentage of alerts resolved. Despite this, we have included a cohort of patients in which MRPs are of particular interest, namely the geriatric population, often excluded in clinical trials.

5 Conclusions

This study demonstrates the pattern and natural course of clinical alerts of an in-hospital implemented CDSS in a real-life clinical setting of hospitalized older patients. Next to the beneficial effect of actions by pharmacists, we have also shown that many alerts are resolved without specific interventions. As such, our study provides insight into the spontaneous course of resolved alerts, since these data are currently lacking in the literature. Further research is needed to completely understand the effectiveness of CDSS interventions and our proposed ‘natural course of clinical alerts’ may be an important parameter in assessing the quality of clinical rules and thereby a potential target to optimize CDSS.