INTRODUCTION

Despite decades of research, inappropriate prescribing rates for acute respiratory complaints remain high. For example, half of all antibiotic prescriptions for acute respiratory infections (ARIs) are inappropriate1; 10% of sore throats are estimated to be bacterial, but antibiotic-prescribing rates have remained at 60% since 20002; and 71% of acute bronchitis cases are prescribed antibiotics, despite repeated guidelines advising against this3.

Many studies have attempted to use clinical decision support (CDS) tools within the electronic health record (EHR) to reduce antibiotic-prescribing ARIs with mixed success.4, 5 One cluster randomized trial observed a significant decline in antibiotic prescribing for acute bronchitis with a CDS tool (from 74 to 61%, P = 0.01).6 Two other cluster randomized trials evaluating two different CDS tools in primary care practices found no change in antibiotic prescribing with low adoption of the intervention by providers.7, 8 Another study reported a 10% increase in antibiotic prescribing after implementing a CDS, despite a 19% decrease in antibiotic prescribing among the small number of visits where the CDS was used.9 Inappropriate antibiotic prescription was lower where primary care sites used peer comparison and accountability.10 These mixed results suggest that CDS tools’ ability to reduce inappropriate antibiotic prescribing is constrained by clinicians’ variable usage.

We previously developed and tested a novel tool with two integrated clinical predication rules (iCPRs) to reduce antibiotic prescribing for ARIs in primary care for the iCPR1 study. Specifically, the two iCPRs were the rule by Heckerling et al. for pneumonia and the rule by Centor et al. for streptococcal pharyngitis.11 The iCPRs were embedded directly into the EHR to help clinicians identify patients most likely to have a bacterial infection and increase appropriate antibiotic-prescribing rates. The study observed a 35% reduction in antibiotic prescribing for relevant ARIs. These results were supported by high adoption of the tool (58%).11 However, the study was limited by a single site design.

To assess the generalizability of the original findings in iCPR1, the tool was adapted and implemented across diverse primary care settings; effectiveness was tested in these new contexts12. Our approach promoted provider adoption of this complex CDS at the point of care by seamlessly blending CDS into provider workflow.

The iCPR2 study consisted of 2 phases. Phase 1 leveraged user-centered design and usability techniques used in the original CDS tool development to adapt to diverse workflows across the sites (as described in prior publication11).

This paper reports on phase 2, where we conducted a cluster randomized clinical trial (RCT) studying (1) adoption of the tool for two target conditions and (2) effects of the new CDS tool on provider diagnostic and treatment patterns.

METHODS

We evaluated the effectiveness of the user-centered design adapted iCPR tool in combination with academic detailing versus academic detailing alone to reduce antibiotic-prescribing rates among intervention practices. The study process was evaluated using the RE-AIM framework described below.

Setting

The institutional review board at all participating sites approved the study protocols. The CDS RCT was conducted within two large academic health system primary care networks in Wisconsin and Utah with a mix of internal and family medicine providers (attending physicians, residents, fellows, physician assistants, and nurse practitioners).

Exclusion and Inclusion Criteria

All internal medicine and family medicine primary care practices within each health system were invited to participate. If consent was provided by practice leadership, all providers were considered participants. The single site used in iCPR1 was excluded.

Study Design

Primary care practices were randomized using a random number generator to a control group (usual care + academic detailing) or an intervention group (iCPR2 CDS tool + academic detailing) (see Appendix A in the ESM).

Academic Detailing

Providers in both arms received a 30–45-minacademic detailing session during a regular practice meeting. Academic detailing included a review of the evidence and appropriate use of the Centor (McIsaac for children) and Heckerling CPRs used in the study, discussion of evidence-based diagnosis of streptococcal pharyngitis and pneumonia, and review of standard guidelines for treating the two diseases. Participants received handouts with the CPRs and treatment guidelines and links to online resources. Intervention providers received a live demonstration of the iCPR tool embedded in the EHR. Providers who were unable to attend were given access to printed and online training materials.

Adaptation of CDS Workflow

In the original CDS workflow for iCPR1, the tool was triggered by a matching reason for visit, diagnosis, or diagnosis in addition to antibiotic order related to an acute respiratory tract infection (Fig. 1); the participant then received an alert that, if accepted, would reveal the risk calculator.12 Following calculator completion, the participant was provided a link in an additional alert to bundled order sets that included diagnosis and next steps.12 Based on usability testing, the CDS tool was adapted for distinct workflows at each site (Fig. 1).13 There was a strong desire by providers to have the tool trigger earlier in their workflow. Therefore, the CDS tool triggered first upon entry of the reason for visit (cough, sore throat, URI) in the EHR and included a second trigger upon diagnosis entry (as done in the original iCPR1 study).14 The iCPR2 triggers are detailed in Appendix B in the ESM. We activated the CDS tool for providers in the intervention practices from October 22, 2015 through May, 2016, based on timing of academic detailing and collected data until June 30, 2018.

Figure 1
figure 1

Adapted iCPR workflow for Wisconsin and Utah. *MA, medical assistant; RFV, reason for visit (cough, sore throat, or upper respiratory infection); BPA, best practice alert; iCPR, integrated clinical prediction rule.

Full size image

CDS Scoring and Recommendations

Based on chief compliant, the CDS prompted providers to complete an iCPR risk score calculator and gave management recommendations and bundled order sets with documentation based on the score. CPRs and recommendations for each score based on the risk of streptococcal pharyngitis and pneumonia are described in Table 1. Patient encounters with the study providers were monitored, and data were collected from the EHR for the intervention period. If dismissed, the iCPR alert would not appear again during that visit; providers could retrieve the iCPR order set anytime from the order entry workflow.

Table 1 Clinical Prediction Rules and Recommendations Based on Calculated Risk Score
Full size table

Outcomes

Provider demographics (age, gender, degree, training level, and specialty) were collected from available EHR data. Patient characteristics (age, gender, and race) were selected for analysis based on availability in the EHR and their potential to influence provider decisions.

Process Outcomes

This study process and measures of implementation are adapted from the RE-AIM framework12 and have been modified to reflect a clinical context. Reach pertains to percentages of participating primary care clinics at each site. Efficacy pertains to differences in antibiotic prescription rates between intervention and control groups as discussed in primary outcome. Adoption pertains to utilization rates of the tool in the intervention group for eligible visits in which (1) the provider opened the tool after it was triggered and (2) the provider completed the risk score calculator embedded in the tool (see Appendix C in the ESM). Implementation pertains to the fidelity of use of the intervention tools based on the rate at which the provider signed the associated bundled order set (see Appendix C in the ESM). Maintenance pertains to The comparison of adoption and efficacy trends between years 1 and 2 (see Appendix D in the ESM).

Primary Outcome

Efficacy of the tool was assessed based on the differences in provider antibiotic ordering. Antibiotic orders were evaluated by examining EHR data from patient encounters where patients had a visit reason that triggered the tool in the intervention group or would have triggered the tool in the control group. Positive antibiotic orders were identified as any antibiotic prescribed during the encounter. We measured the frequency, rates, and type of antibiotics prescribed for eligible encounters.

Secondary Outcomes

A secondary outcome of inappropriate antibiotic prescribing was calculated using a list of previously published diagnoses considered appropriate to prescribe antibiotics for an ARI.10 Any oral antibiotic prescribed during an iCPR ARI encounter that was not connected with an appropriate visit diagnosis was considered inappropriate. Antibiotic prescription appropriateness was considered a secondary outcome to be consistent with the original iCPR1 primary outcome (overall antibiotic prescribing), and because its determination of inappropriateness is dependent on EHR diagnosis coding which has reliability issues. Additional secondary outcomes measured were order frequencies of chest radiographs, rapid streptococcal tests, and pharyngitis throat cultures during eligible encounters.

Statistical Analysis

Differences in the characteristics of patients seen by providers in the two arms were measured using t test and χ2 test, as appropriate. However, due to a large amount of skewness, the number of patients per provider was compared with the Mann-Whitney U test. Patient characteristics were compared using similar methods. Process and primary and secondary outcomes were compared for all intervention and control visits and were stratified by pneumonia or streptococcal iCPR tool. Absolute risk (risk difference) and relative risk (RR) ratios of antibiotic prescribing in the intervention group versus the control group were compared using a generalized estimating equation model (with a logit link) with robust standard errors to account for clustering of patient visits (encounters) within practices and subsequent correlated outcomes. The outcomes were further stratified by primary care networks in Wisconsin and Utah in Appendix E in the ESM. Because the median age of patients differed in the intervention and control groups, we performed a secondary analysis adding age as a covariate in our age-adjusted models. Analyses were conducted with the R statistical software (R version 3.5.2) using 2-sided P values.

RESULTS

From October 22, 2015, through June 30, 2018, 33 primary care practices with 541 providers participated in the study. Two practices declined to be part of the study. There were no statistically significant differences in demographics between providers in the intervention and control groups. Over 100,573 patient visits occurred during the study period. Patient age, sex, race, and number of diagnosis recorded were statistically different but were small and likely clinically insignificant (Table 2).

Table 2 Demographic Characteristics of Providers and Patients
Full size table

Process Outcomes

Regarding reach, 94% of the practices approached to be part of the study in the two institutions participated (33 out of 35). The CDS tool was triggered 42,126 times among intervention practices (Table 3). This tool had low adoption rates, with providers completing the calculator in 6.9% of triggered encounters. Low utilization rates were observed in both health systems (Wisconsin 9.6%, Utah 2.5%) for strep (7.9%) and pneumonia (6.3%). Regarding implementation, low calculator completion rates were complemented by low rates of bundled order set completion overall (0.9%) and at each site (Wisconsin 1.3%, Utah 0.1%). Regarding maintenance, Appendix D in the ESM shows rates of antibiotic prescription from 0 to 33 months, where the completion rate of iCPR calculator decreased steadily over time.

Table 3 Process Measures by Practice Randomization Status
Full size table

Primary Outcomes

Regarding efficacy, there were no statistically significant differences in overall antibiotic prescriptions rates (Table 4), rapid streptococcal test orders, or chest X-ray orders between the control and intervention sites (Table 5). Providers in the intervention group were equally likely to order antibiotics (age-adjusted RR, 0.99; 95% CI, 0.87–1.11; P = 0.78) (Table 4).

Table 4 Antibiotic Prescriptions by Practice Randomization Status
Full size table
Table 5 Test Orders by Practice Randomization Status
Full size table

Secondary Outcomes

There were no differences in the rates of total inappropriate antibiotic prescribing for ARIs between intervention and control sites (age-adjusted RR, 0.9; 95% CI, 0.7–1.2; P = 0.59; see Appendix E in the ESM). When stratified by pneumonia and pharyngitis, the RR of antibiotic ordering remained insignificant for both (age-adjusted RR, 0.1.0 and 0.6; 95% CI, 0.8–1.2, 0.3–1.1; P = 0.72, 0.08, respectively). The intervention group was not less likely to order rapid streptococcal tests in sore throat encounters or chest radiographs in cough encounters (Table 5).

DISCUSSION

Our study found no evidence for an effect of the CDS system along with academic detailing on antibiotic prescribing for visits that triggered the tool (sore throat, cough, URI) in diverse primary care settings. The results were robust to analyses stratified by practice, provider, and provider nested within practice. This lack of efficacy may be driven by the very low adoption and utilization rates of the tool at all clinical sites. While contrary to the results from our previous trial, our results are consistent with several CDS trials targeting antibiotic ordering in ARIs.6, 8

These results are surprising given the integrated use of multiple user-centered design methods grounded in relevant implementation frameworks. The study comprehensively assessed provider workflows and derived insights through iterative usability testing and adaption phase.12 Careful attention was paid to ensuring leadership buy-in from the participating practices. When lower than expected rates of use of the tool became apparent, we conducted repeated assessments of barriers at the sites and implemented several adjustments over time. Despite these efforts, the low adoption rates highlight several important barriers that we were unable to overcome, detailed in Mann et al.14

The first was the rise in usage of CDS and in particular of clinical alerts since the original iCPR study. At most institutions using a major EHR vendor system, there are now often hundreds of alerts targeting primary care providers.15 In contrast, during the original iCPR study at a large academic health center in 2009, there was only a handful of alerts (interruptive or non-interruptive) built to interact with primary care providers on a daily basis (not including drug interaction alerts). This change in the context of CDS and the well-documented rise in “alert fatigue” are likely contributors to the low adoption rate in this study.16, 17

In addition, while our implementation of user-centered design sought to iteratively tailor our CDS tool to the diverse primary care workflows we encountered, we hypothesize that this potentially caused some “over engineering” that may have contributed to low adoption rates. In order to foster provider and leadership support, we adapted our previously used tool to fit new workflows. For example, some providers used their medical assistants to “queue up” their documentation, which bypassed the typical trigger points for the iCPR2 tool. During usability testing, changes were made to the tool at sites using this workflow to insert the tool into a more amenable location, one that was not disruptive but may have been too passive and simply blended into the background—never having a chance to influence provider behavior.

Moreover, while our results likely represent a lack of engagement with CDS (suboptimal design, workflow barriers, etc.), they also suggest that, despite being supportive of the study and having tailored the tool to their preferences, providers may not have derived sufficient clinical relevance from the tool to warrant interaction. Low clinical relevance is a leading cause of CDS “ignoring” and fatigue.18 Providers may have felt they did not need “support” to appropriately prescribe antibiotics. This attitude is a well-known barrier to changing provider antibiotic-prescribing behavior and has been attributed to patient-, provider-, and system-level factors.5, 19 Issues such as patient expectations for antibiotics, time pressures, diagnostic uncertainty, provider overconfidence in the accuracy of their diagnosis, and skepticism towards the true risk of overprescribing continue to undermine efforts to reduce inappropriate prescribing.19, 20

The literature on CDS adoption rates is variable, reflecting the heterogeneity of designs, interventions, and the fast-changing environment within which CDS systems are deployed. Prior investigations of CDS achieved a wide range of adoption rates from less than 10% to more than 60% depending on the tool and context.4, 6, 8, 11, 21 Our results are similar to prior studies showing adoption rates of less than 10% with CDS tools targeting inappropriate antibiotic prescribing for ARIs.6, 8 For example, a similar study seeking to reduce antibiotic prescriptions for URIs was used in only 7% of eligible visits.6

Limitations

Our study had several limitations. Our use of an adaptive trial design allowed accommodation of diverse workflows, sizes, and other characteristics of the sites—but created challenges to directly comparing and understanding our results in relation to the original iCPR study. Secondary analyses identified small differences in antibiotic prescribing at the site level but additional analyses using a pre-post measure of prescribing determined that these were due to a random imbalance of baseline antibiotic prescribing. This, alongside the academic nature of the sites, limits the generalizability of our results; our observations show the capabilities of this tool, in this context, with our implementation approach. While overall utilization was low, there was slightly more utilization at the University of Wisconsin (UW). Interviews with participating providers at each institution revealed no difference in the reasons of use of CDS between sites. We also acknowledge that there may be significant provider-to-provider variation in the use of the CDS tools (though the observed overall usage rate was quite low). We are currently investigating these potential individual variations in tool adoption.

We also did not include pediatric sites, which may have altered our results for this age group. Although our background rate of antibiotic prescribing was 45% (cough/URI at 47% and sore throat at 39%), which is a little lower than the national CDC estimates of 50%1, these rates were similar to the iCPR1 study. Although we did measure and observe no changes in follow-up visits to primary care, the emergency room or the hospital after iCPR encounters, we did not perform a manual chart review of follow-up visits to corroborate visit diagnoses. However, our prior studies had established the safety of the iCPR CDS, and the implemented tools are considered best practices11.

CONCLUSION

In conclusion, we demonstrated that a CDS tool iteratively adapted to diverse primary care workflows did not significantly reduce antibiotic prescribing for ARIs. This contradicted our previous single site study. Failure to change provider behavior was likely driven by low usage rates of the CDS tool; a phenomenon that may have been affected by system factors like alert fatigue and complexity of EHR workflows, patient factors such as patient expectations for antibiotics, and multiple provider factors. The observed drop in engagement with the CDS tools between studies suggests that the mixed results of other studies in this field may be in part due to the changing healthcare delivery context. Future CDS tools targeting antibiotic prescribing for ARIs will need to overcome these barriers to change provider-prescribing behavior and reduce inappropriate antibiotic prescribing. Potential next steps may include pivoting the target of the CDS from providers to other members of the care team who are less burdened by the barriers we observed.