Advertisement

Journal of General Internal Medicine

, Volume 32, Issue 11, pp 1210–1219 | Cite as

Primary Care Providers’ Opening of Time-Sensitive Alerts Sent to Commercial Electronic Health Record InBaskets

  • Sarah L. CutronaEmail author
  • Hassan Fouayzi
  • Laura Burns
  • Rajani S. Sadasivam
  • Kathleen M. Mazor
  • Jerry H. Gurwitz
  • Lawrence Garber
  • Devi Sundaresan
  • Thomas K. Houston
  • Terry S. Field
Original Research

Abstract

Background

Time-sensitive alerts are among the many types of clinical notifications delivered to physicians’ secure InBaskets within commercial electronic health records (EHRs). A delayed alert review can impact patient safety and compromise care.

Objective

To characterize factors associated with opening of non-interruptive time-sensitive alerts delivered into primary care provider (PCP) InBaskets.

Design and Participants

We analyzed data for 799 automated alerts. Alerts highlighted actionable medication concerns for older patients post-hospital discharge (2010–2011). These were study-generated alerts sent 3 days post-discharge to InBaskets for 75 PCPs across a multisite healthcare system, and represent a subset of all urgent InBasket notifications.

Main Measures

Using EHR access and audit logs to track alert opening, we performed bivariate and multivariate analyses calculating associations between patient characteristics, provider characteristics, contextual factors at the time of alert delivery (number of InBasket notifications, weekday), and alert opening within 24 h.

Key Results

At the time of alert delivery, the PCPs had a median of 69 InBasket notifications and had received a median of 379.8 notifications (IQR 295.0, 492.0) over the prior 7 days. Of the 799 alerts, 47.1% were opened within 24 h. Patients with longer hospital stays (>4 days) were marginally more likely to have alerts opened (OR 1.48 [95% CI 1.00–2.19]). Alerts delivered to PCPs whose InBaskets had a higher number of notifications at the time of alert delivery were significantly less likely to be opened within 24 h (top quartile >157 notifications: OR 0.34 [95% CI 0.18–0.61]; reference bottom quartile ≤42). Alerts delivered on Saturdays were also less likely to be opened within 24 h (OR 0.18 [CI 0.08–0.39]).

Conclusions

The number of total InBasket notifications and weekend delivery may impact the opening of time-sensitive EHR alerts. Further study is needed to support safe and effective approaches to care team management of InBasket notifications.

KEY WORDS

electronic health records health services research health information technology healthcare communication 

INTRODUCTION

Electronic health records (EHRs) are used by healthcare teams as a platform for communication. EHR notifications are delivered asynchronously to secure InBaskets (e.g., Epic’s “InBasket,” GE Centricity’s “Documents,” Allscripts’ “Task List”), and address a variety of clinical tasks including triage of urgent patient issues, care coordination, test result review, medication refill management, and direct patient communication via tethered patient portals.1 12 In the Veterans Health Administration (VHA), “View Alerts” fill a similar role and are used for some (though not all) of these tasks.

While the level of urgency varies, some of these EHR notifications convey time-sensitive information requiring prompt attention,5 , 7 , 13 15 and a missed or delayed review can impact patient safety and compromise care.7 , 8 , 12 , 16 Previous studies have documented the burden associated with notifications (including View Alerts).1 12 A nationwide survey of Veterans Health Administration (VHA) primary care providers (PCPs) found that more than two-thirds (69.6%) received more messages than they could effectively handle, and almost a third (29.8%) missed results that led to care delays.17 These studies often focus on VHA systems; less is known about factors associated with timing of message opening outside VHA. Understanding factors associated with message viewing would inform efforts to improve the rate at which time-sensitive material receives prompt attention.

In a previous randomized controlled trial,14 automated asynchronous notifications were delivered to PCP InBaskets and highlighted medication safety concerns and the need for follow-up post-hospitalization. Given the time-sensitive nature of these study-generated notifications, we refer to them as “alerts.” These alerts failed to improve outcomes among older patients.14 , 18 20 Primary care teams receiving alerts were no more likely to schedule timely post-hospital office visits or order recommended blood tests; rehospitalization rates were also unaffected. Our current study examines care team response to the medication safety alerts generated in that trial. We tracked timing of alert opening, with the goal of identifying patient, provider, and contextual factors associated with alert opening. Our analysis was informed by a physician focus group, assessing attitudes toward and experiences with EHR notifications, including these alerts.

METHODS

Study Design

We present results of a secondary data analysis of PCP opening of time-sensitive study-generated alerts for older patients following hospital discharge. The data were collected from the intervention arm of a prior randomized experiment.14 The goal of the prior experiment was to increase rates of office visits following hospital discharge and reduce rates of undesirable patient outcomes (rehospitalization, adverse drug event) among an older population. At the time of hospital discharge, patients were randomized to the intervention (EHR alerts sent to the patients’ PCP if medication concerns were identified) or usual follow-up care.

The institutional review boards at Reliant Medical Group and the University of Massachusetts Medical School approved this study, and waiver of consent was obtained.

Setting

This study was conducted at a large Massachusetts healthcare system with 217 multispecialty outpatient physicians at 15 sites throughout central Massachusetts. All sites used an EHR from Epic Systems Corporation.

Study Sample

Approximately 140,000 adults received primary care through this healthcare system; approximately 24,000 were 65 years of age or older and were members of a local health plan. Patients who were members of this health plan and who were discharged to home from the primary hospital used by the health system from August 26, 2010 to August 25, 2011 were included in the prior trial. We included in our analysis only patient discharges for which PCP alerts were triggered, totaling 799 patient discharges in the intervention group (corresponding to 713 patients).

EHR-Delivered Automated Care Transition Alerts

The processes of developing the automated EHR-based intervention and medication alert guidelines have been described previously.18 , 20 The inpatient facility to which the healthcare system admitted its patients used a different EHR than that used by the healthcare system. An interface engine was linked to the hospital’s admission, discharge, transfer (ADT) registration system. The hospital transmitted information including discharge dates for the healthcare system’s patients; these were automatically incorporated into the healthcare system’s EHR (information on medication prescriptions was not transferred).

Informed by health plan data reflecting new prescriptions filled, alerts were then generated for patients who had (a) new drugs added upon hospital discharge and (b) a medication concern identified. Concerns included warnings about selected drug–drug interactions, recommendations for dose changes (e.g., change in digoxin dose due to rise in creatinine), need for laboratory monitoring for high-risk medications, and notification of new medications at discharge for which PCP awareness was clinically important. Alerts were designed to convey actionable time-sensitive concerns; all had the heading: “[Hospital Name] Discharge Alert.” Over the 4 months prior to implementation, two physicians from the healthcare system reviewed every alert generated, suggesting modifications to ensure that alerts would be perceived as necessary, useful, and brief. Upon implementation, alerts were automatically triggered on day 3 following hospital discharge and were sent to the Test Results Folder within PCP EHR InBaskets, arriving at 10:00 a.m. The reason for the 3-day delay was to allow information on newly filled prescriptions (from health plan data) to be captured, allowing new medications to be identified and alerts to be generated.

Dependent Variables

EHR audit logs have been used in previous studies to track viewing of documentation in the medical record by members of the care team.21 23 We used audit and access logs to capture the time of alert opening. We calculated the percentage of alerts opened by PCPs within 24 h. We focus on 24 h for alert opening because this was an at-risk population (post-hospital discharge, aged ≥ 65 years) for whom new medication concerns had been identified that were deemed urgent (based on two-physician review during pre-implementation phase). We also performed a sensitivity analysis examining characteristics associated with 48-h opening. Since all alerts were delivered at 10:00 a.m. on the day of delivery, a 48-h window included standard weekday clinic hours for all alerts.

Independent Variables

We classified patients’ age (65–74; 75–84; ≥85 years) and gender. We used administrative data to examine patient-level variables including number of office visits in the previous 12 months (≤6, 7–11; 12–18 or >18 visits) and comorbid medical conditions. We created a modified Charlson comorbidity index score for patients (categorized as 0, 1, 2, and 3+) using data on medical diagnoses and procedures in the year prior to admission to the hospital,24 and gathered information on length of hospital stay by tertile (≤ 2, > 2 and ≤ 4, > 4).

PCP characteristics included in the analyses were age (< 50 vs. ≥ 50 years), gender, number of completed patient clinical encounters over the 1-year study period (expressed in quartiles), and specialty (internal medicine, family medicine, non-MD PCP such as nurse practitioner, and subspecialist acting as PCP).

Using data from audit logs and access logs, we identified the following variables and categorized each by quartile: total number of notifications (already opened notifications + unopened notifications) in the PCP’s InBasket at the time of alert delivery; number of unopened notifications in the PCP’s InBasket at the time of alert delivery; and number of notifications delivered to the PCPs InBasket in the 7 days prior to alert delivery. In addition, we analyzed the total number of notifications in the PCP’s InBasket as a continuous variable. We compared alerts delivered on Saturdays to those delivered on all other days, because those arriving on Saturday were the only alerts for which 24-h post-delivery did not include any weekday time. For exploratory purposes, we also compared (a) alerts delivered on Fridays and (b) alerts delivered on Mondays to those delivered on all other days. We collectively refer to the above variables as contextual variables.

We also captured timing of log into the EHR (logged in at time of alert delivery or within the 24 h of delivery vs. not logged in); alert opening by (a) a physician who was not the PCP and (b) a staff member (opened within 24 h vs. not); and opening of a follow-up notification by PCP or other physician (opened within 24 h vs. not). Follow-up notifications are created by staff and contain a copy of the alert contents.

Statistical Analysis

Descriptive Analysis

We performed descriptive analyses, calculating basic frequencies and percentages for patient and PCP characteristics, and calculating frequencies, medians, and interquartile ranges for all contextual variables. We then calculated time to PCP opening of alerts, using survival curves measuring time (in days) from the date of alert delivery (day 3 post-hospital discharge) to 30 days post-alert delivery. Survival curves were stratified for independent variables that demonstrated a statistically significant association with message opening in the multivariate analysis described below.

In an exploratory analysis of workflow, we calculated frequencies and percentages of alerts opened by (a) the intended PCP and (b) another physician within 24 h. We also examined frequencies and percentages of alerts opened within 24 h by a staff member, and for these, we tracked frequency and percentage of cases in which follow-up notifications were opened by PCPs or covering physicians within 24 h of alert delivery. We were unable to capture surrogate assignment of InBasket coverage, and therefore could not estimate the percentage of alerts delivered on days when a covering clinician was assigned.

Bivariate Analyses – All PCPs

We used generalized estimating equations (GEE) with a logit link and a binomial distribution to estimate the bivariate association between PCP characteristics, patient characteristics, contextual variables, and opening of the alerts within 24 h. The GEE models account for clustering of measures within PCPs, since some PCPs received multiple alerts over the 1-year period. In order to check for the existence of a linear trend in association with 24-h opening, we also examined the bivariate association between total number of InBasket notifications (using this variable as a continuous measure) and alert opening within 24 h.

Multivariate Analyses (and Sensitivity Analyses) – All PCPs

We performed multivariate analyses using GEE equations as described above, studying the association between covariates of interest and PCP alert opening within 24 h. Covariates included in the multivariate models encompassed all PCP characteristics, patient characteristics, and contextual variables. All PCPs were included in this analysis. We performed a sensitivity analysis, studying the bivariate and multivariate associations between covariates of interest and PCP alert opening within 48 h.

Multivariate Analyses – PCPs Logged in at Time of Alert Delivery

We also constructed multivariate GEE models using the approach described above, but limiting our population to PCPs who were logged in at the time of alert delivery or who logged in within the following 24 h.

Analyses were carried out using the SAS package version 9.3 (SAS Institute, Cary, NC, USA) and STATA version 13.1 (StataCorp LP, College Station, Texas, USA).

Physician Focus Group

We conducted a single physician focus group intended to inform our research regarding factors influencing InBasket management. Email invitations were sent to 100 PCPs in the healthcare system, and five respondents were recruited. A one-hour session (audio recorded and professionally transcribed) addressed strategies for managing EHR InBaskets and factors influencing speed of notification opening including a discussion of the subject header for the alerts in this study. The transcript was reviewed for major themes by two authors (LB, SC).

RESULTS

Descriptive Analysis

The 75 PCPs received 799 automated alerts over the 1-year period. The median number of alerts per PCP over the entire 1-year period was nine, with an interquartile range (IQR) of 3–16.

To place these alerts in context, the median number of total notifications in PCP InBaskets at the time of alert delivery was 68.7 (IQR 41.3, 156.3). The median number of notifications arriving in PCP InBaskets during the 7 days prior to alert delivery was 379.8 (IQR 295.0, 492.0).

Characteristics of patients for whom alerts were triggered and PCPs to whom the alerts were sent are described in Table 1. The distribution of alerts by PCP and patient characteristics and for contextual variables is shown in Table 2. Overall, 47.1% (n = 376) of alerts were opened by the intended PCP within 24 h (Table 2), 59.5% were opened within 48 h, and 77.2% were opened at the end of 30 days (not shown in table).
Table 1

Patient and Primary Care Provider (PCP) Characteristics

Characteristics of Participating Patients

N (%)

Age

 65–74 years

224 (31.42)

 75–84 years

315 (44.18)

 ≥85 years

174 (24.40)

Gender

 Female

371 (52.03)

 Male

342 (47.97)

Number of office visits w/in prior 12 months

 ≤6 visits

195 (27.35)

 >6 and ≤11 visits

201 (28.19)

 >11 and ≤18 visits

173 (24.26)

 >18 visits

144 (20.20)

Charlson comorbidity score

 0

72 (10.10)

 1

88 (12.34)

 2

106 (14.87)

 3+

447 (62.69)

Length of hospital stay

 ≤2 days

324 (45.44)

 >2 and ≤4

251 (35.20)

 >4 days

138 (19.35)

Characteristics of participating PCPs

N (%)

Age

 <50 years

38 (50.67)

 ≥50 years

37 (49.33)

Gender

 Female

36 (48.00)

 Male

39 (52.00)

Number of clinical encounters over 1-year study period

 >0 and ≤2326

19 (25.33)

 >2326 and ≤2783

19 (25.33)

 >2783 and ≤3173

19 (25.33)

 >3173

18 (24.00)

Specialty

 Family medicine

16 (21.33)

 Internal medicine

51 (68.00)

 Non-MD PCP

6 (8.00)

 Sub-specialty

2 (2.67)

Table 2

Alert Opening among PCPs Receiving Time-Sensitive Alerts Delivered to Their EHR InBaskets

 

Total number of EHR InBasket alerts

Total number (%) of alerts opened by PCP within 24 h

Crude odds ratios (95% CI)*

Adjusted odds ratios (95% CI)

 

799

376 (47.06)

  

PCP Characteristics

Age

 <50 years

296

137 (46.28)

Ref

Ref

 ≥50 years

503

239 (47.51)

0.73 (0.39–1.37)

0.71 (0.33–1.52)

Gender

 Female

264

99 (37.50)

Ref

Ref

 Male

535

277 (51.78)

1.46 (0.8–2.66)

1.28 (0.59–2.77)

Number of patient encounters in study year (quartiles)

 >0 and ≤2326

80

25 (31.25)

Ref

Ref

 >2326 and ≤2783

163

63 (38.65)

1.03 (0.43–2.45)

1.58 (0.52–4.79)

 >2783 and ≤3173

247

104 (42.11)

1.33 (0.58–3.06)

1.99 (0.76–5.18)

 >3173

309

184 (59.55)

1.98 (0.80–4.90)

2.90 (0.89–9.42)

Specialty

 Internal medicine

661

317 (47.96)

Ref

Ref

 Family medicine

118

54 (45.76)

0.82 (0.41–1.66)

0.57 (0.27–1.19)

 Non-MD PCP

11

3 (27.27)

0.47 (0.13–1.61)

0.3 (0.06–1.49)

 Sub-specialty

9

2 (22.22)

0.47 (0.14–1.58)

0.55 (0.19–1.63)

Patient characteristics

Age

 65–74 years

255

133 (52.16)

Ref

Ref

 75–84 years

349

150 (42.98)

0.76 (0.56–1.04)

0.77 (0.55–1.08)

 ≥85 years

195

93 (47.69)

1.00 (0.70–1.43)

0.96 (0.65–1.39)

Gender

 Female

418

182 (43.54)

Ref

Ref

 Male

381

194 (50.92)

1.16 (0.86–1.56)

1.19 (0.82–1.72)

Number of office visits w/in prior 12 months

 ≤6 visits

205

95 (46.34)

Ref

Ref

 >6 and ≤11 visits

219

101 (46.12)

0.92 (0.7–1.22)

0.96 (0.7–1.3)

 >11 and ≤18 visits

194

94 (48.45)

1.07 (0.8–1.44)

1.26 (0.87–1.81)

 >18 visits

181

86 (47.51)

0.9 (0.65–1.24)

1.03 (0.69–1.54)

Charlson comorbidity score

 0

78

31 (39.74)

Ref

Ref

 1

92

49 (53.26)

1.55 (0.94–2.54)

1.79 (1.02–3.14)

 2

118

56 (47.46)

1.31 (0.78–2.18)

1.56 (0.88–2.76)

 3+

511

240 (46.97)

1.14 (0.74–1.74)

1.04 (0.62–1.77)

Length of hospital stay (tertiles)

 ≤2 days

360

167 (46.39)

Ref

Ref

 >2 and ≤4

281

136 (48.40)

1.09 (0.84–1.43)

1.05 (0.77–1.44)

 >4

158

73 (46.20)

1.19 (0.88–1.59)

1.48 (1.00–2.19)

Contextual factors

Total number of notifications (opened + unopened) in InBasket at time of alert delivery

 ≤42.0

207

128 (61.84)

Ref

Ref

 >42.0 and ≤69.0

194

111 (57.22)

0.77 (0.55–1.10)

0.69 (0.45–1.06)

 >69.0 and ≤157.0

199

81 (40.70)

0.56 (0.34–0.92)

0.39 (0.21–0.75)

 >157.0

199

56 (28.14)

0.38 (0.22–0.65)

0.27 (0.14–0.51)

Number of unopened notifications in InBasket at time of alert delivery

 ≤0

251

172 (68.53)

Ref

Ref

 >0 and ≤4.0

183

81 (44.26)

0.57 (0.37–0.87)

0.75 (0.46–1.21)

 >4.0 and ≤9.0

185

67 (36.22)

0.52 (0.31–0.88)

0.87 (0.48–1.58)

 >9.0

180

56 (31.11)

0.50 (0.31–0.82)

0.89 (0.52–1.53)

Notification count in 1 week prior to alert delivery

 ≤344

200

86 (43.00)

Ref

Ref

 >344 and ≤453

201

88 (43.78)

0.96 (0.59–1.56)

0.98 (0.59–1.62)

 >453 and ≤546

199

107 (53.77)

1.19 (0.81–1.76)

1.11 (0.69–1.77)

 >546

199

95 (47.74)

1.07 (0.62–1.85)

1.14 (0.61–2.15)

Day of the week alert sent

 All other days

633

345 (54.50)

Ref

Ref

 Saturday

166

31 (18.67)

0.2 (0.10–0.41)

0.16 (0.08–0.33)

*Crude odds ratios account for clustering by PCP, using a GEE model

Adjusted odds ratios account for clustering by PCP using a GEE model and control for PCP characteristics, patient characteristics and contextual factors

Figure 1 shows the proportion of alerts opened by the PCP over the first 30 days post-alert delivery, stratified by (a) day of the week on which the alert was delivered and (b) total number of InBasket notifications (by quartile).
Figure 1

Proportion of alerts remaining unopened by PCPs* over 30 days, stratified by day of alert delivery (top) and by total number of InBasket notifications at time of alert delivery (bottom), by quartile. Total notifications = already opened notifications + unopened notifications.

Figure 2 shows workflow patterns of alert opening by healthcare team members including the PCP, covering clinicians and other staff. Accounting for these additional care team members did not change our findings substantially; in the first 24 h post-delivery, slightly more than half of alerts (52.7%) contained information that reached a physician, while 41.1% were unopened by either clinician or staff.
Figure 2

Patterns of alert opening within first 24 h post-alert delivery into electronic health record InBasket. * Primary care providers.

Bivariate Analyses – All PCPs

Accounting for clustering by PCP (Table 2), there were no significant bivariate associations between alert opening at 24 h and either PCP characteristics or patient characteristics. Alerts delivered to PCPs with crowded InBaskets (i.e., for whom the total number of InBasket notifications was in the top quartile) were significantly less likely to be opened within 24 h than alerts delivered to PCPs with less crowded InBaskets (i.e., whose total number of InBasket notifications was in the bottom quartile). A similar pattern was seen with the number of unopened InBasket notifications. In the first 24 h post-delivery, alerts delivered on Saturdays were less likely to be opened (Table 2), while alerts delivered on Mondays were more likely (OR 1.56, 95% CI 1.06–2.29). Analyses separating alerts delivered on Friday from all others showed no significant difference in opening within 24 h (OR 1.19, 95% CI 0.78–1.80). We did not detect a significant association between time of alert opening and number of notifications received in the prior 7 days. On analysis using total number of InBasket notifications as a continuous variable, we found that for each additional notification in a PCP InBasket, there was a small but statistically significant decrease in the likelihood of alert opening within 24 h (OR 0.50 (95% CI 0.50, 0.50).

Multivariate Analyses (and Sensitivity Analyses) – All PCPs

Multivariate analyses (Table 2) showed associations similar to those found on bivariate analysis; however, patients with the longest hospital stay (>4 days) were slightly more likely (OR 1.48, 95% CI 1.00–2.19) to have alerts opened within 24 h. There was no longer a significant association between 24-h alert opening and number of unopened messages in the InBasket at alert delivery.

In our sensitivity analysis, bivariate and multivariate analyses of alert opening within 48 h showed similar associations, with two minor differences. Hospital stays greater than 4 days showed no significant association (OR 1.07, 95% CI 0.73–1.56; reference group is length of stay ≤2 days). PCPs whose number of patient encounters (for the study year) fell into the top quartile were more likely (OR 3.29, 95% CI 1.01–10.75) to open alerts within 48 h than those with the fewest number of patient encounters.

Multivariate Analyses – PCPs Logged in at Time of Alert Delivery

For alerts sent to PCPs who were logged in at the time of or within 24 h of alert delivery (Table 3), 375 (59.5%) were opened within 24 h, and the day of the week on which alerts were delivered was no longer significant. Alerts sent for patients aged 75–84 were significantly less likely to be opened than alerts for patients under 75. Patients with a Charlson comorbidity score of 1 were significantly more likely to have their alerts opened than those with a score of zero, but there was no apparent trend by comorbidity score.
Table 3

Alert Opening within 24 Hours among PCPs Who Were Logged in at Time of Alert Delivery*

 

Adjusted odds ratios controlling for PCP and patient characteristics and for contextual factors (95% confidence interval)

PCP Characteristics

Age

 <50 years

Ref

 ≥50 years

0.73 (0.31–1.72)

Sex

 Female

Ref

 Male

1.35 (0.57–3.2)

Number of patient encounters in study year (quartiles)

 >0 and ≤2326

Ref

 >2326 and ≤2783

1.19 (0.37–3.80)

 >2783 and ≤3173

1.56 (0.55–4.46)

 >3173

3.24 (0.85–12.41)

Specialty

 Internal medicine

Ref

 Family medicine

0.53 (0.23–1.24)

 Non-MD PCP

0.32 (0.07–1.46)

 Sub-Specialty

0.84 (0.16–4.3)

Patient Characteristics

Age

 65–74 years

Ref

 75–84 years

0.7 (0.5–0.98)

 ≥85 years

0.74 (0.5–1.1)

Gender

 Female

Ref

 Male

1.05 (0.72–1.55)

Number of office visits within prior 12 months

 ≤6 visits

Ref

 7–11 visits

1.02 (0.75–1.38)

 12–18 visits

1.41 (0.95–2.07)

 >18 visits

1.09 (0.69–1.73)

Charlson comorbidity score (0, 1, 2, 3+)

 0

Ref

 1

1.99 (1.1–3.6)

 2

1.71 (0.98–2.99)

 3+

1.07 (0.62–1.84)

Length of hospital stay

 ≤2 days

Ref

 >2 and ≤4

1.12 (0.78–1.62)

 >4

1.59 (1.07–2.35)

Contextual Factors

Total number of notifications (opened + unopened) in InBasket at time of alert delivery

 ≤42

Ref

 43–69

0.51 (0.34–0.76)

 70–157

0.31 (0.17–0.57)

 >157

0.25 (0.13–0.5)

Number of unopened notifications in InBasket at time of alert delivery

 0

Ref

 1–4

0.83 (0.5–1.38)

 5–9

0.88 (0.5–1.55)

 >9

0.77 (0.41–1.42)

Number of notifications received in the week prior to alert delivery

 ≤344

Ref

 345–453

0.85 (0.49–1.47)

 454–546

0.9 (0.55–1.47)

 >546

0.68 (0.36–1.28)

Day of the week alert sent

 All other days

Ref

 Saturday

0.59 (0.35–1.01)

*A total of 630 alerts were delivered to the InBaskets of PCPs who were logged into the EHR at the time of alert delivery or within 24 h of delivery; of these, 375 were opened within 24 h of delivery

Adjusted odds ratios account for clustering by PCP using a GEE model and control for PCP characteristics, patient characteristics and contextual factors

Physician Focus Group

Our focus group included three men and two women from several primary care specialties (pediatrics, family medicine, internal medicine), each from a different site within the healthcare system. In response to interview questions describing study alerts and their subject headings (“[Hospital name] Discharge Alert”), one respondent replied:

So I find out four different ways…[t]hat they’ve been discharged. So I am getting the discharge summary, I am getting voicemail from the hospitalist…my nurse sends me a message…my case manager sends me a message the patient needs to be seen in 7 days. So I am bombarded with that one discharge by five different ways

Themes that emerged from the general discussion about InBasket notifications included receipt of redundant and un-actionable information (“60% of them do not need to come to Primary Care”) and the need for better prioritization. The categories “addendum” and “cc” were cited as containing numerous low-priority items; anxiety over associated risks was discussed: “…you might get that 117th ‘cc' and God forbid it’s one of those ones that has to be done because it was a missed mammogram,” and “you get the same thing being done or replicated by so many different places … there’s so much data that you can make mistakes.”

Themes identified included burden of InBasket notifications (“it takes your whole life over,” “I am constantly on a treadmill.”). Physicians discussed the number of notifications (“Too full. Too many messages”) and the necessity of reviewing notifications after hours, including evenings (“So you go home and do it at night sometimes because you can’t finish it during the day…”), weekends (“If I don’t go on like a Sunday night…I might have 37 labs there on Monday morning, and you know what? I have to see patients…”), and vacations (“When you’re on vacation, somehow you get into your Epic Box every day; otherwise you are dead when you come back.”)

DISCUSSION

In this study of time-sensitive alerts sent to PCP EHR InBaskets, more than half of the alerts remained unopened by the PCP at 24 h, more than one-third were still unopened after 48 h, and more than one-fifth remained unopened at 30 days. Of those opened at 48 h, 79.2% were opened within the first 24, supporting our focus on characteristics associated with 24-h opening. With the exception of a marginally significant association for those patients with the longest hospital stay, patient characteristics were not associated with likelihood of alert opening at 24 h. Alerts triggered by our most complex patients and by our oldest patients were no more likely to experience prompt review than were those for less complex and younger patients.

PCP characteristics were not significantly associated with timeliness of alert opening. In previous studies, physician characteristics associated with alert opening included trainee status, familiarity with the use of EHR filters, and clarity regarding coverage roles.7 , 16 All PCPs in our study had completed training; our commercial EHR (in contrast to the VHA system) does not allow healthcare teams to adjust InBasket filters, and few alerts were reviewed by covering clinicians.

Perhaps our most important finding relates to contextual factors. PCPs with more notifications in the InBasket at the time of alert delivery were less likely to complete alert opening within 24 h, even if logged in during that period. This relationship was significant when measuring total InBasket notifications (opened + unopened) both as a continuous and as a categorical variable. As highlighted by Murphy et al.,4 a single notification often contains multiple data points. Items in a PCP InBasket may demand significant cognitive effort to appropriately process2 and may also require input and care coordination efforts from other care team members. Management techniques that involve rapid opening and scanning of InBasket contents, with eventual task completion, may be used by PCPs in an attempt to avoid missing urgent information. However, larger numbers of total InBasket notifications were associated with delayed review of urgent alerts, suggesting that this strategy alone may not be sufficient. We did not demonstrate a significant association between number of unopened notifications and alert opening within 24 h. Perceived work burden may be disproportionately influenced by opened notifications that have not yet been completely processed (still competing for attention) compared to unopened notifications. Alternatively, high volumes of opened (but not processed) notifications may be a marker for PCP behavior (e.g., greater time spent thinking through each notification) associated with delays in opening.

Physicians already spend a disproportionate amount of time attending to the EHR; for every hour of direct clinical face time spent in an ambulatory setting, almost 2 clinician hours are spent on EHR tasks.25 Efforts to promote timely review of urgent notifications must therefore include system-level changes, not just feedback or punitive measures for clinicians. Guidelines have recently been issued for notification management within the VHA,26 but to date no comprehensive guidelines exist for commercial EHRs. Particularly in the VHA, where some filters and management tools are available, larger total numbers of InBasket notifications (in comparison to peers with similar panels) could indicate less effective use of available management functions. Targets for improvement relevant for both VHA and non-VHA settings include physician and staff training to improve familiarity with any available EHR InBasket management tools or strategies. Informed by our focus group findings, we would suggest that modification of notification triage policies (e.g., increasing use of staff to offload PCPs), reduction in the creation of redundant or un-actionable messages, and improved labeling of high-priority notifications are other recommended approaches.26 , 27 Systems could also be developed that highlight or escalate unopened alerts after a designated period, converting them to an interruptive mode or escalating to supervisors.

Previous studies indicate that characteristics of alerts may influence timely opening. These include whether they are sent to one person or many7 and whether they address a known vs. new diagnosis.8 The type of report7 and the manner in which the alert appears in the InBasket are also influential.7 Institution-wide or clinic-specific policies on InBasket management are other potential factors.16 Our study of 24-h opening by covering clinicians and staff (Fig. 2) showed that alerts intercepted by staff or coverage did not substantially alter overall patterns of alert opening. In our study, alerts contained time-sensitive material and were designed by study team members (practicing physicians in the healthcare system) to be delivered to a high-priority InBasket folder; however, focus group findings support the possibility that subject headings (uniformly referencing hospital discharge) may not have been interpreted as urgent by receiving PCPs, who (in this medical practice) receive updates on patient discharges through a number of channels.

We found that PCPs receiving alerts on a Saturday had a lower likelihood of opening within 24 h. While this is not surprising, we were concerned to find that it took approximately 3 days for the proportion of Saturday-delivered opened alerts to catch up with the proportion of opened alerts delivered on all other days. This finding should prompt careful consideration on the part of both clinical and research teams seeking to send urgent communications over the weekend.

Our study was innovative in its use of EHR audit logs to gain insight into daily processes of care. Designed to monitor user action within the EHR, logs offer a rich source of information on healthcare team behavior. Review of audit logs provides information on clinician behavior and on interactions between team members.22 , 28 Previous studies have used audit logs to describe physician and care team behaviors including medical information queries,29 31 team communication,32 viewing of EHR notes,23 response to clinical decision support tools,33 and patient interactions through tethered portals.34

This study has several limitations. The extent to which our results are generalizable to other health systems is hard to estimate. However, it was conducted in a healthcare setting that uses Epic, an EHR in widespread use across the United States. We may not have captured all relevant variables that explain alert opening, which may lead to residual confounding. While our study does describe alert opening, we are unable to determine whether alerts were actually read.

We conclude that the number of total InBasket notifications and weekend delivery may impact opening of time-sensitive EHR alerts. In a high-risk older population, time-sensitive alerts were not being opened within 24 h, possibly because physicians were simply overwhelmed with high numbers of InBasket notifications. Further study is needed to support safe and effective approaches for care team management of InBasket notifications.

Notes

Compliance with Ethical Standards

Funders

This study was funded by a grant from the Agency for Healthcare Research and Quality under award number 5R21HS023661. The content is solely the responsibility of the authors and does not necessarily represent the official views of the NIH.

Conflict of Interest

Drs. Cutrona and Mazor have received research funding from Pfizer Independent Grants for Learning and Change for a study in the area of vaccines from 2014 to 2016. Dr. Mazor is currently involved in two studies, one concerning rheumatoid arthritis (2014–2017) and another concerning cardiovascular risk (2015–2017), also funded by Pfizer Independent Grants for Learning and Change. All other authors declare that they do not have a conflict of interest.

References

  1. 1.
    Cook KE, Ludens GM, Ghosh AK, Mundell WC, Fleming KC, Majka AJ. Improving efficiency and reducing administrative burden through electronic communication. Perm J. 2013;17(1):26-30. doi: 10.7812/TPP/12-010 CrossRefPubMedPubMedCentralGoogle Scholar
  2. 2.
    Hysong SJ, Sawhney MK, Wilson L, et al. Provider management strategies of abnormal test result alerts: a cognitive task analysis. J Am Med Inform Assoc: JAMIA. 2010;17(1):71-7.CrossRefPubMedPubMedCentralGoogle Scholar
  3. 3.
    Hysong SJ, Sawhney MK, Wilson L, et al. Understanding the management of electronic test result notifications in the outpatient setting. BMC Med Inform Decis Mak. 2011;11:22. doi: 10.1186/1472-6947-11-22 CrossRefPubMedPubMedCentralGoogle Scholar
  4. 4.
    Murphy DR, Meyer AN, Russo E, Sittig DF, Wei L, Singh H. The Burden of Inbox Notifications in Commercial Electronic Health Records. JAMA Intern Med. 2016;176(4):559-60. doi: 10.1001/jamainternmed.2016.0209 CrossRefPubMedPubMedCentralGoogle Scholar
  5. 5.
    Poon EG, Wang SJ, Gandhi TK, Bates DW, Kuperman GJ. Design and implementation of a comprehensive outpatient Results Manager. J Biomed Inform. 2003;36(1–2):80-91.CrossRefPubMedGoogle Scholar
  6. 6.
    Singh H, Spitzmueller C, Petersen NJ, et al. Primary care practitioners' views on test result management in EHR-enabled health systems: a national survey. J Am Med Inform Assoc: JAMIA. 2013;20(4):727-35.CrossRefPubMedGoogle Scholar
  7. 7.
    Singh H, Thomas EJ, Mani S, et al. Timely follow-up of abnormal diagnostic imaging test results in an outpatient setting: are electronic medical records achieving their potential? Arch Intern Med. 2009;169(17):1578-86. doi: 10.1001/archinternmed.2009.263 CrossRefPubMedPubMedCentralGoogle Scholar
  8. 8.
    Singh H, Thomas EJ, Sittig DF, et al. Notification of abnormal lab test results in an electronic medical record: do any safety concerns remain? Am J Med. 2010;123(3):238-44. doi: 10.1016/j.amjmed.2009.07.027 CrossRefPubMedPubMedCentralGoogle Scholar
  9. 9.
    Sittig DF, Singh H. Improving test result follow-up through electronic health records requires more than just an alert. J Gen Intern Med. 2012;27(10):1235-7. doi: 10.1007/s11606-012-2161-y CrossRefPubMedPubMedCentralGoogle Scholar
  10. 10.
    Smith M, Murphy D, Laxmisan A, et al. Developing software to "track and catch" missed follow-up of abnormal test results in a complex sociotechnical environment. Appl Clin Inform. 2013;4(3):359-75. doi: 10.4338/ACI-2013-04-RA-0019 CrossRefPubMedPubMedCentralGoogle Scholar
  11. 11.
    Wahls T. Diagnostic errors and abnormal diagnostic tests lost to follow-up: a source of needless waste and delay to treatment. J Ambul Care Manag. 2007;30(4):338-43. doi: 10.1097/01.JAC.0000290402.89284.a9 CrossRefGoogle Scholar
  12. 12.
    Wahls TL, Cram PM. The frequency of missed test results and associated treatment delays in a highly computerized health system. BMC Fam Pract. 2007;8:32. doi: 10.1186/1471-2296-8-32 CrossRefPubMedPubMedCentralGoogle Scholar
  13. 13.
    Coiera E, Tombs V. Communication behaviours in a hospital setting: an observational study. BMJ. 1998;316(7132):673-6.CrossRefPubMedPubMedCentralGoogle Scholar
  14. 14.
    Gurwitz JH, Field TS, Ogarek J, et al. An electronic health record-based intervention to increase follow-up office visits and decrease rehospitalization in older adults. J Am Geriatr Soc. 2014;62(5):865-71.CrossRefPubMedPubMedCentralGoogle Scholar
  15. 15.
    Murphy DR, Reis B, Sittig DF, Singh H. Notifications received by primary care practitioners in electronic health records: a taxonomy and time analysis. Am J Med. 2012;125(2):029.CrossRefGoogle Scholar
  16. 16.
    Menon S, Smith MW, Sittig DF, et al. How context affects electronic health record-based test result follow-up: a mixed-methods evaluation. BMJ Open. 2014;4(11):e005985. doi: 10.1136/bmjopen-2014-005985 CrossRefPubMedPubMedCentralGoogle Scholar
  17. 17.
    Singh H, Spitzmueller C, Petersen NJ, Sawhney MK, Sittig DF. Information overload and missed test results in electronic health record-based settings. JAMA Intern Med. 2013;173(8):702-4. doi: 10.1001/2013.jamainternmed.61 CrossRefPubMedGoogle Scholar
  18. 18.
    Field TS, Garber L, Gagne SJ, et al. Technological resources and personnel costs required to implement an automated alert system for ambulatory physicians when patients are discharged from hospitals to home. Inform Prim Care. 2012;20(2):87-93.PubMedGoogle Scholar
  19. 19.
    Kanaan AO, Donovan JL, Duchin NP, et al. Adverse drug events after hospital discharge in older adults: types, severity, and involvement of Beers Criteria Medications. J Am Geriatr Soc. 2013;61(11):1894-9. doi: 10.1111/jgs.12504 CrossRefPubMedPubMedCentralGoogle Scholar
  20. 20.
    Tjia J, Field TS, Garber LD, et al. Development and pilot testing of guidelines to monitor high-risk medications in the ambulatory setting. Am J Manag Care. 2010;16(7):489-96.PubMedGoogle Scholar
  21. 21.
    Boxwala AA, Kim J, Grillo JM, Ohno-Machado L. Using statistical and machine learning to help institutions detect suspicious access to electronic health records. J Am Med Inform Assoc: JAMIA. 2011;18(4):498-505. doi: 10.1136/amiajnl-2011-000217 CrossRefPubMedPubMedCentralGoogle Scholar
  22. 22.
    Hripcsak G, Vawdrey DK, Fred MR, Bostwick SB. Use of electronic clinical documentation: time spent and team interactions. J Am Med Inform Assoc: JAMIA. 2011;18(2):112-7. doi: 10.1136/jamia.2010.008441 CrossRefPubMedPubMedCentralGoogle Scholar
  23. 23.
    Malin B, Nyemba S, Paulett J. Learning relational policies from electronic health record access logs. J Biomed Inform. 2011;44(2):333-42. doi: 10.1016/j.jbi.2011.01.007 CrossRefPubMedPubMedCentralGoogle Scholar
  24. 24.
    Deyo RA, Cherkin DC, Ciol MA. Adapting a clinical comorbidity index for use with ICD-9-CM administrative databases. J Clin Epidemiol. 1992;45(6):613-9.CrossRefPubMedGoogle Scholar
  25. 25.
    Sinsky C, Colligan L, Li L, et al. Allocation of Physician Time in Ambulatory Practice: A Time and Motion Study in 4 Specialties. Ann Intern Med. 2016. doi: 10.7326/m16-0961
  26. 26.
    Singh HA. A Checklist to Improve CPRS 'View Alert' Notifications. Recommendations for VA Facility Leadership and Staff. In: Affairs USDoV, editor. 2016.Google Scholar
  27. 27.
    Murphy DR, Reis B, Kadiyala H, et al. Electronic health record-based messages to primary care providers: valuable information or just noise? Arch Intern Med. 2012;172(3):283-5. doi: 10.1001/archinternmed.2011.740 CrossRefPubMedGoogle Scholar
  28. 28.
    Gray JE, Feldman H, Reti S, et al. Using Digital Crumbs from an Electronic Health Record to identify, study and improve health care teams. AMIA ... Annual Symposium proceedings / AMIA Symposium. AMIA Symp. 2011 :22.Google Scholar
  29. 29.
    Chen ES, Cimino JJ. Automated discovery of patient-specific clinician information needs using clinical information system log files. AMIA ... Annual Symposium proceedings / AMIA Symposium. AMIA Symp. 2003 :145–9.Google Scholar
  30. 30.
    Natarajan K, Stein D, Jain S, Elhadad N. An analysis of clinical queries in an electronic health record search utility. Int J Med Inform. 2010;79(7):515-22. doi: 10.1016/j.ijmedinf.2010.03.004 CrossRefPubMedPubMedCentralGoogle Scholar
  31. 31.
    Yang L, Mei Q, Zheng K, Hanauer DA. Query log analysis of an electronic health record search engine. AMIA ... Annual Symposium proceedings / AMIA Symposium. AMIA Symp. 2011;2011:915-24.Google Scholar
  32. 32.
    Tai-Seale M, Wilson CJ, Panattoni L, et al. Leveraging electronic health records to develop measurements for processes of care. Health Serv Res. 2014;49(2):628-44. doi: 10.1111/1475-6773.12126 CrossRefPubMedGoogle Scholar
  33. 33.
    Feblowitz J, Henkin S, Pang J, et al. Provider use of and attitudes towards an active clinical alert: a case study in decision support. Appl Clin Inform. 2013;4(1):144-52. doi: 10.4338/aci-2012-12-ra-0055 CrossRefPubMedPubMedCentralGoogle Scholar
  34. 34.
    Cutrona SL, Sreedhara M, Goff SL, et al. Improving Rates of Influenza Vaccination Through Electronic Health Record Portal Messages, Interactive Voice Recognition Calls and Patient-Enabled Electronic Health Record Updates: Protocol for a Randomized Controlled Trial. JMIR Res Protocol. 2016;5(2):e56. doi: 10.2196/resprot.5478 CrossRefGoogle Scholar

Copyright information

© Society of General Internal Medicine (outside the USA) 2017

Authors and Affiliations

  • Sarah L. Cutrona
    • 1
    • 2
    • 3
    • 4
    Email author
  • Hassan Fouayzi
    • 3
    • 4
  • Laura Burns
    • 3
  • Rajani S. Sadasivam
    • 4
  • Kathleen M. Mazor
    • 3
    • 4
  • Jerry H. Gurwitz
    • 3
    • 4
    • 5
  • Lawrence Garber
    • 3
    • 5
  • Devi Sundaresan
    • 3
    • 5
  • Thomas K. Houston
    • 1
    • 4
  • Terry S. Field
    • 3
    • 4
  1. 1.Center for Healthcare Organization and Implementation ResearchEdith Nourse Rogers Memorial Veterans HospitalBedfordUSA
  2. 2.Quantitative Health Sciences and Medicine, Division of Health Informatics and Implementation ScienceUniversity of Massachusetts Medical SchoolWorcesterUSA
  3. 3.Meyers Primary Care Institute, a joint endeavor of University of Massachusetts Medical School, Reliant Medical Group, and Fallon Health WorcesterWorcesterUSA
  4. 4.University of Massachusetts Medical SchoolWorcesterUSA
  5. 5.Reliant Medical GroupWorcesterUSA

Personalised recommendations