The Impact of Automated Notification on Follow-up of Actionable Tests Pending at Discharge: a Cluster-Randomized Controlled Trial

Abstract

Background

Follow-up of tests pending at discharge (TPADs) is poor. We previously demonstrated a twofold increase in awareness of any TPAD by attendings and primary care physicians (PCPs) using an automated email intervention

Objective

To determine whether automated notification improves documented follow-up for actionable TPADs

Design

Cluster-randomized controlled trial

Subjects

Attendings and PCPs caring for adult patients discharged from general medicine and cardiology services with at least one actionable TPAD between June 2011 and May 2012

Intervention

An automated system that notifies discharging attendings and network PCPs of finalized TPADs by email

Main Measures

The primary outcome was the proportion of actionable TPADs with documented action determined by independent physician review of the electronic health record (EHR). Secondary outcomes included documented acknowledgment, 30-day readmissions, and adjusted median days to documented follow-up.

Key Results

Of the 3378 TPADs sampled, 253 (7.5%) were determined to be actionable by physician review. Of these, 150 (123 patients discharged by 53 attendings) and 103 (90 patients discharged by 44 attendings) were assigned to intervention and usual care groups, respectively, and underwent chart review. The proportion of actionable TPADs with documented action was 60.7 vs. 56.3% (p = 0.82) in the intervention vs. usual care groups, similar for documented acknowledgment. The proportion of patients with actionable TPADs readmitted within 30 days was 22.8 vs. 31.1% in the intervention vs. usual care groups (p = 0.24). The adjusted median days [95% CI] to documented action was 9 [6.2, 11.8] vs. 14 [10.2, 17.8] (p = 0.04) in the intervention vs. usual care groups, similar for documented acknowledgment. In sub-group analysis, the intervention had greater impact on documented action for patients with network PCPs compared with usual care (70 vs. 50%, p = 0.03).

Conclusions

Automated notification of actionable TPADs shortened time to action but did not significantly improve documented follow-up, except for network-affiliated patients. The high proportion of actionable TPADs without any documented follow-up (~ 40%) represents an ongoing safety concern.

Clinical Trials Identifier

NCT01153451

BACKGROUND

More than half of preventable adverse events after discharge are related to poor communication.1,2 In prior work, we determined that 41% of patients left the hospital before all test results were finalized. Of the results of tests pending at discharge (TPADs), 9.4% were considered actionable by independent review.3 Although physicians had access to an integrated, electronic health record (EHR), they were unaware of 62% of these actionable TPADs.3

But awareness is only part of the problem: what matters clinically is whether appropriate follow-up occurs. Across institutions, the proportion of TPADs with documented follow-up is variable (20 to 62%), representing a substantial problem.4 Depending on the result, follow-up may include establishing a diagnosis, altering therapy, arranging an ambulatory appointment, initiating referral to a consultant, ordering additional tests, or communicating with patients.5,6,7 Although documented follow-up has been evaluated for certain actionable TPADs (e.g., microbiology),5,8 there are few data for other test types, services, clinicians, and EHRs. This research is difficult because clinicians may have different perspectives about whether specific TPADs are actionable depending on context, specialty, and role on the care team.9 Failure to follow-up actionable TPADs could lead to missed diagnoses, delayed treatments, unnecessary healthcare utilization, malpractice litigation, and preventable harm.4,10,11,12,13

Current vendor EHRs can be configured to notify clinicians of important results, such as TPADs, within an “InBasket”; however, time sensitive alerts are often missed and cognitively overloaded clinicians may not follow-up actionable data in a timely manner.14,15 Although clinicians may choose to receive notifications external to the EHR via institutional email or a “push” notification to a mobile app, few take advantage of this functionality. Furthermore, these notifications are typically generic, do not specify the actual result or criticality, do not classify TPADs as a special type of result, and do not provide a mechanism to communicate clinical context and transfer responsibility to the ambulatory clinician who will follow-up with the patient.

We previously designed, developed, and implemented an automated system that notified the responsible discharging attending and copied the primary care physician (PCP) of TPADs via secure, network email. Our system was unique not only because it sent email notifications external to our EHR (i.e., useful even if providers were not using the EHR at that moment), but also because it flagged TPADs as high risk, delineated responsibility for follow-up, and provided a simple mechanism (“reply all”) for attendings to communicate clinical context to ambulatory clinicians who could then assume responsibility.16 We demonstrated a twofold increase in self-reported awareness of any TPAD by attendings (76 vs. 38%) and PCPs (57 vs. 33%) assigned to the intervention group.17 In theory, improved awareness by the responsible discharge attending via this “active” notification should lead to appropriate follow-up, even for patients with non-network PCPs, but as noted above, this is by no means assured. The aim of this study was to evaluate the impact of our system on documented follow-up of actionable TPADs, defined as evidence of documented action or acknowledgment in our EHR by any network-affiliated clinician (attending, subspecialist, PCP). Understanding the benefit of automated notification systems for actionable TPADs on actual follow-up is crucial to resolving this safety concern.18,19,20

METHODS

Study Design, Setting, and Participants

The Partners Human Research Committee approved the study and granted a waiver of consent. We conducted a cluster randomized controlled trial at Brigham and Women’s Hospital, a 793-bed tertiary care hospital in Boston, MA, affiliated with Partners HealthCare, Inc., an integrated healthcare delivery network. Patients discharged from general medicine and cardiology services with at least one TPAD between June 2011 and May 2012 were assigned to intervention or usual care groups based upon the randomization status of their discharging attending and PCP.

Physicians caring for eligible patients included hospitalists, traditional internists, cardiologists, and other subspecialists attending on general medicine services, as well as PCPs. All Partners physicians could access the network EHR and enterprise email securely from a clinical workstation, personal computer, or encrypted mobile device. Non-network PCPs did not have access to Partners clinical information systems, but discharge summaries were faxed or mailed within 48 hours of discharge.

Intervention

The system automatically sent notifications to network clinicians’ institutional email addresses, and was configured to minimize alert fatigue. Each notification was flagged as an “Important Post-Discharge Test Result” in the subject heading, and the discharging attending was assigned responsibility in the body of the email. Network PCPs were carbon copied, thereby allowing the attending to communicate clinical context and transfer responsibility after discharge.16 Non-network PCPs were not notified by email; however, the attending could choose to communicate with non-network PCPs (via phone, letter, etc.) and document these actions in the EHR.

Randomization

The randomization procedures were previously described.17 Briefly, attendings and PCPs (network and non-network) were independently randomized, and patients were assigned to the intervention or usual care group if the attending and PCP were in concordant study arms. Patients of physicians randomized to discordant study arms were excluded to avoid contamination. Non-network patients were included in the intention-to-treat analysis, though non-network PCPs did not receive automated emails.

Identification of Actionable TPAD Results

We adapted an established algorithm to identify actionable TPADs.3,21 All normal, near-normal, and benign results were excluded. Two board-certified, internal medicine physicians (A-reviewers: AS, AD), blinded to randomization status, independently reviewed the discharge summary to determine whether the TPAD was listed, and to categorize the TPAD as definitely actionable, probably actionable, probably not actionable, or definitely not actionable according to a list of actions designated a priori (Box 1). A-reviewers adjudicated all discrepancies until consensus was reached. Thus, patients with TPADs who did not have at least one actionable TPAD identified by independent physician review were excluded.

Box 1 Types of actions taken by responsible providers after acknowledging an actionable TPAD

Determination of Documented Acknowledgement and Actions

Two different physicians (B-reviewers: EG, RP), blinded to randomization, independently reviewed the EHR to identify follow-up documentation for each actionable TPAD identified by A-reviewers. We defined follow-up as explicit documentation of acknowledgment or action (independent events) in the EHR’s note repository similarly to El-Kareh.5 B-reviewers used a natural language processing search tool, Queriable Patient Inference Dossier (QPID)—widely used at our institution to identify clinical information at the point-of-care—to retrieve relevant documentation.22 Specifically, B-reviewers queried the EHR for all documentation pertinent to the actionable TPAD (clinic and hospital notes, phone calls, medication lists, etc.) for up to 6 months after discharge. For patients with a high volume of documentation, reviewers used broad but specific search terms (e.g., A1c) to improve sensitivity when conducting queries while ensuring retrieval of the most pertinent subset of documentation related to the actionable TPAD.

After retrieving relevant documentation, B-reviewers determined whether the result was acknowledged; the name and specialty of the individual who acknowledged the result; the format of acknowledgment (phone call, hospital, or clinic note); and whether one or more actions (Box 1) were documented. B-reviewers resolved all discrepancies between themselves. Chart review was limited to our network’s EHR; out-of-network records of enrolled patients were not reviewed.

Outcomes

The primary outcome was the proportion of actionable TPADs with documented action. Secondary outcomes included the proportion of actionable TPADs with documented acknowledgment, patients with actionable TPADs readmitted within 30-days of hospitalization, and median days to documented follow-up. We quantified types of actions (Box 1) documented.

Sample Size

Based on El-Kareh, we expected the proportion of actionable TPADs with documented follow-up to increase from 13 to 28% when responsible physicians were notified.8 We assumed that patients had approximately one actionable TPAD (Roy et al. observed 1.08 actionable TPADs per patient).3 To achieve 80% power with an alpha of 0.05,23 we estimated that we would need to review records of 144 discharged patients with actionable TPADs to detect the effect size noted above (288 in both groups combined), accounting for clustering by attending (intra-class correlation coefficient of 0.03, cluster size of ten patients with an actionable TPAD per physician based on our prior study17) and a 50% reduction in sample size due to exclusion of patients cared for by physicians randomized to discordant study arms. Sample size calculations were conducted using NCSS PASS, version 12 (Kaysville, UT). We identified this cohort by randomly sampling TPADs from all available patient-discharges for up to 1 year after our prior study.17

Covariates

We collected patient demographics including age, gender, ethnicity, socioeconomic, and insurance status, Elixhauser comorbidity score,24 network affiliation, and length of stay. We collected physician demographics including age, gender, post-graduate year, specialty (general internist, hospitalist, cardiologist, other specialist), and number of years employed from hospital administrative databases, publicly available information (American Board of Internal Medicine), and practice managers.

Statistical Analysis

Patient and physician characteristics were described using means with standard deviations, medians with inter-quartile ranges, and proportions as appropriate. We analyzed the primary outcome as the proportion of actionable TPADs with documented action. We analyzed documented acknowledgment similarly. We analyzed hospital readmissions as the proportion of patients discharged with an actionable TPAD who were readmitted to a network-affiliated hospital within 30 days of index hospitalization. The effect of dichotomous variables on all outcomes was first analyzed using Fisher’s exact test. To measure days to documented follow-up, we calculated adjusted medians with 95% CIs using median regression.25 We then used multivariable logistic regression to analyze action, acknowledgment, and readmission adjusted for a priori selected covariates and general estimating equations to cluster by attending. In sub-group analyses, we used multivariable logistic regression with an interaction term (study arm*characteristic-type) to determine effect modification. Two-sided p values < 0.05 were considered significant. Analyses were conducted using SAS v9.4 (Cary, NC).

RESULTS

Of the 12,999 TPADs (Fig. 1) available during the study period, 3378 were sampled-based on our randomization schema. Of these, 2552 TPADs were excluded based on our screening protocol (normal, near-normal, or benign results). Of the remaining 826 TPADs, 253 (7.5% of all TPADs sampled) were determined to be actionable and underwent subsequent chart review. Of the 253 actionable TPADs, 150 (123 patients discharged by 53 attendings) and 103 (90 patients discharged by 44 attendings) were previously assigned to intervention and usual care groups, respectively, based on provider randomization (Table 1).

Fig. 1

Consort diagram.

Table 1 Characteristics of Actionable TPADs by Service, Test Type, Physicians, and Patients

In general, characteristics of actionable TPADs, patients with those TPADs, and physicians caring for them (Table 1) were equally balanced in the intervention and usual care groups. Most TPADs were generated on patients discharged from general medicine services. The most frequent test type was chemistry/hematology. Most discharging attendings were male and had 5 years or less of experience. Most patients were white and had incomes of $47,000 or less, public/government insurance, one actionable TPAD, and a non-network PCP. Additionally, the proportion of actionable TPADs listed as pending in the discharge summary was 50.7 and 49.5% in the intervention and control groups, respectively.

The proportion of actionable TPADs with documented action was not statistically different for the intervention arm compared to usual care (60.7 vs. 56.3%; adjusted and clustered OR 1.07 [0.58–1.99]; p = 0.82); this finding was similar for documented acknowledgment (Table 2). There were fewer readmissions in the intervention compared to usual care group, but this was not statistically significant (22.8 vs. 31.1%; adjusted OR 0.69 [0.37–1.28]; p = 0.24). The adjusted median days to documented action was significantly shorter for the intervention compared to usual care (9 [6.2, 11.8] vs. 14 [10.2, 17.8]; adjusted OR 0.64 [0.43, 0.97]; p = 0.04); this finding was similar for documented acknowledgment.

Table 2 Primary and Secondary Outcomes

An analysis of the effect of covariates (Appendix A online) on the primary outcome demonstrated that the proportion of actionable TPADs with documented action was significantly higher for pathology compared to other test types (77.2 vs. 48.5%; adjusted OR 4.11 [2.24, 7.56], p < 0.001). There was a non-significant trend towards increased proportion of actionable TPADs with documented action for patients discharged by hospitalists compared to non-hospitalists (62.3 vs. 54.4%; adjusted OR 1.45 [0.71, 2.98], p = 0.31). The effects were similar for documented acknowledgment.

Of the 152 actionable TPADs with documented acknowledgment across both study arms, 148 (97.4%) had evidence of documented action on the same day. Seventy-three (48.0%), 67 (44.1%), and 12 (7.9%) were pathology, chemistry/hematology, and microbiology test types, respectively; and acknowledgment was documented by an ambulatory clinician in 124 (81.6%). Of the 101 actionable TPADs for which no documented acknowledgment was identified, documented action was found in just one (1%); this actionable TPAD was a low vitamin D level, and the action was a new vitamin D supplement on the medication list. A total of 536 types of actions (Table 3) were documented for the 149 actionable TPADs across both study arms (mean (SD): 3.58 ± 1.30 actions per actionable TPAD); there were no significant differences between intervention and usual care groups for types of actions.

Table 3 Types of Documented Actions Recorded in EHR for Actionable TPADs

In sub-group analyses (Table 4), we observed significantly greater impact of the intervention on documented follow-up of actionable TPADs among patients with network PCPs (70 vs. 50%, p = 0.03). There was no evidence of effect modification of the intervention by clinical service, provider type or experience, test type, or number of actionable TPADs per patient.

Table 4 Sub-group Analyses: Documented Action for Actionable TPADs by Characteristic

The proportion of actionable TPADs without documented follow-up was 39.9% for acknowledgment, and 41.1% for action. The majority were chemistry/hematology tests and from non-network patients. Types of actionable TPADs without documented follow-up included abnormal vitamin levels (n = 25), pathology specimens (n = 15, including 6 malignancies); abnormal rheumatology titers (n = 9, including an extremely elevated anti-DS DNA titer); abnormal endocrine results (n = 8); abnormal viral tests (n = 8, including an elevated HCV viral load); positive microbiology cultures (4); and miscellaneous (32). See Appendix B online.

DISCUSSION

We report findings from a cluster-randomized controlled trial assessing the impact of automated notification of TPADs on documented follow-up in the EHR. For nearly all actionable TPADs, acknowledgment and action were documented at the same time, typically by an ambulatory clinician. Although we did not observe significant improvement in the proportion of actionable TPADs with documented follow-up, we did observe significant improvement in time to documented follow-up favoring the intervention. In covariate analyses, the pathology test type was most commonly associated with documented follow-up. Most commonly documented actions included establishing, changing, or confirming a diagnosis and communicating results with the patient. In sub-group analyses by PCP affiliation, we observed significant improvement in documented follow-up favoring the intervention for patients of network PCPs. Notably, a high proportion of actionable TPADs (~ 40%) did not have follow-up documented in the EHR, including a few high-risk results (e.g., malignancies).

Our inability to detect improvement in the primary outcome has several possible explanations. First, most patients in our study (~ 66%) had non-network PCPs, which likely diluted the effect of our intervention. The lack of improvement for non-network patients likely reflects two phenomena: lower impact of the intervention on care for those patients because non-network PCPs were not contacted directly (and inpatient attendings did not usually contact them indirectly), and an artifactual reduction in follow-up rates (among patients in both study arms) due to lack of documentation available to review (see limitations, below). Second, we did not expect follow-up documentation to be as high as observed in usual care (~ 60%). In prior studies, physician awareness of actionable TPADs (72 h after TPADs were reported) was 38%,3,17 and the proportion of TPADs with documented follow-up within 3-days of result finalization was 20–30%.8,26 We conducted chart review for up to 6 months after TPADs were finalized, which may have led to our higher estimates of follow-up. Third, documentation of pending results in the discharge summary may have influenced subsequent follow-up under usual care. However, the proportion of actionable TPADs listed in the discharge summary was modest (about 50%, despite concurrent efforts to improve discharge documentation), and ambulatory clinicians (not inpatient attendings) typically documented follow-up, so this is unlikely to fully explain our findings. Fourth, two other electronic notification systems were operational during the study period: one for abnormal pathology results and another for critical radiology results. The pathology system automatically notified attendings of abnormal results by email; however, unlike our intervention, this system did not provide a description of the result in the body of the email and did not concurrently notify PCPs.16,17 The radiology system—automated notification of critical results (ANCR)—requested electronic acknowledgment by the ordering clinician for clinically significant radiology results (e.g., pulmonary nodule) via a pager or email notification and escalated this request to the attending if acknowledgment was not recorded within a pre-specified timeframe depending on result severity.27,28,29 Both co-interventions may have improved follow-up for actionable pathology and radiology TPADs across both study arms. In fact, we identified only 4 (1.6%) actionable radiology TPADs in our sample; most clinically significant radiology results were acknowledged before discharge.28,29

Most studies have reported variable rates of TPAD documentation (10–80%) in the discharge summary.21,30,31,32,33 Few studies have reported documentation of follow-up actions in the EHR during the period after TPADs are finalized. While our study showed that increased awareness facilitated by automated notification did not significantly improve overall follow-up documentation, it did shorten time to documented follow-up independent of co-interventions, and was associated with a trend towards fewer readmissions. The significant improvement in time to follow-up has important ramifications for certain types of TPADs and should be underscored. Specifically, automated notification may be most relevant for non-immediately life-threatening TPADs (e.g., positive H. pylori Ag) that do not have systems in place to notify clinicians, “fall through the cracks” and have the potential to manifest in harm and re-hospitalization if appropriate treatments are not initiated in a timely manner.13,34

Our study had several limitations. First, review of outside EHR records was out of scope, and as mentioned above, this likely contributed to the apparent lack of effect of the intervention, especially for patients with non-network PCPs. Second, the study was limited to non-surgical services at a single academic institution; however, an automated notification strategy should be generalizable despite service- and institution-specific practice differences. Third, due to the volume of TPADs screened, we opted to exclude normal or near-normal results. We acknowledge that certain normal results (negative blood culture; benign pathology results) may prompt meaningful action (stopping empiric antibiotics; calling patients to provide reassurance), but measuring this was beyond the scope of our study. Fourth, actions restricted to orders but not documented in notes would have been missed. The search tool we used retrieved documentation alone; we did not measure recall and precision of our retrieval strategy. Fifth, we did not conduct a formal evaluation of the impact of our intervention on cognitive burden, but information overload and burn-out attributable to EHRs can clearly contribute to missed test results.14,35,36 Finally, we did not assess patients’ involvement in managing TPADs—in addition to technology and governance, patient engagement is a promising strategy to ensure follow-up.18,20,27,37,38

Conclusion and Implications

Automated notification as we describe is an effective strategy for facilitating closed loop communication of actionable TPADs, potentially decreasing time to action. Most major EHR vendors can extend existing functionality to externally notify clinicians of actionable TPADs via institutional email, secure messaging, or mobile app “push” notifications; flag importance in subject headings or notification banners; facilitate knowledge transfer via messaging threads between key inpatient and ambulatory clinicians; and request acknowledgment electronically.7,16,17,39,40 Institutions considering this type of “active” strategy would benefit from notifying non-network as well as network PCPs—technology to send out-of-network messages securely is increasingly available with growth of online physician directories and professional networks.41 Leveraging EHRs, communication tools, and online resources in this way can address safety concerns related to discontinuity within and across healthcare systems.30,42

Still, automated notifications are insufficient to fully resolve this problem.43 To ensure timely action is taken in the ambulatory setting, tools to transfer responsibility to follow-up providers, facilitate order entry for referrals and future studies, and identify results for which appropriate actions have not been executed are needed, especially for non-immediately life-threatening results.34 Furthermore, enterprise policies regarding appropriate acknowledgment and follow-up intervals are necessary to ensure accountability.27 Finally, future research is needed to evaluate the effectiveness of EHR-integrated notification technologies (including newer “smart” notification platforms), impact on cognitive burden, and utility in empowering patients and care partners to follow-up test results.