Introduction

Lack of timely follow-up of abnormal diagnostic test results is common in healthcare.1,2,3 These “missed test results” are important contributors to diagnostic error,4 including missed and delayed cancer diagnoses.5,6,7 Systematic reviews have also shown increased hospitalization and inappropriate medication adjustment as outcomes of missed test results.8 Prior work has examined barriers and facilitators to proper test result follow-up9 and improved understanding of testing processes and workflows. Test result follow-up at transition points, such as hospital and emergency room discharge, has been particularly vulnerable.10,11,12,13,14

A 2015 report from the National Academies advocates for “approaches to identify, learn from, and reduce diagnostic errors and near misses”.15 Test result tracking and follow-up processes are promising targets for interventions.13 For instance, certain organizations are using electronic surveillance to identify patients with delayed follow-up of test results.16,17 This approach uses structured data in electronic health records (EHRs) to find patients with missed test results. Wider application of such approaches would require a health system-based team to conduct surveillance for missed test results, analyze the data, identify which patients warrant intervention for the near-miss, and finally generate and sustain improvements. Most organizations do not yet have such teams in place. It is also unclear whether and how organizations have translated existing knowledge and interventions related to test result management processes into policies and practices to reduce risks.

Recognizing the need for additional efforts to reduce missed test results at an organization level, our team recently launched a project to develop and implement change strategies to improve test result follow-up across a national sample of Veterans Affairs (VA) health care facilities. We proposed using a Virtual Breakthrough Series (VBTS),18,19 an established approach to assist teams with implementing evidence-based practices. An essential early step in this work is to identify current practices and stakeholders’ perceptions of high-priority risk areas, which will guide development of interventions. To this end, we conducted a qualitative study to identify common, high-priority risk areas and contributing factors related to missed test results from the perspective of multiple stakeholders in VA facilities. Our goal was to synthesize stakeholder perspectives to refine development of change strategies for our project, and more broadly to identify where implementation gaps exist and elicit strategies to improve test result follow-up.

Methods

Study Setting and Population

The VA is one of the largest health care systems in the USA and has had an operational EHR since 1985.20,21 The Computerized Patient Record System (CPRS) is the EHR interface in use since 1997. It includes a system for in-basket type EHR notifications (commonly called “View Alerts”) for communicating test results, among other patient care messages related to orders, medication refills, and referrals. Ordering clinicians are always notified of certain clinically significant abnormal tests, and clinicians can assign covering surrogates to receive their notifications.22

We recruited participants from 12 VA healthcare facilities enrolled in the larger VBTS study. We promoted the study at meetings and conference events, and directly via email to facility-level leaders. We recruited facilities purposively to represent a range of facility sizes and geographical locations. After facility representatives confirmed participation, each identified a site champion to coordinate further activities.

Site champions nominated personnel at their facilities who had experience with patient safety and test result notifications, including physicians, nurses, leadership, and professionals in laboratory, radiology, and patient safety. We individually contacted nominated staff members to invite them to participate. We obtained verbal informed consent before interviews. The study was approved by the local institutional review board.

Interview Guide Development and Content

We developed a semi-structured interview guide to facilitate data collection and to cover key topics.23 Interview content reflected the three primary “drivers” or domains of change represented in the draft set of change strategies for the larger study. These domains, generated by literature review and research team discussions in the first step of our project, were as follows: (1) enhancing patient engagement with test results, (2) improving situational awareness among providers and care teams, and (3) implementing processes to close the loop on test result reporting and follow-up. We incorporated additional concepts about each of the three domains from existing literature24,25,26,27,28 when drafting the interview guide.

The final interview guide included 42 questions (Appendix), although each participant answered a subset of 25 or 26 questions depending on their role. All participants were asked 9 general questions about how test results are reviewed, communicated, and followed up at their institutions. Clinicians, nursing staff, lab personnel, and patient safety professionals were asked 16 questions specific to patient engagement with test results and providers’ management of EHR notifications. Leadership personnel responded to a different set of 17 questions about situational awareness among providers and care teams, “closing the loop,” and monitoring the effectiveness of test result management. Finally, all participants were asked for recommendations about actions to reduce the rate of missed test results.

Data Collection and Analysis

One author (US, a qualitative researcher) conducted individual interviews by telephone between March 2019 and January 2020. Interviews were audio recorded and transcribed verbatim by a third party. Transcripts were used to conduct qualitative content analysis, using an inductive approach, allowing codes and categories to flow from the data and new insights to emerge. The interviewer (US) conducted the initial data analysis by reading transcripts iteratively to fully understand the data as a whole. Codes were derived by highlighting words from the text that captured key thoughts or concepts, which were used to draft a codebook. This process continued on each transcript until no more new or meaningful concepts emerged. After revising the initial coding scheme with the study team, a second coder (UM) with expertise in health informatics independently coded the full data set and added additional codes to the codebook. Both coders then met to refine the codebook by grouping codes with similar meanings into higher-order categories representing contributors to missed test results. The final code book and categories were shared with the research team for review. Recommendations elicited from participants were tabulated and grouped by theme.

Results

We interviewed 27 participants across 12 sites, including clinicians (n = 9, 33%), facility leadership (n = 7, 26%), lab professionals (n = 4, 15%), radiology professionals (n = 2, 7%), nursing (n = 1, 4%), and patient safety professionals (n = 4, 15%). The typical duration of the interviews was 30 min (range 20 to 40 min). Qualitative content analysis revealed several risk areas that were grouped into 7 themes (Table 1), and recommendations grouped by theme (Table 2).

Table 1 Risk Areas Identified by Participants
Full size table
Table 2 Summary of Interview Participants’ Recommendations to Reduce Missed Test Results in the VA Healthcare System
Full size table

Trainees (Residents and Interns)

A majority of participants (n = 17, 63%) voiced serious concerns about test result management when medical trainees (residents) are involved. They described weak or absent teamwork, coordination, and accountability. At some facilities, treating clinicians did not receive notifications for tests ordered by trainees who had completed their rotations, resulting in test results being missed or found by chance.

Participants reported the need for a mechanism to ensure that all test results are seen by the supervisory clinician as a backup. However, other participants shared details about such mechanisms at their facilities. For example, at one facility, orders for labs or imaging that are entered by trainees can generate a notification to a supervising primary care provider (PCP). If a trainee does not acknowledge results within 2 weeks, the supervising PCP receives a notification. Some participants expressed a need for more trainee education on how to manage EHR notifications and hand off information after a rotation. Furthermore, they suggested that treating physicians should be aware of VA’s national policy on communication of test results to providers and patients. The policy establishes that test results should be communicated to patients within 7 calendar days for results requiring action and within 14 days for results requiring no action.

Incidental Findings on Imaging

More than half of participants stated that incidental findings are the results most likely to fall through the cracks, particularly for imaging. Lost to follow-up incidental findings could result in missed diagnosis or delays in patient care, sometimes for years (e.g., in the case of cancer).

Some participants added that there is no backup system to ensure that the ordering provider acted on the results. Many test results that do not require urgent attention still need action to be taken, and participants expressed a need for a clear process for follow-up actions, which all providers including supervising physicians and trainees can follow. Providers also expressed that addressing incidental findings requires better teamwork.

EHR and Related Tracking Systems

Participants from multiple roles discussed that critical test results are always verbally communicated as a closed loop. However, they reported that non-critical abnormalities are transmitted as EHR notifications where the loop may not necessarily close. It was unanimously accepted across participants that the VA lacks a backup system for tracking EHR notifications related to abnormal test results. Furthermore, providers acknowledged that they are overwhelmed by the number of EHR notifications they receive each day. Respondents added that, although there is a column indicating high, medium, or low importance of notifications, the system lacks a more compelling visualization to prioritize notifications based on urgency; many are given the same weight and volume, which leads to a poor signal-to-noise ratio and increases the risk for missing information. Participants recommended introducing a color-coding system to help providers more effectively triage their notifications and take action.

Lack of Updated Patient and Provider Contact Information

Nearly half the participants were dissatisfied with the system of maintaining patient and provider contact information. A few (n = 8) also reported that the VA facility is supposed to maintain current contact information for both patients and providers, but they were not sure whether this was being done. A commonly mentioned recommendation was for patient contact information to be updated at each visit. Nevertheless, many other participants were satisfied with the way their facilities maintain contact information and the ease of communicating critical test results.

Some participants, especially lab personnel, reported substantial difficulty finding the accurate provider contact information. Participants also reported that tier systems or cascading lists that identify the next in command were not consistently updated, and they noted that having updated contact information for both providers and patients is imperative for abnormal results. Multiple participants called for better access to ordering physician contact information, and for backup contacts when the ordering physician is not available.

Challenges in Referral for Consultation/Testing and Follow-up of Results

Several participants reported challenges related to consultation visits and testing referrals, including poor care coordination, breakdown in communication, delays in patient care, and excessive workloads. At times, ambiguity existed regarding who is responsible for test result follow-up after the referral (e.g., the PCP versus the specialist ordering the test). Participants emphasized that after a referral to a specialty clinic outside of their VA facility, the response time for test results is uncertain or long, and responsibility for information retrieval and follow-up may fall onto the referring clinic or the patient. One clinician suggested there should be a tracking person (e.g., a care coordinator) to facilitate communication of these test results. However, the majority of participants were satisfied with referrals and coordination of care within the VA.

Use of Surrogate/Backup Clinicians

A few participants mentioned test results being missed or lost to follow-up during the transition from the primary provider to a covering secondary provider, and vice-versa. Once a provider is no longer covering, notifications must be switched back to the primary provider; otherwise, EHR notifications will be transmitted to the covering clinician who is no longer involved in the patient’s care, creating a risk of missed test results. Multiple participants recommended use of more automation to support the covering physician notification system.

Test Results Pending at Discharge

Participants voiced concerns about pending test results at discharge, who receives the results, and who is responsible for their follow-up. Participants expressed a lack of clarity among providers (including themselves) about whether the responsible clinician for pending results should be the ordering provider, the inpatient attending of record, or the PCP. Participants recommended instituting a process with clear guidance for who is responsible, and automatic generation of a list of tests pending at discharge.

Discussion

As part of a larger project to improve test result follow-up, we identified current implementation gaps and elicited strategies for improvement in a national sample of VA health care facilities. Themes commonly mentioned included concerns about trainees, incidental findings on imaging, and using the EHR effectively for follow-up. Also notable were mentions of problems for which potential solutions were available but apparently not implemented.

Some of our findings reflect the need for strategies to enhance uptake of and adherence to existing policies and best practices. For instance, we found that several participants wanted guidance about who is responsible for tests pending at hospital discharge, even though a policy for this already exists, in line with previous recommendations for such guidelines.8 The national VA policy states, “When results of tests ordered and performed while the patient is inpatient become available after discharge, they are communicated to the patient by the ordering inpatient provider, or their designee, unless responsibility is transferred to an outpatient provider, or their designee, and the transfer is documented in [the EHR].”.29

All VA facilities are expected to follow national policy and develop local processes and procedures to address the requirements set forth therein. Several participants were unaware of any strategies implemented at their facilities to improve follow-up of test results pending at discharge, despite prior research on effective strategies.10,11,12 This suggests a need not only for diffusion (passive spread) and dissemination (active spread) of research and policies, but implementation, adoption, and sustainability.30,31 Prior work has suggested that sustainable policy implementation requires not just education of the users of the policy but also consistent and specific reinforcement.32 Sustainability has been under-researched, and qualitative approaches are needed to identify ways to sustain policy implementation.33

Several recommendations from interviewees have been described in prior work.34 This illustrates the presence of an implementation gap in this area, which is not surprising because passive diffusion of practices has been challenging.35,36 Given the complexity and the sociotechnical context of the problem,37 implementation methods warrant a careful evaluation. Moreover, a single solution is unlikely to provide substantial benefit. For instance, because of variation in EHR usage skills from provider to provider,38 users need effective ways to achieve competencies in required skills but also better EHR workflows for dealing with test result notifications. Moreover, PCPs have previously reported lack of sufficient protected time for EHR notification management, as well as desire for better visualizations,35 similar to recommendations elicited in the present work. Recommendations frequently related to EHR features that interviewees found lacking. Prior work has described the workarounds that EHR users use39 and the need for the evolution of the EHR to match workflow needs.40 A sociotechnical approach is needed to address deficiencies and complexities of electronic test ordering, alerting, and follow-up.37,40,41,42

Lack of timely follow-up of tests ordered by trainees has also been described in several prior works.43,44 This phenomenon is common in academic medical centers (not only in VA medical centers) because residents and interns rotate frequently from one specialty to the next, they may be unfamiliar with follow-up methods at new hospitals or specialties, and because it may be months before they return to the same clinical setting. Multiple themes found in this study could be addressed by automatically escalating unacknowledged test result notifications to individuals who could take action. This was only being done at a single facility despite being considered a useful safety practice.45 Our findings suggest that organizations could identify high-risk scenarios for test result follow-up (such as incidental imaging findings) and scale up strategies such as call cascades, a successful strategy in other research.44

Strategies to enhance test result follow-up have not seen widespread adoption despite many years since publication, and additional reasons for this lack of implementation should be explored. Improvement models such as the Institute for Healthcare Improvement’s Breakthrough Series collaborative have shown promise in bridging the gap between knowledge and practice. This model has been applied in previous studies, including those in the VA, and is especially useful for improvement efforts across facilities. In the next steps of our work, we plan to use a Virtual Breakthrough Series model to foster collaboration and learning among staff at participating facilities as they implement change strategies informed by our findings.

Our study has several limitations. First, we interviewed employees only within the Veterans Health Administration; recommendations may not generalize to other health systems or EHRs. However, all of the risk area and recommendation categories (Tables 1 and 2) are not specific to the VA system and are relevant to any health system. Several prior systematic reviews have outlined the problem and examined the effects of interventions to improve follow-up, such as education, manual review12, automated tools10,13, and other electronic systems.46 Most of the individual trials cited were performed outside of VA, but they examine the same broad categories of intervention that our participants proposed. Second, recommendations were collected without regard for feasibility, cost-effectiveness, or unintended consequences, all of which should be assessed before implementation. Several recommendations involve adding new steps (additional workload) to existing processes and changes to the EHR, which need involvement of EHR designers.47 Third, our sample although national may not be representative. Certain roles were not well represented in our sample (e.g., radiology and nursing) and participants were eligible only if their facility expressed an interest in the project. Finally, VA is in the early stages of an enterprise-wide EHR modernization effort.48 Nevertheless, technology is only one part of a larger sociotechnical system, and many of these lessons would be applicable to its future EHR or to any other EHR system. Future work should also explore cultural and “hidden curriculum” factors leading to missed test results, such as time pressures and training hierarchies.49

In conclusion, we used qualitative methods to identify factors contributing to the persistent problem of missed test results. Several of these factors have been elicited before and reflect the presence of an implementation gap at the organization level.50 Strategies to enhance uptake of and adherence to existing policies and best practices are needed. Our work is a first step as part of a Virtual Breakthrough Series approach to close implementation gaps around missed test results51,52,53. Long-term goals are to develop resources and practices that can be disseminated within and outside of VA. However, in the short term, all health care facilities should address fixable issues such as responsibility for follow-up of test results, updated contact information for clinicians and patients, and escalating or backup procedures for tests ordered by trainees.