Download PDF

Implementation Science

, 12:79 | Cite as

Implementation science for ambulatory care safety: a novel method to develop context-sensitive interventions to reduce quality gaps in monitoring high-risk patients

  • Kathryn M. McDonald
  • George Su
  • Sarah Lisker
  • Emily S. Patterson
  • Urmimala Sarkar
Open Access
Research
First Online:
Received:
Accepted:

Abstract

Background

Missed evidence-based monitoring in high-risk conditions (e.g., cancer) leads to delayed diagnosis. Current technological solutions fail to close this safety gap. In response, we aim to demonstrate a novel method to identify common vulnerabilities across clinics and generate attributes for context-flexible population-level monitoring solutions for widespread implementation to improve quality.

Methods

Based on interviews with staff in otolaryngology, pulmonary, urology, breast, and gastroenterology clinics at a large urban publicly funded health system, we applied journey mapping to co-develop a visual representation of how patients are monitored for high-risk conditions. Using a National Academies framework and context-sensitivity theory, we identified common systems vulnerabilities and developed preliminary concepts for improving the robustness for monitoring patients with high-risk conditions (“design seeds” for potential solutions). Finally, we conducted a face validity and prioritization assessment of the design seeds with the original interviewees.

Results

We identified five high-risk situations for potentially consequential diagnostic delays arising from suboptimal patient monitoring. All situations related to detection of cancer (head and neck, lung, prostate, breast, and colorectal). With clinic participants we created 5 journey maps, each representing specialty clinic workflow directed at evidence-based monitoring. System vulnerabilities common to the different clinics included challenges with: data systems, communications handoffs, population-level tracking, and patient activities. Clinic staff ranked 13 design seeds (e.g., keep patient list up to date, use triggered notifications) addressing these vulnerabilities. Each design seed has unique evaluation criteria for the usefulness of potential solutions developed from the seed.

Conclusions

We identified and ranked 13 design seeds that characterize situations that clinicians described ‘wake them up at night’, and thus could reduce their anxiety, save time, and improve monitoring of high-risk patients. We anticipate that the design seed approach promotes robust and context-sensitive solutions to safety and quality problems because it provides a human-centered link between the experienced problem and various solutions that can be tested for viability. The study also demonstrates a novel integration of industrial and human factors methods (journey mapping, process tracing and design seeds) linked to implementation theory for use in designing interventions that anticipate and reduce implementation challenges.

Keywords

Organizational interventions Diagnostic error Ambulatory care Cancer Patient safety Human factors Journey mapping Design seeds Patient monitoring 

Abbreviations

AHRQ

Agency for Healthcare Research and Quality

EHR

electronic health record

ENT

ear nose and throat

f/u

follow-up

GI

gastroenterology

HIT

Health Information Technology

NAM

National Academy of Medicine

PCP

primary care provider

Download fulltext PDF

Background

A seminal National Academy of Medicine (NAM) report asserts that most people will experience at least one diagnostic error -- a delayed or inaccurate diagnosis -- in a lifetime, “sometimes with devastating consequences.” [1]. In ambulatory care, one of 20 patients in the United States experiences potentially preventable diagnostic errors annually [2, 3]. Missed cancer diagnoses are the leading reason for paid medical malpractice claims in the ambulatory setting [4, 5].

Widespread research across specialties demonstrates that inadequate monitoring in high-risk outpatients leads to preventable high-risk events and significant patient harm [5, 6]. For example, patients who have a positive fecal blood test but no follow up colonoscopy within a reasonable time period may experience a missed opportunity to detect and successfully treat colon cancer [7, 8]. The National Comprehensive Cancer Network has monitoring guidelines for screening (active and initial) as well as post-treatment cancer recurrence surveillance [7, 9, 10, 11, 12, 13, 14, 15, 16, 17]. Proactive and efficient strategies to respond to high-risk situations—such as overdue imaging or blood tests—are urgently needed to reduce the safety gap in evidence-based monitoring for cancer [18, 19, 20, 21, 22, 23].

Impoverished populations (those served by “safety-net” settings in the United States) are particularly prone to failures in patient monitoring, given high prevalence of limited health literacy, as well as barriers like lack of transportation, inability to leave work for medical appointments, and a myriad of other obstacles to engaging with the health care system [24, 25, 26, 27]. At the same time safety-net health systems often lack critical Health Information Technology (HIT) infrastructure and resources (e.g., personnel time) to devote to monitoring these vulnerable populations [28]. Common software development practices favor mass production and rapid adoption over user-specified customization necessary for long-term sustainability for challenging workflows [29, 30]. To accomplish robust patient monitoring and prevent adverse events, it is critical to identify setting- and population-specific vulnerabilities and needed attributes of effective interventions, whether technical, social, organizational or a combination.

Theories for complex delivery system interventions stress the importance of studying human and contextual aspects of change [31, 32]. The interactions between and within multiple levels of the health care system are posited to affect quality improvement projects [33]. The recent NAM Improving Diagnosis report included a high-level conceptual framework with these factors, among others such as clinical reasoning and teamwork that contribute to diagnostic safety and improvement [1]. The NAM framework explicates that patient and systems outcomes are produced by the diagnostic process which evolves over time, within the context of a larger work system composed of diagnostic team members, tasks, technologies and tools, organizational elements, and the physical environment [34, 35, 36, 37]. To reduce the chance of missing a cancer diagnosis, vulnerabilities need to be addressed within both the ambulatory care’s diagnostic process and work system. In other words, what work system factors produce robust monitoring (systems outcomes) and fewer diagnostic errors (patient outcomes)? Research in this area is nascent, with many unknowns about specific vulnerabilities, patient safety intervention opportunities, and subsequent implementations [1, 38]. Taylor et al identified four theoretical domains of contextual features determined by expert consensus as important for patient safety intervention implementations: safety culture, teamwork and leadership involvement; structural organizational characteristics; external factors; and availability of implementation and management tools [39, 40].

To develop theory-based, context-informed organizational interventions for closing the safety gap, our study introduces a unique integration of user experience and human factors methodologies: journey mapping and design seeds. We apply journey mapping to clinician-centered workflow focused on patients at high risk for a missed monitoring opportunity to diagnose cancer [41]. Previous applications have taken the perspective of an individual patient’s journey within a health care setting [42, 43, 44]. Design seeds are solution attributes that separate the goal of a modular intervention (e.g., alerting patients that they need to return to the clinic) from the means for achieving it (e.g., use of a web portal messaging system) [45]. They have the advantage of generating multiple solutions to the same problem so unknown vulnerabilities and preferences can be uncovered, interventions can be tailored to different contexts, and more solution variations can be considered to evaluate correct fit [46]. To apply design seeds to patient monitoring,1 we draw from a somewhat analogous situation studied outside of health care: intelligence analysts who experience time pressure and data overload as they cull through numerous documents to identify national security threats [47]. Our approach will inform prototyping, piloting and full-scale testing of technical and organizational interventions, with the aim of producing robust population-level monitoring solutions for widespread implementation.

Methods

Design

We conducted formative research, following a 6-stage co-development process between the research team and frontline clinicians (attending doctors, residents, nurse practitioners, registered nurses) to identify solution attributes of a comprehensive intervention for more robust monitoring of high-risk cancer conditions over time (Table 1). The research team applied human factors strategies and organizational theory about complex adaptive systems within five specialty clinics to identify vulnerabilities and generate desirable solution attributes for interventions [1, 39, 48, 49].
Table 1

Co-Development Research Process

Key Questions Based on NAM Improving Diagnosis Framework

PROBLEM: What vulnerabilities exist in monitoring outpatients for high risk conditions (e.g., cancer)?

SOLUTION: What elements of work systems and the diagnostic process are important to produce robust monitoring & thereby reduce diagnostic errors?

Stage 1: Identify 5 High-risk Populations and Clinical Informants

• Review literature

• Corroborate with local clinicians taking care of these patients

• Determine with clinical leadership who to interview, based on responsibility for patient monitoring (2-3 clinicians/clinic; 11 total)

Stage 2: Develop Journey Maps

• Identify key participants to learn about workflows for each high-risk population

• Elicit with semi-structured interview a description of the patient and data flow from worker’s vantage

• Visualize this information into swim lanes or “clusters” of activities

• Show swim lanes to participants and revise (as needed)

• Visit clinic sites to observe critical parts of process (as needed)

Stage 3: Generate Vulnerability List

• Abstract vulnerabilities from interview notes and journey maps

• Return to clinic participants to validate the list (one or more clinic has indeed experienced vulnerability)

• Map validated list of items to theory domains from applicable patient safety frameworks [1, 60]

Stage 4: Analyze Journey Maps for Commonalities

• Categorize types of activities in the journey using human factors method of process tracing (novel extension to derive tracings from journey maps)

• Generate process trace sequences for each clinic’s workflow [61]

• Look for patterns of workflow that are similar and variable across the 5 populations

Stage 5: Develop Design Seeds for Interventions and Link to Implementation Theory

• State what a solution would need to do to address vulnerabilities identified from previous stage

• Reduce the list to solution attributes (design seeds) that address common problems and needs across clinics

• Aim for design seeds that meet the generic needs of robust monitoring and that enable evaluation

• Hypothesize which contexts are likely to affect the effectiveness of the implementation of the interventions emanating from the design seeds using Taylor et al’s contextual domains and features (see Additional file 3) [39]

Stage 6: Seek Reactions from Clinics on Design Seeds

• Assess anticipated impact (improved monitoring of patients, reduced time spent by clinic team) and relative priority of each design seed (see Additional files 1 and 2 for script and data collection instrument used in each clinic)

Setting

The San Francisco Health Network is a publicly funded, integrated health network operating under the auspices of San Francisco’s Department of Public Health and includes 14 primary care clinics, urgent care, and specialty care at Zuckerberg San Francisco General Hospital. Last year, there were 539,310 outpatient visits at the hospital alone [50].

The health system serves many of the most medically and socially vulnerable patients in San Francisco. Patients seen at the network’s main clinic and hospital are diverse: 17% are African American, 35% are Latino, 21% are White, and 21% are Asian. Services are provided in over 20 languages. Based on outpatient days, only 1% of the population has commercial insurance, 10% uninsured, 57% Medi-Cal, 21% Medicare, and the remaining 11% covered by other mostly public sources [51]. Others have categorized hospitals according to safety-net burden, with high-burden ranging from 33 or 36% to 100% of patients as those with Medicaid or no coverage [52, 53].

Like many safety-net systems and ambulatory practices, the health system does not have a comprehensive electronic health record system and struggles with information transfer as well as fragmentation of health information across over 50 electronic platforms. Despite some of the HIT challenges and known workarounds typical of these safety-net settings, the organization has a longstanding commitment to both human-centered strategies (patient-centered medical home, plan-do-study-act cycles) and Lean management methods [54].

Evidence-based Safety Gaps Targeted (Stage 1)

Based on literature about missed and delayed diagnoses, including reports from medical malpractice, we selected five high-risk cancer situations—incidentally-discovered pulmonary nodules, and monitoring for breast, colorectal, prostate, and ear, nose, and throat (ENT) cancers—for which coordination and timely use of data are important for patient safety surveillance but challenging to implement, particularly in safety-net and other low resource settings [4, 5, 55]. These challenging high-risk situations require recurring and timely follow-up care to prevent harm [7, 9, 10, 11, 12, 13, 14, 15, 16, 17, 56].

Our team (GS, SL, KM) conducted a series of theoretically informed semi-structured interviews with participants from each of five specialty clinics responsible for these high-risk patients: pulmonary medicine, breast cancer, gastroenterology, urology and otolaryngology. As part of these interviews, we corroborated the specific safety targets by asking frontline clinicians: “What keeps you up at night?” and “What are your clinical hunches about who might fall through the cracks?” Although providers talked about the types of patients lost to follow-up, none of the clinics were enabled with a standardized and efficient method for quantifying how many patients were lost to follow-up, why patients were lost to follow-up, or even which patients were lost to follow-up. Many other health networks share similar struggles with incomplete documentation and measuring the real-time scope of patient safety problems [57].

Mapping and Analyzing Clinical Workflows (Stage 2 through 4)

The interviews in each of the five cancer clinical settings followed a user-centered design approach called journey mapping, a tool widely used across multiple industries [41, 58, 59]. Journey mapping derives from user experience initiatives in industry that informed the framework proposed in the recent NAM Improving Diagnosis report [1, 35, 60]. The method articulates and documents a process through a specific point of view (typically, a customer). In the health care field, it has been applied to elicit individual patient journeys through the clinical workflow [43]. Our team-based variation of journey mapping has a patient population management view. We elicit specialty care management through the experiences of the clinical team as they try to track the host of patient data required to monitor their high-risk population. To our knowledge, this technique has seldom been applied to the ambulatory setting, and has not been targeted to clinic workflow efficiency or patient safety intervention development.

We directed these journey mapping sessions to: (1) isolate the steps in the patient monitoring journey that are the most critical, time-intensive, and risky relative to the safety gap, (2) identify critical data elements needed to effectively and safely monitor patients, and (3) gather potential attributes of organizational and technical interventions to ameliorate workflow problems. To construct the journey maps, investigators (GS, SL, KM) probed clinical participants with questions such as: “What are you working on?” to elicit actions taken; “Who is responsible for which task?” to learn about monitoring-related activities; “Are there external stakeholders?” and “How important are they?” to surface coordination challenges outside of the clinic. Based on what participants articulated, we constructed a journey map for each clinical pathway with their review and endorsement. The maps focus on the transfer of patient data throughout the patient’s monitoring experience, starting with the initial diagnostic assessment and ending with the ongoing follow-up. Whenever participants verbalized elements of the pathway that were particularly vulnerable to error or poor monitoring, we marked the activity with a bull’s eye target, also referred to by clinicians as a ‘pain point’.

From the journey mapping sessions, we listed all of the vulnerabilities experienced by at least one clinic. To verify the list and gauge how many of the clinics experienced each of the vulnerabilities, we returned to the clinic with a data collection instrument (Additional files 1 and 2). We also mapped the vulnerabilities corroborated by at least one clinic to domains from patient safety theoretical frameworks [1, 60].

Using standard process tracing techniques from human factors, we categorized and summarized the sequence of activities described in journey maps [61]. The trace sequences are used to determine the similarity of activity flow among clinics that monitor high-risk populations as well as any differences between clinics to inform well-designed interventions.

Developing Design Seeds and Linking to Implementation Theory (Stages 5)

Design seeds and the human factors approaches from which they stem have been used outside of health care for development of complex socio-technical interventions [47, 62]. They serve as bridges to technical and organizational solution options that can be designed differently depending upon context, but that use common attributes. As such, they offer an appealing addition to the implementation science toolkit. In simple terms this approach replaces the typical technical approach (Fig. 1a) with a theoretically based socio-technical system understanding (Fig. 1b). As shown in Fig. 1b, design seeds link the vulnerabilities experienced to potential solutions in a specific and evaluable way. This promotes the evaluation of a “seed” to a solution, rather than a full-fledged solution itself as is practiced in software development cycles often used in HIT [63]. By jumping directly from “problem” to “solution,” one opens the door to various misdirected applications that do not appropriately mediate the diverse instantiations of a problem. The evaluation of design seeds prior to the development of a solution creates an opportunity for more cost-effective and user-customized solutions [47].
Fig. 1

a Technical Intervention Development Cycle. b Socio-Technical Intervention Development Cycle: Design Seed Theory. The figures show that the socio-technical design seed intervention development adds an intermediate step that translates expressed vulnerabilities into multiple solution possibilities and evaluation markers. In contrast to a singular solution provided when linking a problem directly to a proposed solution, design seeds tease apart the expressed vulnerability to offer a distinct set of evaluable solutions that can be tested independently. Design seeds yield a menu of modular options for implementers considering differing organizational context

Since a design seed features a series of evaluable statements, the approach enables intervention testing at the right point in the pathway for a specific action (e.g., does the intervention work according to the design seed criteria? yes, no, or partially). In order to set up theory-based implementation, we (KM, SL) independently used these statements to hypothesize which contexts are likely to affect the effectiveness of implementation of the interventions emanating from the design seeds. We used Taylor et al’s four contextual domains and 13 specific features that a technical expert panel judged as high priority for assessment (as opposed to simple description) in the evaluation of a varied range of patient safety intervention implementations (Additional file 3) [39].

Assessing Clinician Reactions to Design Seeds (Stage 6)

To gain insight about the importance of the design seeds, we developed and tested a data collection script and instrument (Additional files 1 and 2). We then returned to the frontline clinician participants from each clinic, who reviewed each design seed, assessed likelihood of improved monitoring and likelihood of reducing time spent monitoring, and ranked the set of seeds for relative overall importance.

Results

From January 2015 to February 2016, we convened one or more journey mapping sessions with clinicians and staff at five specialty clinics to establish the workflow for monitoring high-risk patients. As expected, all clinics participate in teaching alongside patient care, have similar patient demographics with accompanying operational challenges (e.g., translation services, transportation needs), and use the same underlying electronic health record system but work within a larger system of fragmented record-keeping systems (e.g. different specialty-specific EHRs, electronic systems restricted to on-site devices, paper-based systems). The mapping process also revealed variability in organizational approaches to monitoring high-risk patients, including the types of personnel involved in various monitoring-related tasks (e.g. resident versus nurse responsibility for tracking) and the specific steps taken to monitor high-risk patients (e.g., use of notebook-based list of patients versus lack of a structured tracking tool).

Journey Maps: How Specialty Clinics Monitor High-Risk Patients

For each clinic, we constructed a journey map as shown in Fig. 2a, the abnormal colonoscopy workflow, and Fig. 2b, the ENT cancer workflow. These journey maps follow the management of patients with concerning conditions requiring cancer surveillance, diagnosis, monitoring and treatment, starting with referral to the respective subspecialty clinic. Each journey map contains swim lanes (visual columns) to group similar activities, flow arrows to represent patient and information movement, and targets to highlight areas of vulnerability for monitoring as expressed by clinic personnel.
Fig. 2

a Abnormal Colonoscopy Journey Map. b Ear Nose and Throat (ENT) Cancer Journey Map. Investigators constructed journey maps that follow the management of patients who are being monitored and treated for cancer, as articulated by clinical participants. Similar activities and actions are clustered into vertical “swim lanes,” arrows indicate the flow between actions, and bull’s eye targets mark actions that are particularly vulnerable to missed monitoring. For example, an action that does not have an “owner” may instigate a higher risk for patient loss to follow-up

For example, an abnormal colonoscopy triggers entry into the gastroenterology clinic workflow (Fig. 2a), which is adjudicated by the attending doctor. The first swim lane clusters the activities related to referrals. The next two swim lanes separate two different levels of diagnostic concern and coordination – one for benign lesions which just require notifying the primary care doctor, and the other for “sinister” lesions which precipitate a series of actions within the specialty clinic, as well as coordination with others based on subsequent findings (e.g., pathology, primary care, oncology). The bull’s eye target on the box -- “if no-show, patient falls off the list” -- means that the clinic is aware of this vulnerability, but does not have any further, regular steps to reduce the risk of losing a patient to follow-up. The bottom of the diagram illustrates that patient-related contacting happens throughout the workflow; the associated target conveys the challenges in reaching patients outside of clinic and assuring that they can make it to follow-up encounters.

The ENT clinic (Fig. 2b) reported similar challenges contacting patients monitored and treated for cancer, as did all other clinics (Additional file 4). The activities performed by the ENT clinic for cancer monitoring cluster into four swim lanes—case identification and referral, coordination, consultation, and care pathway. In this clinic, the coordination activities do not follow from a particular clinical scenario (the benign versus sinister lesion), but instead relate to a particular role, the chief resident. As a result, this clinic identified four separate vulnerabilities related to the busy chief resident’s responsibility to keep patients on the “ENT Radar” without any specific tools besides paper notecards, while also coordinating resources such as transportation for patients, tumor board presentations, and communication of follow-up requirements to primary care providers (PCPs). The care pathway swim lanes sketch out a series of diagnostic activities and pre-treatment preparation. The last stage of this clinic’s care pathway is patient surveillance after treatment. No specific responsibility assignment exists for patients who require regular surveillance to monitor for cancer recurrence, so the ongoing surveillance activity box is labeled with a bull’s eye target, indicating another vulnerability.

The journey maps intentionally tell only part of the story as they represent the journey told from a single clinic’s perspective. For example, all of the subspecialty clinics have interactions with PCPs, but only when an individual clinic spoke about dependencies on the PCP for the patients that they monitor did we include the PCP in a journey map. For incidental lung nodules, breast cancer oncology navigation service, and abnormal colonoscopies, the specialty clinics rely on the PCP to remind patients to follow-up at the necessary intervals since they have minimal contact with these patients.

Challenges Experienced in the Clinics

Based on the interview notes and journey maps from all five clinics we developed a comprehensive list of vulnerabilities described by at least one clinic (see Table 2). We identified 45 distinct vulnerabilities, and mapped these to domains from patient safety theoretical frameworks: 36 relate to work system factors that are inherent to environment, task, technology, organization and people, while 9 vulnerabilities correspond to process factors that reflect interactions between people or with systems [1, 60]. Each clinic reviewed the list at least four months after journey mapping to validate high-priority vulnerabilities that persist over time despite ongoing organizational changes and to record differences between clinics. Only two vulnerabilities—1) have to track some patients in own mind or side system, and 2) creating list of patient requiring monitoring takes time – were experienced by all five clinics. At least two clinics (in varying combinations) experienced most of the vulnerabilities. Four of the five clinics verified multiple problems related to the time expended on tasks related to monitoring. The breast cancer clinic experienced only 7% of the full list of possible vulnerabilities, while the others experienced 12% to 34%. This lighter vulnerability burden is perhaps because the breast clinic has separate philanthropic funding that supports patient navigation services, referred to by a participant as a “human tracking system”.
Table 2

Vulnerabilities Experienced by Each Clinic

Vulnerability from Specialty Clinician Perspective

# of Clinics Experiencing

Clinic (X = experienced)

Classified by Framework Domaina

B

P

GI

E

U

Work System: Task

 Have to track some patients in own mind or side system

5

X

X

X

X

X

 Creating list of patients requiring monitoring takes time

5

X

X

X

X

X

 Looking up each patient’s information takes time

4

 

X

X

X

X

 Maintaining list of patients requiring monitoring takes time

4

 

X

X

X

X

 Outside of visit-based care, don’t always know when patients need follow-up monitoring

4

X

X

 

X

X

 Manually monitoring patients is time intensive

4

 

X

X

X

X

 Don’t always know which patients need to be called back for monitoring

3

 

X

 

X

X

 Have to spend too much time scheduling

2

 

X

 

X

 

 Manually monitoring patients is error-prone

2

   

X

X

Work System: Technology and Tools

 Analyzing data in ad hoc manner is time intensive

4

 

X

X

X

X

 Inefficient system to create personal, siloed reminders for follow-up

4

 

X

X

X

X

 List of patients we use outdates quickly

3

  

X

X

X

 Can’t divert alerts to other providers

3

 

X

X

X

 

 Analyzing data in ad hoc manner is error-prone

3

 

X

 

X

X

 Don’t always know when patient data is missing

2

 

X

 

X

 

 Can’t find missing data from outside clinic

1

   

X

 

 Don’t always want alert when patient status changes

1

    

X

 Don’t have adequate real-time data

1

   

X

 

 Can’t edit patient’s care pathway as needed based on frontline data

1

 

X

   

 Can’t find missing data within clinic

1

   

X

 

Work System: Organization

 Systems don’t talk to each other

4

 

X

X

X

X

 Don’t have a system that puts patients into subgroups for more efficient monitoring

4

 

X

X

X

X

 Can’t share patient list with entire care team

3

 

X

 

X

X

 Don’t always have the time to perform the assigned role

2

 

X

 

X

 

 Hard to stratify patients into subgroups for monitoring due to many individual patient differences

2

 

X

 

X

 

 Care plan is poorly documented

2

 

X

 

X

 

 Don’t know what types of scheduling challenges occur most often

1

   

X

 

Work System: People

 Overlapping efforts

4

X

 

X

X

X

 Don’t always know when the loop closes

3

  

X

X

X

 Everyone inputs data differently

2

 

X

 

X

 

 Knowing who is managing at each stage is unclear

2

   

X

X

 Mapping patient to care plan requires clinical judgment

2

 

X

 

X

 

Work System: Environment

 Coordinating scheduling efforts across care teams is difficult

3

 

X

 

X

X

 Little or no performance data about monitoring so don’t know where to focus any improvement efforts

3

X

X

 

X

 

 Stretched for resources to reach out to all patients in need of follow-up

3

 

X

X

X

 

 Unaware of clinic’s performance in patient monitoring

2

X

  

X

 

Process: System-Patient Interaction

 Don’t know when patient misses appointment

4

 

X

X

X

X

 Don’t always know when patient doesn’t have PCP

4

X

X

 

X

X

 Don’t always know patient’s vulnerabilities relevant to monitoring (e.g. patient’s work schedule, can’t get to clinic, substance abuse)

3

 

X

 

X

X

 Difficulty communicating patient needs with entire care team

2

 

X

 

X

 

 Don’t know when patient changes status

2

 

X

 

X

 

Process: System-Provider Interaction

 Inconsistent process for informing PCP

3

 

X

 

X

X

 Can’t use patient data for operational improvement

2

 

X

 

X

 

 Involving PCP when not necessary

1

X

    

Process: Patient-Provider Interaction

 PCP doesn’t have overview of all patient info/care pathway

3

 

X

 

X

X

aAdapted from the National Academy of Medicine Improving Diagnosis Framework, 2015 and Sarkar et al’s System-related Factors, 2014 to classify each reported vulnerability into Work System versus Process, as well as subdomains of these two framework categories [1, 60]

Legend: Clinics designated as B = Breast, P = Pulmonary, G = GI, E = Ear Nose and Throat, U = Urology

Several broad work system challenges emerged from the clinic visits:

  • Organized for visit-based care (as opposed to patient management over time)

  • Rotating care providers from visit-to-visit due to being a teaching environment (as opposed to having doctors with long-term organizational know-how)

  • Lack of clear ownership for the monitoring-over-time function (as opposed to task responsibility and adequate time allocated for this population management function)

  • No aggregated real-time lists of those who require follow-up monitoring (as opposed to supportive tools)

  • Lack of systematic and transparent approach to patient’s care plan (as opposed to widely known and specified benchmarks and timing for monitoring follow-ups)

  • Substantial time pressure limits frontline attention to learning from missed monitoring incidents (as opposed to efforts to analyze data about misses, understand vulnerabilities and develop organization-wide solutions)

This work environment analysis that utilizes the NAM framework underscores the lack of infrastructure and processes organized to support population-level tracking of patients undergoing diagnosis of initial cancer, progressing cancer or recurring cancer. One noteworthy finding was the lack of population-level descriptions of the different types of monitoring care pathways commonly used within a given clinic. For example, the urology clinic participants – an attending doctor, a resident and a nurse – described a composition book where the resident logs all urologic patients who had a pathology result. The composition book is a starting point for population-level tracking of those who are at some risk for being lost to follow-up despite likelihood of needing it. However, the list is not sub-divided or categorized based on findings, conditions, anticipated follow-up pathway (e.g., testing, timing of next visit). The clinic participants noted that they preferred a system to monitor for all urologic cancers rather than restricting to prostate cancer monitoring (journey map focus) and that the composition book re-emerged as a workaround after a technical monitoring system was unsuccessful.

Process Trace Sequences: Four Critical Activities for Monitoring High-Risk Patients

To simplify the journey maps and enable pattern recognition across clinics (see Additional file 5 for color-coded journey maps), we categorized each action into one of three functional clusters:

  • Communicate/coordinate

  • Patient activity (contact patient, patient shows up)

  • Review or enter data/data systems

Figure 3 shows the resultant process trace sequences derived from the journey maps for each of the five clinics. The workflows have similar patterns: review and entering data at the beginning of the journey; a couple activities to communicate and coordinate within the clinic team before seeing a patient, a series of tests and appointments where the patient has to show up, and some patient contact outside the appointments punctuating the middle of the journey; and more communication or coordination actions marking the end of the journey. As noted in the thematic analysis, a fourth critical activity weaves through the sequence:
  • Track progress related to patients and their follow-up needs
    Fig. 3

    Process Trace Sequences. The display shows process trace sequences of major activities, and the constant tracking to monitor high-risk patients. Each clinic sequence is derived from a tricolor-coded version of its original journey map (Additional file 5).

Design Seeds: Elements of a Comprehensive and Adaptable Intervention to Save Lives and Time

To inform intervention development, we looked for leverage points to alleviate the vulnerability areas that held the highest consequence for failure. We generated a list of 13 leverage points, called design seeds, which correspond to the critical activities for robust patient monitoring, as shown in Table 3. One of the clinics, urology, told us that they had a registry but it was not used. This situation exemplifies the typical solutions pathway, as shown in Fig. 4a. In contrast, based on socio-technical theory, stating simply that a registry “is needed” is too minimalistic and fails to take the organizational context and its potential variations into account. Figure 4b provides an example of the design seed description for functions needed in a population registry of high-risk patients requiring monitoring. The design seed communicates the intent behind the recommendation resulting in a modular - therefore more evaluable - set of solution attributes. Each of the four functions (e.g., groups patients by PCP) shown can support different components of an intervention. In addition, each design seed functional statement can easily be converted into an evaluation question, such as “does the intervention use data visualization in a way that enables rapid identification of patients in need of follow-up?” or “does the intervention allow our clinic to prioritize work in a way that assures that the highest risk patients receive follow-up first?” (Additional file 1 has the detailed functional descriptions for each of the design seeds, as presented to the clinics for feedback.) These descriptions also support hypothesis-generation about contextual features that may have variable effects on whether the intervention is able to achieve its intended design goals(Table 3) (Additional file 3 shows our hypothesized relationships between context features and design seeds).
Table 3

Design Seeds Relationship to Critical Activity Categories and Implementation Context

Critical activity category

Design seed

Relevant Context Domains [39]

Safety Culture, Teamwork, Leadership

Structural Organizational Characteristics

External Factors

Implementation/Management Tools

Communicate/coordinate

Ability to control data access

X

X

X

X

Scheduling functionality

X

X

 

X

Assign roles and responsibilities

X

X

 

X

Triggered notifications

X

X

X

X

Patient activity

Patient support

X

X

X

X

Complete patient information

X

X

 

X

Review or enter data

Keeps list up-to-date

X

X

 

X

Standardized data entry

  

X

X

Complete data capture

 

X

X

X

Performance data

X

X

X

X

Track progress

Population registry functionality for high-risk patients

X

  

X

Figure out what patients are “on the list”

X

X

X

X

Customize the patient list

X

X

 

X

Legend: Design seeds correspond to the four critical activities performed by clinics. To maximize effectiveness in diverse and dynamic settings, designed interventions are considered within the context of a larger work system, split into four major domains by Taylor et al. Hypothesized relationships between context features (e.g., leadership at unit level, local tailoring of intervention) within the four context domains and each design seed are shown in Additional file 3

Fig. 4

a Technical Intervention Development Cycle: Example. b Socio-Technical Intervention Development Cycle: Design Seed Example. The Pivotal Role of Design Seeds for Intervention Development: Design seeds offer an important bridge between identifying problems and solutions. For the design seed– a population registry for high-risk patients – the diagram shows the evaluable components of the design seed in the light blue boxes. At the solution development stage, different technical and/or organizational interventions can be tested to see if they meet the design seed requirement. Solution components, shown in dark blue boxes, that meet the design seed requirement (based on iterative testing) can be assembled as a comprehensive intervention for further testing and deployment

When clinic participants were presented the preliminary findings from this formative research, they were asked to prioritize the design seeds and suggest revisions of the design seed descriptions. Twelve of the 13 design seeds received ranking in the top 5 in at least one clinic (Table 4). Although the design seed for scheduling functionality did not receive a top 5 ranking, four clinics ranked it seventh, right in the middle of the list, so it was hardly a low priority. The design seed for keeping the list of patients who need monitoring up-to-date received top three or better ranking at four clinics. Three other design seeds – triggered notifications, customize the patient list, and population registry functionality – were ranked in the top 5 by three or more clinics. As noted by one participant, the high priority seeds or solution attributes “were those that bring the right information to the right person at the right time.” Some design seeds had higher salience for only one clinic. Complete data capture, for example, is more problematic for clinics whose patients receive some of their care at other institutions that use other record-keeping systems and EHRs. In these instances, patient data is either inaccessible or must be faxed between sites.

Based on average ratings shown in Table 4, as well as individual clinic ratings, the design seeds received agreement that they would improve monitoring and save time in most clinics. Only one design seed (assigning roles) received disagreement for improving monitoring and reducing time in one clinic because all monitoring is performed by a “one-woman show” (a registered nurse). All design seeds except one (patient support) received the most favorable rating (5) for time saved by at least one clinic. No clinic provided very strong agreement (5) that the design seed for performance data would improve monitoring, although representatives of three clinics (breast, GI and urology) agreed that it would improve monitoring (ratings of 4 on 5-point scale). As one respondent noted, “the scope of the problem would be good to know, but secondary to other needs.” This view is consistent with other studies showing frontline concern that monitored activities will be artificially prioritized over core clinical work [64]. Design seeds viewed as having higher impact potential for saving time and improving monitoring were generally ranked closer to the top by more clinics.
Table 4

Importance Ranking of Design Seeds from Five Specialty Clinics

Design Seed

Ranked in Top 5

Rank

(Avg)

Improved Monitoring

(Avg)

Reduce Time Spent

(Avg)

Keeps list up-to-date

P, G, E, U

3.4

4.6

4.8

Triggered notifications

B, G, E

4.2

4.8

4.8

Customize the patient list

B, P, G, U

5.2

4.2

4.6

Ability to control data access

E, U

6.2

4.4

4.2

Population registry functionality for high-risk patients

P, E, U

6.6

4.4

4.2

Complete patient information

G, E

7.2

4.6

4.6

Standardized data entry

G

7.2

4.2

4.4

Performance data

B

7.2

3.6

3.8

Patient support

B, P

7.8

4.2

3.6

Complete data capture

B

8

3.8

4.2

Scheduling functionality

-

8.4

4

4

Figure out what patients are “on the list”

P

9.8

4.2

4.2

Assign roles and responsibilities

U

9.8

3.4

3.6

Legend: Clinics designated as B = Breast, P = Pulmonary, G = GI, E = Ear Nose and Throat, U = Urology

Discussion

This research highlights the unique and innovative integrated application of methods drawn from human factors engineering (design seeds, process tracing analysis) and user experience studies (journey mapping) to derive context-sensitive and theory-based interventions at the local level. Such focused and potentially scalable work is particularly needed for patients who may be lost to follow-up in systems that are stretched for dollars and time. This project focused on high-risk patients, both clinically due to a potentially concerning finding during an outpatient visit, and due to challenges from a socio-demographic viewpoint. When a patient has a warning signal for a serious condition that has yet to materialize, but may in the future, the ability of a clinical team to watch the patient closely over time hinges on incredible vigilance on the part of individual clinicians - hardly an ideal solution.

These challenges mirror those reported in other health settings with incomplete documentation and limited knowledge of the magnitude of patient safety problems [57]. Providers will often create informal workarounds in response to the lack of comprehensive and coordinated record-keeping systems, which can result in errors as well as redundant efforts [65, 66]. Accompanied by an understanding of these workarounds, resource strapped settings offer a unique opportunity to apply user-centered approaches to redesign socio-technical strategies by integrating user and client needs, the possibilities of technology, and requirements for economic viability [67].

Through mapping how patients are currently monitored for specific high-risk conditions according to evidence-based practice in five specialty clinics in our safety-net setting, we identified 45 different vulnerabilities. Repeatedly, we heard that clinicians worry about properly tracking these patients, and are troubled by the significant personnel time required in carrying out patient-level monitoring activities without tools and organizational approaches for population-level monitoring. In addition, no ongoing performance data currently exists related to the frequency of missed opportunities to monitor these high-risk patients, though efforts are underway [27].

To ameliorate the difficulties identified, we worked iteratively with the clinics to develop the basis for a sound approach to population management of diagnostically high-risk patients. We adapted the journey mapping technique to capture activities and experiences of the clinic team as they manage cohorts of such patients, focusing on the clinician’s monitoring journey. Previous applications have focused on patient journeys and experiences. While each clinic had a different journey map, all teams carried out the same four basic functions with some variation in sequencing and specifics. For example, one function, ‘patient activities’, includes scheduling the patient, assisting patients with barriers to making it to a critical test, seeing the patient when they come into the clinic, conducting an imaging study, and so forth.

Once we understood the clinic teams’ many concerns, particularly the time implications of the current monitoring workload, as well as the potential for errors, we did not jump straight to solutions. The use of design seeds, as a bridge between problems and effective organizational interventions, offers three advantages to leaping over this step. First, design seeds are simple descriptions that state what a solution needs to do, and can be described in a way that allows validation by the users, those on the frontlines at the clinics. For example, clinicians can easily imagine scenarios where patients might not be monitored according to evidence-based guidelines because of ambiguity in who is responsible for tracking high-risk situations (i.e., addressed by the design seed for assigning roles and responsibilities between primary care practitioner and specialist for a patient flagged for further follow-up). Second, design seeds can be supplied to other clinics to learn whether they have face validity outside of this particular safety-net setting. Design seeds support flexibility and tailoring to context, a critical feature for effective implementation of patient safety interventions in different settings [39, 68]. Other clinics could use the feedback exercise to determine whether the 13 design seeds are perceived to improve monitoring and save time in their setting prior to investing in a solution. As a result, one organization could implement and test interventions based on one set of design seeds (e.g., #3, 5, and 7), while another might choose another set (e.g., #2, 3, 4, 6) based on differing contextual enablers, barriers and interactions [33, 37, 39, 40]. Third, design seeds are, by definition, an assessment tool during testing of potential solutions. Does the solution do what the design seed prescribed? Some of the design seeds may result in primarily HIT solutions (triggered notifications), while others may need significant organizational changes (patient support). But most, if not all, will likely require both technical and organizational change.

The use of design seeds, previously applied for complex cognitively rich tasks outside of health care, is adaptive to any organizational setting coordinating layers of cognitively taxing activities meant to accomplish a particular organizational goal [69, 70]. Health networks fragmented by technology, location, and organizational elements are ripe environments for the design seed method as it captures differences in context while moving towards a cohesive end-goal: a solution that works across settings while also targeting specific needs to provide high value to local settings. In our case, we sought to use journey mapping coupled with process tracing and design seeds to identify features of population management interventions for high-risk conditions and treatments to reduce diagnostic error. The flexible structure of these tools, anchored to touch points with end users, enable a generalizable strategy for identifying leverage points, reducing diagnostic delays related to suboptimal monitoring, and increasing organizational effectiveness.

Limitations

While designed for adaptability across systems, our proposed strategy for developing design seeds would be strengthened by further assessment within other health care systems. At this stage, we know from testing in multiple specialty settings that common themes and variations exist. While each of the clinics in this study has its own leadership, electronic and paper-based systems, and organizational design, we showed that journey mapping paired with process tracing captured both differences and similarities across five settings.

Because data collection was part of quality improvement work in a low resource setting, we relied on a small number of informants from each clinic. An additional limitation is that our design seeds have not received feedback from stakeholders outside the specialty clinic workforce (e.g. patients, information technology providers, caregivers). Any vulnerability tied to patient knowledge or preferences would best be elicited from patients directly, and most likely generate refinement of the patient support design seed. By focusing on the “holders” of the patient data – those stakeholders that most frequently engage with, and bear responsibility for, patient monitoring activities – we have established a foundation from which to build. The approach used fosters an iterative process for data collection that will loop in other stakeholders. Our adaptation of journey mapping and design seeds summarizes a broad, but possibly incomplete, list of activities related to patient monitoring when approached from a cohort perspective.

Future Work

We sought to draw from organizational analysis used outside of the healthcare setting to inform a practical and scalable intervention geared to reduce missed and delayed diagnosis in high-risk patient populations. Ideally, this approach would be replicated in other specialty areas and sites, including those that are better resourced. We will translate the validated design seeds into a prioritized list of solution attributes to use in development and evaluation of socio-technical interventions. During the organizational change process, we intend to continually reference and iterate journey maps. One of the design seeds – figure out what patients are “on the list” – will require work within the clinics as well as literature reviews targeted to trigger algorithms for identifying patients in need of close, but not urgent, follow-up during their diagnostic journeys [71, 72, 73]. We anticipate that interventions evaluated against user needs that are generated with intention and context will be more sustainable, user-friendly, and implemented more successfully than those generated without this human factors approach.

As a nascent area of research, strategies to close gaps in diagnostic safety built from the ground up, as in this study, will first be followed by pilot testing, and ultimately full-scale implementation evaluations with additional measures related to the people (patient, provider), organizational, technology and structural factors predicting desired implementation outcomes [74]. The NAM Improving Diagnosis framework shares a similar multi-level structure with those of implementation science, anticipating future research to improving diagnostic care in an organizationally effective and sustainable way.

Conclusions

We carried out a multi-stage research process with specialty clinics at an urban publicly funded health system to address an important evidence-based safety gap in ambulatory care: potentially preventable and consequential diagnostic and monitoring delays. Based on surfacing a large number of common vulnerabilities among the clinics, we specified and validated key attributes for a robust socio-technical approach to improving outpatient monitoring that is geared to enable context-sensitive implementation, utilizing industrial and human factors methods linked to implementation theory.

Footnotes

  1. 1.

    Patient monitoring for cancer, in this paper, is broadly construed to include an expansive set of diagnostic opportunities, not just one definitive and staged cancer diagnosis. Ambulatory safety risk in this context includes identification of high-risk patients, pre-diagnosis testing, definitive diagnostic procedures (e.g., biopsies), and even longitudinal post-diagnosis follow up (e.g., keeping track of patients for whom treatment is delayed on purpose, or following patients after treatment for cancer recurrence).

Notes

Acknowledgements

We would like to thank all interviewees.

Funding

This research is supported through a grant from the Agency for Healthcare Research and Quality (P30HS023558). AHRQ had no role in the design, data collection, analysis, or presentation of results. There are no conflicts of interests to disclose.

Availability of data and materials

All data generated or analyzed during this study are included in this published article [and its supplementary information files].

Authors’ contributions

US, EP, GS and KM conceptualized the study and guided the study design. GS, SL, and KM led interviews and performed data acquisition, data analysis, and interpretation. KM drafted the manuscript with support of SL. GS, EP, and US critically reviewed the manuscript. All authors have fulfilled the criteria for authorship established by the International Committee of Medical Journal Editors and approved submission of the manuscript. All authors read and approved the final manuscript.

Authors’ information

KM is the executive director of the Stanford Center for Health Policy and Center for Primary Care and Outcomes Research, and a senior scholar at the centers. Trained in organizational science, management and engineering, she studies ways that health care delivery systems innovate to achieve high levels of quality, safety, patient-centeredness, and workforce capacity.

GS is an Associate Professor of Medicine at the Division of Pulmonary and Critical Care, San Francisco General Hospital, and the University of California, San Francisco.

SL is a Research Analyst at UCSF in the Division of General Internal Medicine.

EP is an Associate Professor in the Division of Health Information Management and Systems in the School of Health and Rehabilitation Sciences, College of Medicine at The Ohio State University. Her research interests are human factors, informatics, and patient safety.

US is an Associate Professor at UCSF in the Division of General Internal Medicine and a primary care physician at Zuckerberg San Francisco General Hospital and Trauma Center’s Richard H Fine People’s Clinic. Dr. Sarkar’s research focuses on patient safety in outpatient settings, including adverse drug events, missed and delayed diagnosis, and failures of treatment monitoring.

Competing interests

The authors declare that they have no competing interests.

Consent for publication

Not applicable.

Ethics approval and consent to participate

Ethical approval for semi-structured interviews and feedback sessions with clinic personnel for quality improvement purposes was reviewed and waived by the institutional review board of the University of California, San Francisco.

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary material

13012_2017_609_MOESM1_ESM.docx (19 kb)
Additional file 1:Script for data collection on design seeds. (DOCX 19 kb)
13012_2017_609_MOESM2_ESM.pptx (136 kb)
Additional file 2:Feedback form for data collection on design seeds. (PPTX 135 kb)
13012_2017_609_MOESM3_ESM.docx (26 kb)
Additional file 3:Hypothesized relationships between context features within context domains and each design seed. (DOCX 25 kb)
13012_2017_609_MOESM4_ESM.pptx (109 kb)
Additional file 4:Additional workflows (breast, gastroenterology, urology) with targets. (PPTX 108 kb)
13012_2017_609_MOESM5_ESM.pptx (109 kb)
Additional file 5:All workflows color-coded as foundation for process trace sequences. (PPTX 109 kb)

References

  1. 1.
    Committee on Diagnostic Error in Health Care. Improving Diagnosis in Health Care. Washington, DC: The National Academies Press; 2015. http://www.nap.edu/catalog/21794/improving-diagnosis-in-health-care.Google Scholar
  2. 2.
    Newman-Toker DE, Pronovost PJ. Diagnostic errors--the next frontier for patient safety. JAMA. 2009;301:1060–2. doi:10.1001/jama.2009.249.CrossRefPubMedGoogle Scholar
  3. 3.
    Singh H, Meyer AN, Thomas EJ. The frequency of diagnostic errors in outpatient care: estimations from three large observational studies involving US adult populations. BMJ Qual Saf. 2014;23:727–31. doi:10.1136/bmjqs-2013-002627.CrossRefPubMedPubMedCentralGoogle Scholar
  4. 4.
    Hoffman J, editor. Annual Benchmarking Report: Malpractice Risks in the Diagnostic Process. Cambridge: CRICO Strategies; 2014;1–20.Google Scholar
  5. 5.
    Gandhi TK, Kachalia A, Thomas EJ, Puopolo AL, Yoon C, Brennan TA, et al. Missed and delayed diagnoses in the ambulatory setting: A study of closed malpractice claims. Ann Intern Med. 2006;145:488–96. doi:10.7326/0003-4819-145-7-200610030-00006.CrossRefPubMedGoogle Scholar
  6. 6.
    Schiff GD, Hasan O, Kim S, Abrams R, Cosby K, Lambert BL, et al. Diagnostic error in medicine: analysis of 583 physician-reported errors. Arch Intern Med. 2009;169:1881–7.CrossRefPubMedGoogle Scholar
  7. 7.
    Provenzale D, Gupta S, Ahnen DJ, Bray T, Cannon JA, Cooper G, et al. NCCN Clinical Practice Guidelines in Oncology (NCCN Guidelines) Colorectal Cancer Screening. 2016. https://www.nccn.org/professionals/physician_gls/f_guidelines.asp#colon.Google Scholar
  8. 8.
    Singh H, Daci K, Petersen LA, Collins C, Petersen NJ, Shethia A, et al. Missed opportunities to initiate endoscopic evaluation for colorectal cancer diagnosis. Am J Gastroenterol. 2009;104:2543–54. doi:10.1038/ajg.2009.324.CrossRefPubMedPubMedCentralGoogle Scholar
  9. 9.
    Mohler JL, Armstrong AJ, Bahnson RR, D’Amico A, Davis BJ, Eastham JA, et al. NCCN Clinical Practice Guidelines in Oncology - Prostate Cancer. 2016.Google Scholar
  10. 10.
    Pfister DG, Spencer S, Adelstein D, Adkins D, Brizel DM, Burtness B, et al. NCCN Clinical Practice Guidelines in Oncology - Head and Neck Cancers. 2016. https://www.nccn.org/professionals/physician_gls/f_guidelines.asp#head-and-neck.Google Scholar
  11. 11.
    Gradishar WJ, Anderson BO, Balassanian R, Blair SL, Burstein HJ, Cyr A, et al. NCCN Clinical Practice Guidelines in Oncology (NCCN Guidelines®) Breast Cancer. Version 2.2016. 2016. p. 1–202. https://www.nccn.org/professionals/physician_gls/f_guidelines.asp#breast.Google Scholar
  12. 12.
    Wood DE, Kazerooni EA, Baum SL, Eapen GA, Ettinger DS, Hou L, et al. National Comprehensive Cancer Network (NCCN) Clinical Practice Guidelines for Lung Cancer Screening. Thorac Surg Clin. 2015;25:185–97.CrossRefPubMedGoogle Scholar
  13. 13.
    Benson AB, Venook AP, Cederquist L, Chan E, Chen Y-J, Cooper HS, et al. Clinical Practice Guidelines in Oncology (NCCN Guidelines) Colon Cancer. 2016. https://www.nccn.org/professionals/physician_gls/f_guidelines.asp#colon.Google Scholar
  14. 14.
    Bevers TB, Helvie M, Bonaccio E, Calhoun KE, Daly MB, Farrar WB, et al. NCCN Clinical Practice Guidelines in Oncology (NCCN Guidelines) Breast Cancer Screening and Diagnosis. 2016. https://www.nccn.org/professionals/physician_gls/f_guidelines.asp#detection.Google Scholar
  15. 15.
    Carroll PR, Parsons JK, Andriole G, Bahnson RR, Castle EP, Catalona WJ, et al. Prostate cancer early detection, Version 2.2016: Featured updates to the NCCN guidelines. JNCCN J Natl Comprehensive Cancer Netw. 2016;14:509–19.CrossRefGoogle Scholar
  16. 16.
    Ettinger DS, Wood DE, Aisner DL, Akerley W, Bauman J, Bazhenova LA, et al. NCCN Clinical Practice Guidelines in Oncology (NCCN Guidelines) Non-Small Cell Lung Cancer. 2016. https://www.nccn.org/professionals/physician_gls/f_guidelines.asp#detection.Google Scholar
  17. 17.
    Kalemkerian GP, Loo BWJ, Akerley W, Attia A, Chow LQ, Decker R, et al. NCCN Clinical Practice Guidelines in Oncology (NCCN Guidelines) Small Cell Lung Cancer. 2016. https://www.nccn.org/professionals/physician_gls/f_guidelines.asp#detection.Google Scholar
  18. 18.
    McCarthy BD, Yood MU, Boohaker EA, Ward RE, Rebner M, Johnson CC. Inadequate follow-up of abnormal mammograms. Am J Prev Med. 1996;12:282–8. http://www.ncbi.nlm.nih.gov/pubmed/8874693.PubMedGoogle Scholar
  19. 19.
    Haas JS, Cook EF, Puopolo AL, Burstin HR, Brennan TA. Differences in the quality of care for women with an abnormal mammogram or breast complaint. J Gen Intern Med. 2000;15:321–8.CrossRefPubMedPubMedCentralGoogle Scholar
  20. 20.
    Chang SW, Kerlikowske K, Napoles-Springer A, Posner SF, Sickles EA, Perez-Stable EJ. Racial differences in timeliness of follow-up after abnormal screening mammography. Cancer. 1996;78:1395–402.CrossRefPubMedGoogle Scholar
  21. 21.
    Loeb S, Walter D, Curnyn C, Gold HT, Lepor H, Makarov DV. How Active is Active Surveillance? Intensity of Follow-Up During Active Surveillance for Prostate Cancer in the United States. J Urol. 2016;196:721–6. doi:10.1016/j.juro.2016.02.2963.CrossRefPubMedGoogle Scholar
  22. 22.
    Falchook AD, Hendrix LH, Chen RC. Guideline-discordant use of imaging during work-up of newly diagnosed prostate cancer. J Oncol Pr. 2015;11:e239–46. doi:10.1200/JOP.2014.001818.CrossRefGoogle Scholar
  23. 23.
    Blagev DP, Lloyd JF, Conner K, Dickerson J, Adams D, Stevens SM, et al. Follow-up of Incidental Pulmonary Nodules and the Radiology Report. J Am Coll Radiol. 2016;13:R18–24.CrossRefPubMedGoogle Scholar
  24. 24.
    Goldman LE, Walker R, Hubbard R, Kerlikowske K. Timeliness of abnormal screening and diagnostic mammography follow-up at facilities serving vulnerable women. Med Care. 2013;51:307–14.CrossRefPubMedPubMedCentralGoogle Scholar
  25. 25.
    Martin J, Halm EA, Tiro JA, Merchant Z, Balasubramanian BA, McCallister K, et al. Reasons for Lack of Diagnostic Colonoscopy After Positive Result on Fecal Immunochemical Test in a Safety-Net Health System. Am J Med. 2016;130:93.e1-93.e7Google Scholar
  26. 26.
    Oluloro A, Petrik AF, Turner A, Kapka T, Rivelli J, Carney PA, et al. Timeliness of Colonoscopy After Abnormal Fecal Test Results in a Safety Net Practice. J Community Health. 2016;41:864–70.CrossRefPubMedGoogle Scholar
  27. 27.
    Issaka RB, Singh MH, Oshima SM, Laleau VJ, Rachocki CD, Chen EH, et al. Inadequate Utilization of Diagnostic Colonoscopy Following Abnormal FIT Results in an Integrated Safety-Net System. Am J Gastroenterol. 2017;112:375-82. doi: 10.1038/ajg.2016.555.
  28. 28.
    Bickell NA, Moss AD, Castaldi M, Shah A, Sickles A, Pappas P, et al. Organizational Factors Affect Safety-Net Hospitals’ Breast Cancer Treatment Rates. Health Serv Res. 2016. [Epub ahead of print].Google Scholar
  29. 29.
    Wachter RM. Expected and unanticipated consequences of the quality and information technology revolutions. JAMA. 2006;295:2780–3. doi:10.1001/jama.295.23.2780.CrossRefPubMedGoogle Scholar
  30. 30.
    Ash JS, Berg M, Coiera E. Some unintended consequences of information technology in health care: the nature of patient care information system-related errors. J Am Med Inf Assoc. 2004;11:104–12.CrossRefGoogle Scholar
  31. 31.
    Moore GF, Audrey S, Barker M, Bond L, Bonell C, Hardeman W, et al. Process evaluation of complex interventions: Medical Research Council guidance. BMJ. 2015;350:h1258.CrossRefPubMedPubMedCentralGoogle Scholar
  32. 32.
    Kent EE, Mitchell SA, Castro KM, DeWalt DA, Kaluzny AD, Hautala JA, et al. Cancer Care Delivery Research: Building the Evidence Base to Support Practice Change in Community Oncology. J Clin Oncol. 2015;33:2705–11.CrossRefPubMedPubMedCentralGoogle Scholar
  33. 33.
    Kaplan HC, Provost LP, Froehle CM, Margolis PA. The Model for Understanding Success in Quality (MUSIQ): building a theory of context in healthcare quality improvement. BMJ Qual Saf. 2011;21:13 LP–20. http://qualitysafety.bmj.com/content/21/1/13.abstract.CrossRefGoogle Scholar
  34. 34.
    Carayon P, Hundt AS, Karsh B-T, Gurses AP, Alvarado CJ, Smith M, et al. Work system design for patient safety: the SEIPS model. Qual Saf Heal Care. 2006;15 Suppl I:i50–8.CrossRefGoogle Scholar
  35. 35.
    Carayon P, Wetterneck TB, Rivera-Rodriguez AJ, Hundt AS, Hoonakker P, Holden R, et al. Human factors systems approach to healthcare quality and patient safety. Appl Erg. 2014;45:14–25.CrossRefGoogle Scholar
  36. 36.
    Smith MJ, Sainfort PC. A balance theory of job design for stress reduction. Int J Ind Ergon. 1989;4:67–79. http://dx.doi.org/10.1016/0169-8141(89)90051-6.CrossRefGoogle Scholar
  37. 37.
    Croskerry P. Context Is Everything or How Could I Have Been That Stupid? Healthc Q. 2009;12 Sp:e171–6. doi: 10.12927/hcq.2009.20945.
  38. 38.
    Bruce BB, El-Kareh R, Ely JW, Kanter MH, Rao G, Schiff GD, et al. Methodologies for evaluating strategies to reduce diagnostic error: report from the research summit at the 7th International Diagnostic Error in Medicine Conference. Diagnosis. 2016;3:1–7.CrossRefGoogle Scholar
  39. 39.
    Taylor SL, Dy S, Foy R, Hempel S, McDonald KM, Ovretveit J, et al. What context features might be important determinants of the effectiveness of patient safety practice interventions? BMJ Qual Saf. 2011;20:611–7.CrossRefPubMedGoogle Scholar
  40. 40.
    Øvretveit J. Understanding the conditions for improvement: research to discover which context influences affect improvement success. BMJ Qual Saf. 2011;20 Suppl 1:i18 LP–-i23. http://qualitysafety.bmj.com/content/20/Suppl_1/i18.abstract.CrossRefGoogle Scholar
  41. 41.
    Richardson A. Using Customer Journey Maps to Improve Customer Experience. Harvard Business Review. 2010. https://hbr.org/2010/11/using-customer-journey-maps-to. Accessed 10 Aug 2016.
  42. 42.
    A Guide to Service Improvement. Edinburgh: NHS Scotland; 2005. http://www.gov.scot/resource/doc/76169/0019037.pdf. Accessed 10 Aug 2016.
  43. 43.
    Boyd H, McKernon S, Mullin B, Old A. Improving healthcare through the use of co-design. N Z Med J. 2012;125:76–87.PubMedGoogle Scholar
  44. 44.
    Toward a Veteran-Centered VA: Piloting Tools of Human-Centered Design for America’s Vets. U.S. Department of Veterans Affairs; 2014. http://www.innovation.va.gov/docs/Toward_A_Veteran_Centered_VA_JULY2014.pdf. Accessed 10 Aug 2016.
  45. 45.
    Militello LG, Patterson ES, Saleem J, Anders S, Asch S. Supporting Macrocognition in Health Care: Improving Clinical Reminders. In: Schraagen JM, Ormerod T, Lipshitz R, Militello LG, editors. Naturalistic Decision Making and Macrocognition. Hampshire: Ashgate; 2008. p. 203–20.Google Scholar
  46. 46.
    Patterson ES, Rogers ML, Tomolo AM, Wears RL, Tsevat J. Comparison of extent of use, information accuracy, and functions for manual and electronic patient status boards. Int J Med Inform. 2010;79:817–23. doi: 10.1016/j.ijmedinf.2010.08.002.
  47. 47.
    Patterson ES, Woods DD, Roth EM. Using Cognitive Task Analysis (CTA) to Seed Design Concepts for Intelligence Analysts Under Data Overload. Hum Factors Ergon Soc Annu Meet Proc. 2001;45. doi:10.1177/154193120104500437.
  48. 48.
    Begun JW, Zimmerman B, Dooley K. Health Care Organizations as Complex Adaptive Systems. Adv Heal Care Organ Theory. 2003;33:253–88. doi:10.1177/009430610403300325.Google Scholar
  49. 49.
    Sittig DF, Singh H. A new sociotechnical model for studying health information technology in complex adaptive healthcare systems. Qual Saf Heal Care. 2010;19 Suppl 3:i68–74.CrossRefGoogle Scholar
  50. 50.
    Ehrlich S. Zuckerberg San Francisco General FY1516 Annual Report. San Francisco; 2016. https://www.sfdph.org/dph/files/hc/HCAgen/HCAgen2016/Nov%2015/01a%20ZSFG%20FY1516%20Annual%20Report_20161103.pdf.
  51. 51.
    San Francisco Department of Public Health Annual Report 2015-2016. San Francisco; 2016. https://www.sfdph.org/dph/files/reports/PolicyProcOfc/SFDPH-AnnualReport-2015-2016.pdf. Accessed 17 Jan 2017.
  52. 52.
    Hoehn RS, Wima K, Vestal MA, Weilage DJ, Hanseman DJ, Abbott DE, et al. Effect of hospital safety-net burden on cost and outcomes after surgery. JAMA Surg. 2016;151:120–8. doi:10.1001/jamasurg.2015.3209.CrossRefPubMedGoogle Scholar
  53. 53.
    Wakeam E, Hevelone ND, Maine R, Swain J, Lipsitz SA, Finlayson SR, et al. Failure to rescue in safety-net hospitals: Availability of hospital resources and differences in performance. JAMA Surg. 2014;149:229–35. doi:10.1001/jamasurg.2013.3566.CrossRefPubMedGoogle Scholar
  54. 54.
    Vest JR, Gamm LD. A critical review of the research literature on Six Sigma, Lean and StuderGroup’s Hardwiring Excellence in the United States: the need to demonstrate and communicate the effectiveness of transformation strategies in healthcare. Implement Sci. 2009;4:35. doi:10.1186/1748-5908-4–35.
  55. 55.
    Saber Tehrani AS, Lee H, Mathews SC, Shore A, Makary MA, Pronovost PJ, et al. 25-Year summary of US malpractice claims for diagnostic errors 1986–2010: an analysis from the National Practitioner Data Bank. BMJ Qual Saf. 2013;22:672–80. doi:10.1136/bmjqs-2012-001550.CrossRefPubMedGoogle Scholar
  56. 56.
    Franco J, Elghouche AN, Harris MS, Kokoska MS. Diagnostic Delays and Errors in Head and Neck Cancer Patients: Opportunities for Improvement. Am J Med Qual. 2016;32:330–35.Google Scholar
  57. 57.
    Paul MM, Greene CM, Newton-Dame R, Thorpe LE, Perlman SE, McVeigh KH, et al. The state of population health surveillance using electronic health records: a narrative review. Popul Heal Manag. 2015;18:209–16.CrossRefGoogle Scholar
  58. 58.
    Carayon P Cartmill RS, et al KB. Incorporating Health Information Technology Into Workflow Redesign--Summary Report. Rockville: Agency for Healthcare Research and Quality; 2010.Google Scholar
  59. 59.
    Lowry SZ, Patterson ES, Brick D, Gurses AP, Ozok A, Simmons D, et al. Integrating Electronic Health Records into Clinical Workflow: An Application of Human Factors Modeling Methods to Ambulatory Care. NIST Interagency/Internal Rep. 2014;7988.Google Scholar
  60. 60.
    Sarkar U, Simchowitz B, Bonacum D, Strull W, Lopez A, Rotteau L, et al. A Qualitative Analysis of Physician Perspectives on Missed and Delayed Outpatient Diagnosis: The Focus on System-Related Factors. Jt Comm J Qual Patient Saf. 2014;40:461–70. https://doi.org/10.1016/S1553-7250(14)40059-X. Accessed 10 Aug 2016.
  61. 61.
    Woods DD. Process-tracing methods for the study of cognition outside of the experimental psychology laboratory. In: Klein GA, Calderwood R, Zsambok CE, Orasanu J, editors. Decision Making in Action: Models and Methods. Norwood: Ablex Publishing Corporation; 1993. p. 228–51.Google Scholar
  62. 62.
    Cahill J, McDonald N, Losa CG. A Sociotechnical Model of the Flight Crew Task. Hum Factors J Hum Factors Ergon Soc. 2014;56:1337–63. doi:10.1177/0018720814532684.CrossRefGoogle Scholar
  63. 63.
    Middleton B, Bloomrosen M, Dente MA, Hashmat B, Koppel R, Overhage JM, et al. Enhancing patient safety and quality of care by improving the usability of electronic health record systems: recommendations from AMIA. J Am Med Informatics Assoc. 2013;20:e2–8. doi:10.1136/amiajnl-2012-001458.CrossRefGoogle Scholar
  64. 64.
    Patterson ES, Cook RI, Render ML. Improving patient safety by identifying side effects from introducing bar coding in medication administration. J Am Med Inf Assoc. 2002;9:540–53. doi: 10.1197/jamia.M1061.
  65. 65.
    Cresswell KM, Mozaffar H, Lee L, Williams R, Sheikh A. Safety risks associated with the lack of integration and interfacing of hospital health information technologies: a qualitative study of hospital electronic prescribing systems in England. BMJ Qual Saf. 2016. doi:10.1136/bmjqs-2015-004925.Google Scholar
  66. 66.
    Cresswell KM, Mozaffar H, Lee L, Williams R, Sheikh A. Workarounds to hospital electronic prescribing systems: a qualitative study in English hospitals. BMJ Qual Saf. 2016. doi:10.1136/bmjqs-2015-005149.Google Scholar
  67. 67.
    Flanagan ME, Saleem JJ, Millitello LG, Russ AL, Doebbeling BN. Paper- and computer-based workarounds to electronic health record use at three benchmark institutions. J Am Med Inf Assoc. 2013;20:e59–66.CrossRefGoogle Scholar
  68. 68.
    Braithwaite J, Marks D, Taylor N. Harnessing implementation science to improve care quality and patient safety: a systematic review of targeted literature. Int J Qual Heal care J Int Soc Qual Heal Care. 2014;26:321–9.CrossRefGoogle Scholar
  69. 69.
    Simon HA. Administrative Behavior. A Study of Decision-Making Processes in Administrative Organization. 3rd ed. London: The Free Press, Collier Macmillan Publishers; 1976.Google Scholar
  70. 70.
    Rasmussen J. Skills, Rules, and Knowledge; Signals, Signs, and Symbols, and other Distinctions in Human Performance Models. IEEE Trans Syst Man Cybern. 1983;13:257–66.CrossRefGoogle Scholar
  71. 71.
    Murphy DR, Meyer AN, Bhise V, Russo E, Sittig DF, Wei L, et al. Computerized Triggers of Big Data to Detect Delays in Follow-up of Chest Imaging Results. Chest. 2016;150:613–20.Google Scholar
  72. 72.
    Murphy DR, Laxmisan A, Reis BA, Thomas EJ, Esquivel A, Forjuoh SN, et al. Electronic health record-based triggers to detect potential delays in cancer diagnosis. BMJ Qual Saf. 2014;23:8–16. doi:10.1136/bmjqs-2013-001874.CrossRefPubMedGoogle Scholar
  73. 73.
    Murphy DR, Thomas EJ, Meyer AN, Singh H. Development and Validation of Electronic Health Record-based Triggers to Detect Delays in Follow-up of Abnormal Lung Imaging Findings. Radiology. 2015;277:81–7.CrossRefPubMedPubMedCentralGoogle Scholar
  74. 74.
    Chaudoir SR, Dugan AG, Barr CHI. Measuring factors affecting implementation of health innovations: a systematic review of structural, organizational, provider, patient, and innovation level measures. Implement Sci. 2013;8:1–20. doi:10.1186/1748-5908-8-22.CrossRefGoogle Scholar

Copyright information

© The Author(s). 2017

Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.

Authors and Affiliations

  • Kathryn M. McDonald
    • 1
    • 2
  • George Su
    • 3
  • Sarah Lisker
    • 3
  • Emily S. Patterson
    • 4
  • Urmimala Sarkar
    • 3
  1. 1.University of California Berkeley, School of Public HealthBerkeleyUSA
  2. 2.Stanford University, Center for Health Policy/Center for Primary Care and Outcomes ResearchStanfordUSA
  3. 3.Department of Medicine, School of MedicineUniversity of California San FranciscoSan FranciscoUSA
  4. 4.Ohio State University, College of Medicine, School of Health and Rehabilitation SciencesDivision of Health Information Management and SystemsColumbusUSA

Personalised recommendations