INTRODUCTION

Integrating evidence-based innovations (EBIs) into sustained use is a well-documented challenge for health systems. Organizational leaders report that most efforts to implement change fail.1, 2 Implementation science (IS) emphasizes closing the gap between evidence and practice,3 usually within a prescribed period of time.4 For example, obesity and related lifestyle behaviors like poor diet and physical inactivity impact life expectancy. Among US veterans, the prevalence of obesity has been documented at 41%, higher than that of the US population and higher than that of the previous decade. Despite strong evidence that comprehensive lifestyle interventions (CLIs) are effective treatment for obesity, these interventions are not reliably delivered to patients who would benefit.

The Veterans Health Administration (VHA) is one of the largest integrated health systems in the world, serving about six million enrolled veterans.5 In 2006, VA established the MOVE! Weight Management Program for Veterans (MOVE!), aligned with the strong evidence base for CLIs.6, 7 MOVE! has demonstrated modest short-term weight loss.8, 9 Though coordinators who lead local MOVE! programs at facilities across VHA have access to the robust CLI and MOVE! evidence base, plus implementation resources,10 program delivery is highly variable across VHA.11, 12 While local clinicians have worked diligently to implement MOVE! at their respective facilities, much work remains to optimize program delivery to improve patient outcomes.

We have published four implementation evaluations of four CLIs, including MOVE!.12,13,14,15 Each evaluation was guided by the Consolidated Framework for Implementation Research (CFIR),16 which describes contextual factors that arise as barriers to successful program implementation. Our evaluations consistently revealed barriers related to lack of the following: (1) planning, (2) engaging key individuals, and (3) reflecting and evaluating on progress and impact. The CFIR highlights that these activities “…can be accomplished in any order [using an] incremental approach to implementation; e.g., using a plan-do-study-act [PDSA] approach to incremental testing. [emphasis added].”16 These findings reinforce our contention that frontline staff should drive efforts to achieve optimal program implementation to benefit their patients.

The Dynamic Sustainability Framework (DSF)17 is a helpful conceptual framework for guiding optimized implementations to overcome identified barriers (Fig. 1). The DSF asserts that organizational learning should be a core value because of the need to continue to optimize the intervention over time. Thus, implementation cannot be just a one-time burst of effort but rather a dynamic process that engages teams over time. It places PDSA cycles of change at the heart of sustained implementation and reinforces that PDSAs are critical for accomplishing planning, engaging key individuals, and reflecting and evaluating progress and impact. With its focus on PDSAs, the DSF points squarely to the use of quality improvement (QI) techniques as a means of sustaining CLI implementation. Through PDSA cycles, frontline providers can systematically identify and address the barriers that prevent MOVE! from being optimally delivered. Based on the DSF, fully optimized implementation of MOVE! will lead to better patient outcomes.6, 7 Our goal was not to change the fundamental content and structure of MOVE! but rather to address variability in program delivery because, based on our earlier work, it was apparent that clinicians struggled to optimize their program to adapt to local needs and align with clinical practice guidelines for weight management.

Figure 1
figure 1

Dynamic Sustainability Framework (DSF).

Full size image

Drawing on the DSF, the objective of this project was to train frontline clinical teams in QI to dynamically optimize MOVE! delivery. In busy clinical work environments, MOVE! clinicians need a QI training program that meets the following criteria: (1) provides easy-to-understand and accessible content, (2) allows hands-on learning within a busy clinical setting, and (3) provides coaching support and a supporting learning community to enhance learning and accountability. The specific aim of this study was to evaluate the “Learn. Engage. Act. Process.” (LEAP) program, which was developed to address these requirements.

METHODS

This is a pre-post study to assess the capability of teams to conduct PDSA cycles and improve readiness for implementation. This study was deemed a non-research operations activity and follows the Standards for Reporting Implementation Studies (StaRI).18

Team Recruitment

MOVE! program sites were selected via random selection, stratified by patient participation rate (calculated as the number of patients who completed at least one group-based MOVE! session in the first six months of FY 2016 divided by the number of patients eligible for MOVE!). MOVE! programs were categorized as high reach (above median participation rate) versus low reach (below median participation rate). Half of MOVE! programs were randomly selected to receive an e-mail invitation to participate in LEAP in fiscal year 2017. Twenty-four programs were randomly selected from the list of MOVE! coordinators who expressed interest in participating in LEAP and assigned to one of four starting dates.

LEAP Intervention

LEAP is a virtual, structured training program that trains frontline teams how to run PDSAs through coaching with hands-on learning as teams develop and execute a Project Charter (see Appendix 1) to complete a PDSA cycle of change. Teams were assigned to cohorts with five teams from other MOVE! programs within VHA, to establish a learning community. The LEAP curriculum was adapted for teams and streamlined for busy clinical settings based on a Massive-Open Online Course (MOOC) developed by HarvardX in collaboration with IHI.19 Appendix 2 provides a brief summary of LEAP’s curriculum.

LEAP spanned 21 weeks to give teams the time to learn and apply new skills while simultaneously executing an initial cycle of change. LEAP includes hour-long coaching calls or virtual collaboratives (VCs) involving teams with MOVE! programs from other facilities each week, to enhance learning and accountability. Teams are encouraged to convene local weekly meetings to learn and apply the curriculum to their own project; teams complete assignments, review data, and plan for future PDSAs. Teams did not have formally allocated time to dedicate to LEAP or QI activities. Each individual, however, committed to carving out 2–4 h per week for LEAP. At the end of LEAP, all teams gave a final presentation in a VC and received feedback from their coaches and peer teams within the collaborative. The final presentations included plans for their next PDSA cycle; each team was given a completion certificate and a copy of The Improvement Guide.20

Data-driven PDSAs require program metrics to help teams identify improvement opportunities. A user-centered design approach guided development of audit and feedback reports for MOVE! teams. Personas21 were developed based on semi-structured interviews with MOVE! program leaders to capture their data and reporting needs. The study team designed reports based on this input and refreshed them monthly or quarterly.

Measures

Organizational Readiness for Implementing Change

Organizational readiness for implementing change (ORIC) refers to team members’ shared determination to implement a change (change commitment) and shared belief in their collective capability to do so (change efficacy).22 Shea and colleagues developed and validated a 12-item instrument with two scales: (1) collective change efficacy (seven items, Cronbach’s reliability ɑ = 0.93) and (2) collective change commitment (five items, Cronbach’s reliability ɑ = 0.95). A 5-point Likert scale ranging from disagree to agree (1 to 5 points) was used to assess each item (see Appendix 3).

Self-Rated Quality Improvement Skills

A self-assessment, adapted from The Improvement Guide20 and the IHI Improvement Coach Professional Development program,23 was used to elicit participant ratings of confidence in applying QI methods using a six-point scale ranging from “no knowledge” to “expert” (see Appendix 4). The survey consisted of 19 items across the following topics: (1) support a change with data (4 items), (2) develop a change (4 items), (3) test a change (3 items), (4) implement a change (2 items), (5) spread a change (2 items), and (6) the human side of change (4 items).

Assignment Completion and Aim Achievement

Assignment completion was tracked based on materials uploaded by each team to the online platform. Aim achievement was determined based on each team’s final presentation of progress toward achieving their aim. Teams were encouraged to collect data for at least 12 time points to assess reliability.

Program Satisfaction

Upon completion of LEAP, a 21-item survey of satisfaction across five LEAP domains (improvement coach support, quality of curriculum materials, organization of materials online, number of assignments, and technology requirements of the program) and future intentions to continue with QI was administered to team members (see Appendix 5). Comment space was available for additional feedback. Responses were summarized for each cohort and shared in the final week’s VC.

Data Collection

Surveys were administered online via Qualtrics survey software (Qualtrics, LLC, Provo, UT). Surveys of organizational readiness and QI skills were completed in weeks 4 and 6 of LEAP, respectively, and again at the end of LEAP. The satisfaction survey was administered in week 20.

Data Analysis

Descriptive statistics were generated for all measures. Paired t tests were used to test for differences in readiness and QI skill ratings from early in the program to completion of LEAP. Analyses were conducted using SAS 9.4 (SAS Institute, Inc., Cary, NC).

RESULTS

Participation

Invitations were emailed to MOVE! coordinators of 68 randomly selected group MOVE! programs. As shown in Figure 2, 29 coordinators (43%) responded “yes” and 24 coordinators were randomly chosen and assigned to one of the four start dates; of these, 71% (n = 17) participated. Table 1 characterizes the 17 MOVE! programs involved. Contrary to evidence-based policy guidance, 59% (10/17 teams) reported open (instead of closed) enrollment, and 30% (5/17 teams) offered fewer than the recommended 16 sessions in their standard curriculum.

Figure 2
figure 2

MOVE! Program team recruitment flow.

Full size image
Table 1 Characteristics of Participating Programs and Teams (n = 17)
Full size table

LEAP coaches worked with the team leader (the MOVE! coordinator was nearly always the team leader for LEAP) to build their team by inviting colleagues. Altogether, 97 individuals participated across the 17 teams, an average of 5.8 (1–9) individuals per team. Most LEAP team members were dietitians or nurses (see Table 2).

Table 2 Participant Profession
Full size table

Self-Rated Quality Improvement Skills

Fifty-five participants, including all team leaders, completed the baseline self-rated QI skills assessment; 40 participants completed the follow-up assessment. Table 3 shows scores for baseline and follow-up for the 28 participants who completed the survey at both time points. Each of the measures increased significantly from early participation in LEAP to its completion. There were no significant differences in responses from participants who only responded at baseline (n = 27) versus those who completed both time points (n = 28).

Table 3 Organizational Readiness for Implementing Change (ORIC) and Self-Rated QI Skill Assessment
Full size table

Organizational Readiness for Implementing Change

There was a marginally significant increase in change efficacy (p < 0.08) but not change commitment (p = 0.66) (see Table 3). There were no significant baseline differences for individuals who only responded at baseline (n = 16) versus those who completed both time points (n = 23).

Assignment Completion and Aim Achievement

Assignment completion varied widely from 35 to 100% (Table 4). For example, all teams uploaded their project charter, but only 35% uploaded a process map. Completion rates may be underestimated because teams completed assignments but did not upload them to the online platform.

Table 4 Completion of LEAP Assignments
Full size table

Teams developed improvement aims related to eight topics; most focused on increasing MOVE! enrollments (n = 11). Teams also worked to improve patient retention, decrease wait times, improve documentation of weights, improve weight outcomes, increase patient goal setting, or increase patients’ use of logs to record physical activity. Nine teams reported they had achieved their aim, though some of those teams did not document enough data points to confirm.

LEAP Program Satisfaction

Altogether, 55 out of 97 (57%) LEAP team members completed the satisfaction survey. Most participants were satisfied or very satisfied (81–89%) with all LEAP components. Comments described technical issues with navigating the online platform and unreliability and unavailability of webcams as common issues.

Though almost all respondents (96%) agreed or strongly agreed that LEAP was relevant to the needs of their MOVE! program, 53% disagreed that they had enough time to do the required work. Respondents commented that finding time and available staff to participate on the team were significant challenges.

Despite the reported lack of time, most respondents agreed or strongly agreed that they would have time to continue to apply LEAP methods to their MOVE! programs in the future (74%), their LEAP team would continue working together after completing LEAP (81%), and they planned to participate in future monthly sessions (57%). This was reinforced by comments that expressed a strong desire for continued availability of LEAP coaches, data reports, and help interpreting data to inform future improvements.

DISCUSSION

This is a story about how implementation researchers, guided by the DSF, landed in the sphere of quality improvement in their quest to increase team readiness to optimize an evidence-based CLI within challenging clinical settings. We began by drawing on IS findings informed by the CFIR that pointed to key barriers. The CFIR and DSF frameworks pointed to a strategy of engaging teams in PDSAs as a pathway to optimize CLIs. This led to the design of the LEAP QI training program for frontline teams. Thus, LEAP is rooted within the intersection of IS and QI. LEAP draws on QI curriculum adapted for hands-on learning by frontline teams. Weight management is a function of both individual behavior change by patients and CLIs that are designed to help patients achieve outcome goals.6 LEAP increased capability of clinical teams to dynamically optimize MOVE! over time; the DSF asserts this is essential for continued program optimization.17

This study amplifies the challenge of integrating precepts of IS with QI. QI approaches are rooted in local expertise, leading to “bottom-up,” team-derived goals.24 In contrast, IS goals are often imposed “top-down” based on getting evidence into practice.4 The improvement aims teams selected often focused on increasing enrollments in reaction to disruptive system changes that dramatically impacted their referral processes. During this study, a “direct scheduling” approach was mandated so patients could enroll in MOVE! without a referral; enrollment workflows had to be redesigned to inform, invite, and assist patients. This was high priority for teams versus an IS goal of implementing/optimizing an evidence-based program component (e.g., closed versus open enrollment). Contributing processes like enrollment workflow are often overlooked in intervention design.25, 26 In fact, a recent implementation study involving over 1700 primary care practices recommended filling a void in a widely used compilation of implementation strategies27, 28 by adding workflow redesign to the list.29

It will take time to see measurable impacts on clinical outcomes (e.g., weight loss), which are commonly the focus of IS projects that are usually funded for a time period inadequate to capture slower but, in theory, better optimized and more sustainable implementations.4, 17 Not all LEAP teams met their aim, but even when teams did not achieve their aim, they communicated valuable lessons learned and felt better positioned to plan and execute future PDSAs; this is one of the main goals of PDSAs.24

The context in which LEAP was delivered is extremely important to consider. The MOVE! weight management program was not a high priority in VHA, relative to other higher priority issues, such as increasing access for patients,30 reducing suicide rates,31 addressing the opioid epidemic,32 and increasing access to community providers,33 all of which experienced focused media attention within and outside VHA during the course of this study. Regardless of topic, many organizations fail to sufficiently support investments in continuous QI by frontline clinicians/staff.34,35,36 Thus, a key design goal for LEAP was to coach frontline teams without requiring formal leadership commitment, leveraging a bottom-up “learn by doing” approach that could be accomplished within a demanding clinical setting and in the face of invisibility with mixed levels of support. Even with this approach, teams struggled with finding time to accomplish the work. Based on input from participating teams, we plan to extend the duration of LEAP to 26 weeks (without adding more content) to allow more time for teams to implement their planned change(s).

QI training is often delivered through intensive in-person, multi-day workshops or collaboratives requiring face-to-face interaction with other teams and coaches.37,38,39 Workshop-based training often fails to show impact because trainees do not apply this new knowledge within everyday clinical work.40,41,42,43 Virtual (internet- and/or phone-enabled) adaptations of collaboratives are increasingly used to mitigate participation barriers but often still require dedicated learning sessions and often last a year or more.44,45,46,47,48,49,50 One large-scale quality collaborative in VHA provided virtual monthly coaching but only after teams started executing their planned PDSA cycle.47, 48 LEAP coaches, in contrast, walked teams through every step, week by week, tailoring guidance to the needs of teams and providing feedback on interim progress. LEAP curriculum was paced; though team leaders might spend up to four hours per week on learning and activities, team members usually spent much less time.

Health system leaders are putting increasing attention and urgency on engaging frontline teams in QI to provide greater participation and control over clinical processes, and as a potential antidote for clinician burnout.51, 52 System leaders, however, continue to struggle to accomplish this34,35,36, 53 though there are pockets of success.52, 54, 55 More organizations are recognizing that investing in QI is “an enlightened strategy, not an expense;52, p51” LEAP can help engage teams in QI and increase readiness for implementing change in a constantly changing environment.17 We plan to partner with VHA Lean belt-trainers to understand circumstances under which Lean and LEAP produce team engagement in continuous QI. Continuous QI is increasingly pursued by healthcare systems41, 43, 56,57,58; it aligns with the assertions of the DSF to optimize interventions,17 and was a core strategy in one large-scale implementation initiative involving over 1700 primary care practices in the USA, but also encountered challenges.29

This study has several limitations. The goal of LEAP was to increase QI capability among teams leading the MOVE! weight management program. Developing this capability does not, however, address capacity to engage in QI, which requires a well-rooted learning culture and/or alignment with organizational priorities. LEAP did increase QI capability, but measurable impact on clinical outcomes will take time and will only happen if teams continue to engage in PDASs. Future work needs to focus on how to develop stronger alignment between frontline QI foci and organizational priorities to help resolve time constraints and the pressure from competing priorities experienced by teams.41, 55, 59

CONCLUSION

LEAP was developed to address implementation gaps identified through IS-guided evaluations and embraces QI as a foundational strategy for frontline clinical teams to optimize an evidence-based intervention. LEAP is scalable and cost-saving (no travel required), does not require absence from clinical responsibilities, and increased teams’ QI capability. However, the capacity of teams to continue QI is challenged by nearly universal time constraints and competing clinical priorities.