INTRODUCTION

Since its inception1 in the early 1990s, primary care has been the entry point to coordinated care for Veterans, offering long-term relationships with specialists and providers in the Veterans Health Administration (VHA) http://www.who.int/topics/primary_health_care/en/.2,3,4 Although the VHA primary care system has been called “The Best Care Anywhere,”5 accessing such care can be complex, unreliable, and dissatisfying for Veterans.6 As a result, transformation to a more patient-centered culture is a top VHA organizational priority.

The Patient-Centered Medical Home (PCMH) Model7,8,9,10,11,12,13 is a systematic approach to coordinating and integrating services that ensures optimal value and health outcomes. Based on Veterans’ feedback, VHA developed their version of the PCMH Model, called “Patient Aligned Care Team” (PACT), to improve Veteran experience, staff satisfaction, and quality of primary care.14,15,16 To address the challenge of implementing PACT in an estimated 7,200 VHA practices at approximately 1,700 sites, VHA strategically designed and implemented a Collaborative involving PACT leadership and front-line teams from every medical center (approximately 1,500 participants) over a 21-month period. This manuscript describes the VHA’s experience utilizing the Collaborative method to spread the PACT Model—including methods of evaluation, results, and lessons learned—in order to provide information to others considering wide-scale improvement initiatives.

PACT (MEDICAL HOME) MODEL

After reviewing PCMH literature and standards,17,18,19,20,21,22 the national VHA leadership team created a PACT conceptual model designed as a “house” (Fig. 1) to guide and facilitate improvement efforts. The foundation is based on a patient-centered approach, critical resources, and Vision-Analysis-Team-Aim-Map-Measure-Change-Sustain (VA-TAMMCS) improvement framework23 (Appendix A; all appendixes available online). The model has three pillars: 1) Access; 2) Care Coordination and Management; and 3) Practice Redesign. The respective goals of the pillars are to: 1) achieve optimal access by offering patients single or group appointments and non-appointment options; 2) ensure that high-risk transitions, or “hand-overs,” are managed appropriately, providing a continuous link with core teams even when care is provided at a facility away from the primary care location; and 3) improve communication and work processes, ensuring the right person does the right work at the right time. Teams were required to achieve seven aims with associated measures (Appendix B). Industrial engineers (IEs) and coaches assisted with map and measure processes, and improvement potentials within each pillar. Teams were given many resources (Table 1), such as a “change package” providing suggestions for improvement within each pillar (Appendix C).

Figure 1
figure 1

PACT (Medical Home) Model.

Full size image
Table 1 PACT Collaborative Resources (People, Tools, and Technology)
Full size table

This study is an exploratory evaluation of VA’s PACT Collaborative, a major component of VA’s national PACT implementation strategy. This evaluation asks whether participants found the Collaborative process useful, whether participants carried out the Collaborative’s improvement activities and achieved its aims, and whether adherence to the PACT model improved nationally during the Collaborative period.

PACT COLLABORATIVE INTERVENTION METHODS

PACT Collaborative Model

PACT Collaborative was modeled after the Institute for Healthcare Improvement’s (IHI) Breakthrough Series (BTS) Collaborative Model24,25 (Appendix D). A National Collaborative Leadership Team (NCLT) designed the intervention by considering resource constraints, complexity, scale of desired change, and anticipated time required to make changes.

Participants

The NCLT divided all VHA facilities into five regions (26–33 hospitals each) and provided guidance on identifying teams at each facility (seven to nine member) to travel to their regional PACT Collaborative. Teams (Appendix E) were comprised of leadership (i.e. associate chief of staff/nurse manager/service line manager for Primary Care) and direct-care staff (i.e. MD, RN, LPN/MOA, clerical staff and pharmacy, social work and behavioral health staff, as appropriate).

Coaches and Faculty

The NCLT recruited regional co-directors, co-coordinators, coach captains, faculty and coaches predominately from VHA with outstanding leadership and teaching skills, who participated in past national improvement programs, and agreed to lead the effort as a collateral duty. Five IEs and five coordinators (one for each region) were recruited from the Veteran Engineering Resource Centers to serve as coaching, teaching, and process improvement experts. Based on the Collaborative’s seven aims, the IEs developed standardized measurement tools (in Microsoft EXCEL) to collect data, track improvement progress, and make process improvement decisions. Most of the Collaborative leadership had no prior experience leading a collaborative; therefore, NCLT offered just-in-time training and resources. Most of the IEs had limited experience with healthcare, so they were partnered with VHA coaches to provide a more effective coaching experience for the teams. Each of the five regions identified a “coach captain” to be the master-level resource for approximately ten coaches per region. They led teaching calls, answered questions, problem-solved, and provided feedback.

Setting

Learning Sessions (LSs) were the major integrative events of the Collaborative, where new material was introduced, past material was reviewed, questions were answered, best practices were shared, and action plans were made. Prior to the LSs, the seven aims were shared with participants at the prework26 stage. Due to the complexity of PACT, the NCLT decided on six strategically timed LSs (compared to 3–4 in the IHI model). The regional Collaborative leadership planned and held six 3-day, face-to-face LSs, each in a different city within the five regions (totaling 30 LSs nationally) from June 2010 through February 2012 (Fig. 2).

Figure 2
figure 2

Schematic of the PACT Collaborative Model.

Full size image

To keep the regional Collaborative consistent with the national plan, approximately 80 % of agenda content was consistent across all regions, while 20 % was elective content planned by regional leadership. NCLT conducted bi-monthly calls to plan, educate and guide regional leadership. Across each region, PACT material was introduced sequentially, with LS1 focusing on the first pillar, LS2 reviewing the first pillar and introducing the second pillar, and so forth. LSs were highly interactive, including selected team reports, topical small group breakouts, activities, posters, and audience feedback system sessions. Key national faculty traveled to all 30 LSs across all five regions.

Action Periods (between LSs) were equally important. After each LS, all teams tested and implemented Plan-Do-Study-Act (PDSA) cycles of change identified during LSs, and performed “teach back” of PACT concepts to members of “home” teams who did not travel to the LSs. The PDSA Cycle consists of: 1) Plan the test or observation, including collecting data; 2) Do/try the test on a small scale; 3) Study the results and analyze the data; and 4) Act, or refine the change based on what was learned from the test. All teams were assigned two coaches (Clinical and IE) who assisted them with these PDSAs to achieve their seven aims. Teams used standard monthly reports to describe narrative and numerical progress (trend graphs standardized measurement tools). IEs also collected Collaborative tools to be included in the national “PACT Toolkit,”27 which served as the virtual vehicle to spread ideas to other teams.

Pact COLLABORATIVE EVALUATION METHODS

Measures of Team Performance: Coach Assessment Score and PDSA Analysis

Each team was expected to attend all six LSs and participate in coach-facilitated monthly phone calls and weekly team meetings. Teams were required to adopt the seven aims and associated measures, and submit monthly written reports. Assigned coaches and IEs provided feedback and assessed progress based on narrative descriptions and graphs from the standardized measurement tools. IEs and coaches assessed each of the 141 teams’ monthly reports to identify, collect, and categorize teams’ PDSAs and progress toward their aims. The clinical coaches assigned a score reflecting progress for each of the seven aims based on the IHI Collaborative Assessment Scale28 (Appendix F). The stated goal was to achieve the aims (represented by a score of “4”).

VHA Measures: PACT Compass & ACP Medical Home Builder Survey

PACT Compass metrics from VHA’s information systems were obtained and organized in broad domains, such as access, coordination of care, and continuity. The American College of Physicians (ACP®) Medical Home Builder Survey,29 was initially conducted in October 2009 (pre-Collaborative) and repeated in July 2011 (during the Collaborative). It provides data on patient care, organization, and workflow status, and, when administered sequentially, overall practice improvement (Appendix G).

Insights on PACT Collaborative: Data Collection Survey

While the monthly reports, standardized measurement tools, and PACT Compass provided an assessment on the effectiveness of PDSA activities to achieve the Collaborative aims, surveys were used for participants to share insights on lessons learned, participants’ satisfaction, learning, skill acquisition, and perceived barriers and enablers of PACT implementation (Appendix H). Participants were asked to complete a survey after each regional LS, and two online “follow-up surveys”30 (Appendix I) were administered approximately 90 days after LS4 and LS6.

ANALYSIS

Data analysis was conducted in SPSS 16.0. One-way ANOVA provided a common mean and identified statistical differences in the comparison of PACT Compass Measures before, during, and after the PACT Collaborative. A p-value less than 0.05 identifies less than a 5 % chance that the identified statistical differences were due to chance. Descriptive statistics and means scores were used to describe the coach assessment score and survey results (four point scale).

RESULTS

Measures of Team Performance: Coach Assessment Score/PDSA Analysis

Based on the 1,670 attempted PDSAs collected from 3,186 monthly reports, 93 % (n = 1547) of PDSAs were successfully implemented; 2 % (n = 29) were abandoned; and 6 % (n = 94) were work-in-progress. Eighty-one percent (n = 1253) of successful PDSAs (across all regions) focused on nine principles. Teams attempted to reduce demand and balance supply-and-demand for appointments to improve access and transitions, focus on high-risk patients, create an infrastructure to manage and coordinate care, anticipate patients’ needs, synchronize visit start times, and optimize environments/improve efficiency to improve practice function (Appendix J). Table 2 and Fig. 3 summarize the coach assessment score. Teams achieved the most success in offering same-day appointments, increasing non-face-to-face care, improving team communication, identifying and actively managing high-risk patients, and starting appointments within 10 min of scheduled times. Improving handovers with inpatient and specialty care were the most challenging aims.

Table 2 Summary of Coach Assessment Score at End of PACT Collaborative by Pillar and Aims
Full size table
Figure 3
figure 3

Trend of cumulative PDSAs and overall Coach Assessment Score across the six learning sessions.

Full size image

VHA Measures: PACT Compass & ACP Medical Home Builder Survey

Assessment of PACT Compass metric performance of teams participating directly in the Collaborative is beyond the scope of this paper. Instead, we focused on the Collaboratives as a national program with potential indirect as well as direct effects. Data sets for October of FY2010 to December of FY2012 were analyzed to capture change in PACT Compass performances during the pre-Collaborative, Collaborative, and post-Collaborative periods. Table 3 (below) indicates that there was a statistically significant improvement in ACP® Medical Home Builder modules during the Collaborative period compared to ‘before the Collaborative’ period. Also, there were statistically significant increases in average performance in each of the following eight measures: 1) percentage of patients seen in primary care within 7 days; 2) same-day appointment with primary care provider ratio; 3) primary care group encounters; 4) ratio of primary care telephone encounters to all primary care encounters; 5) total inbound primary care secure messages; 6) total outbound primary care secure messages; 7) second-day contact post-discharge; and 8) average primary care staff satisfaction.

Table 3 Comparison of PACT Compass Measures Before, During and After PACT Collaborative
Full size table

Insights on PACT Collaborative

One thousand, two hundred and thirty-three unique participants from 141 teams attended the six LSs. A total of 5,116 (59 %) participants provided feedback after each LS across the six LS in five regions (Appendix K). The online follow-up survey response rates after LS4 (n = 469; 33 %) and LS6 (n = 254; 21 %) were low. Based on a 4.0 scale, mean scores for all satisfaction items and scales were identical or slightly higher for the later learning sessions (see Table 4 below).

Table 4 Participants’ Assessment at the End of Each Learning Session
Full size table

We report here on the combined follow-up surveys (723 unique responses, or 57 % of Collaborative participants), including physicians (14 %), nurses (37 %), administrators (21 %), associated health professionals (6 %), and leadership participants (21 %) with VA experience ranging from 1 to 18 years (Appendix L). Participants reported success in gaining new skills and accomplishing learning objectives. 98 % of respondents reported that they developed new skills/knowledge as a result of participating in PACT Collaborative (Appendix M). Participants tended to be satisfied after each LS (Table 4) and regional Collaborative (Appendix N). Improvements included: 49 % increase in team-led huddles per week; 38 % increase in team meetings per month; 35 % contact with patient within 48 h/7 days after discharge from inpatient care (Appendix O).

PACT Collaborative Evaluation

Ninety-one percent of respondents reported that the PACT Collaborative was necessary to implement PACT (91 %); the format was a helpful (86 %); they gained expected knowledge (93 %), developed (92 %) and shared (93 %) successful strategies, and called it a good networking opportunity (91 %) (Appendix P). Additionally, 82 % agreed that they had a clear understanding of their own and team members’ roles; however, only 61 % reported having the necessary influence to make changes needed at their home sites (Appendix Q).

Usefulness of PACT Collaborative Resources

Ninety-one percent reported their knowledge and understanding of how to implement PACT increased (ranged from 50 to 100 %) based on their participation in the Regional PACT Collaborative (Appendix R). Eighty-two percent reported that the resources and tools provided in the PACT Collaborative were useful in implementing changes, and 79 % reported that the resources and tools were helpful in sharing information at their facilities. Respondents rated the following resources from “slightly useful” to “extremely useful”: coaching (95 %), regional Microsoft LiveMeeting/conference calls (95 %), Prework Guide (97 %), change package (98 %), Collaborative coordinators (97 %), Collaborative directors (96 %), IEs (84 %), standardized measurement tools (90 %), and monthly reports (92 %) (Appendix S).

Barriers and Enablers

The most common enablers included: opportunities to practice and use the skills learned (65 %), usefulness of skills/behaviors (72 %), senior leadership support (45 %), supervisor support (59 %), impact on career development goals (56 %), support from local PACT teams (71 %), productivity of local PACT teams (72 %), customer support (58 %), VHA’s commitment (60 %), priority level (58 %), and confidence to use the process (65 %). Barriers included availability of coaching assistance (50 %) and availability of measurement support (45 %) (Appendix T).

PACT Collaborative Success Factors

Respondents reported the following PACT Collaborative success factors: recruiting the right people on the PACT team (82 %), setting clear and mutually agreeable aim statements (88 %), establishing simple measures (87 %), flow mapping processes (83 %), reporting teams’ progress toward aims (83 %), conducting PDSA cycle tests of change (79 %), obtaining buy-in from others in the organization to successfully implement recommended changes (77 %), and spreading PACT to other teams at the facility (82 %) (Appendix U).

Participants’ Perspectives on Change

Respondents reported the information received about PACT changes had been timely (68 %), was useful (78 %), and allowed them to participate in the implementation of the proposed changes (71 %). Respondents reported that the PACT Collaborative adequately answered their questions about the changes (60 %), allowed them to ask questions about proposed changes (69 %), and was more effective than past efforts (59 %). Respondents reported that adequate information was provided about the forthcoming changes (55 %), and that they had some control over implementing the proposed/occurring changes (54 %). (Appendix V).

DISCUSSION

The traditional BTS Collaborative includes three LS and lasts less than 12 months; VHA’s PACT Collaborative consisted of six LSs and lasted 21 months. Teams adopted PSDAs at a steady rate, where the majority of PDSAs were accomplished in the second half of the Collaborative. The slow increase of PDSAs during the first half of the Collaborative may reflect the teams’ formation, learning improvement methods, and challenges to implement aims. The timing of PDSA completion to achieve Collaborative aims suggests that for a complex intervention like PACT, the longer timeline may be a better match for Quality Improvement project completion in the PACT environment.

The majority of participants considered the Collaborative to be necessary for successful PACT implementation. Collaborative coaches judged most PDSA cycles as successful for PACT pillars; participants reported behavior change stemming from Collaborative activities. While the specific effects of the Collaborative cannot be separated from the many other implementation strategies employed by VA, the overall significant national PACT improvement in the specific areas addressed by the Collaborative and in line with the Collaborative’s time line may have been influenced by the large-scale Collaborative process.

While participants valued the Collaborative overall, the data collected identifies several areas for further investigation and improvement. Improving handovers between PACT and its affiliated inpatient and specialty services proved especially challenging for improvement teams. Nearly half of participants did not feel empowered to make changes, and a majority did not feel they had input into decisions about their organization’s future. Pilot teams’ organizational influence may be important to the VHA’s ability to achieve improved performance. Future investigation of these findings may be important to PACT and its ability to achieve an ongoing improvement-oriented culture.

LIMITATIONS AND FUTURE DIRECTIONS

The analyses presented here are formative and not designed to address the effectiveness of Collaboratives. VHA’s PACT Collaborative was part of a larger organizational strategy that included resource infusion, system-wide performance goals, and complementary approaches to education. For this reason, it is difficult to ascribe the level of contribution made by the Collaborative effort to VHA’s progress in PACT implementation. Other limitations include: Coach Assessment scores are subjective, and PACT Compass scores reflect all PACTs, not the subgroup of Collaborative participants. The survey response rate was 59 % (723/1233) and dropped to 21 % for the last survey. Levels of satisfaction remained stable across survey waves; the non-responders, however, may have been more or less satisfied than the responders.

Since the PACT Collaborative, the skills, PACT toolkit, change and measure tools, and relationships created have been a valuable and reused resource, and many VA primary care practices have adopted the features of patient-centered care embodied by the PACT Collaborative. Achieving all the goals will involve continued strategic redeployment of resources, realignment of priorities, and continued cultural change toward the PACT mission to provide Veterans’ healthcare delivery of the highest quality, safety, and effectiveness.

As the VHA continues to disseminate the PACT Model through primary care and address improvements in specialty/surgical care and mental health, the Collaborative Learning Model may be an effective way to begin to create change across a large system.

CONCLUSION

The VHA PACT Collaborative—one of the largest and longest conducted to date—was implemented in a highly standardized and disciplined manner. Participants reported positive feedback; written reports and measures documented progressively increasing numbers of changes; and coaches assessed that teams achieved most of their aims. Moreover, the PACT Compass measures reported whole system improvements in PACT performance measures. This carefully designed and executed VHA PACT Collaborative was an effective component of the overall VHA strategy of implementing wide-scale transformation in primary care for Veterans.