Developing trauma-informed organizations is foundational to implementing trauma informed care in behavioral health care (Substance Abuse & Mental Health Services Administration, 2014), yet evidence-based practices are difficult to implement and sustain across health care systems (e.g., Hailemariam et al., 2019). Thus, there is a critical need to understand the means by which trauma-informed practices can be successfully implemented and sustained at the organizational level. A learning collaborative (LC) is a common implementation strategy by which clinical innovations are introduced across networks of healthcare provider organizations through shared educational experiences and problem-solving exercises (Brar et al., 2021; Hacker et al., 2014; Lloyd et al., 2015; MacDonald-Wilson et al., 2017). LCs are intended to facilitate specific, measurable, and sustained changes in clinical care in the context of a collaborative learning environment (Nadeem et al., 2013, 2016; Powell et al., 2015,).

The Institute of Healthcare Improvement’s Breakthrough Series Collaborative Model (Institute for Healthcare Improvement, 2021; Kilo, 1998) is a popular framework for designing LCs that incorporates learning principles across participants at multiple levels of a provider organization (e.g., patient, clinical, administrative, management). The Breakthrough Series model structures a sequence of learning sessions where new knowledge and skills are imparted, followed by action periods where the new information can then be applied and tested. Organizations are encouraged in the action period, referred to as a Plan-Do-Study-Act (PDSA) cycle (Kilo, 1998), to establish goals to change the delivery of care (e.g., train staff, identify measurement tools, create workflows), measure progress towards these goals, review progress data as a team, and then use these data to inform goals for the next learning cycle. Regularly scheduled LC meetings encourage accountability for tracking these outcomes and allow participants to benefit from the knowledge gained from other organizations with similar implementation goals. For instance, participants may learn how other organizations troubleshoot barriers to implementing new trainings or screening tools with similar client populations, or they may gain ideas around how to motivate or incentivize staff who are facing similar work-related stressors.

Sustainability Evidence

Several previous studies have investigated the sustainment of clinical changes following the completion of LCs. There is evidence to suggest that LCs can achieve sustainable shifts in community mental health care up to two years after active learning activities have ended (Cavaleri et al., 2007; Helseth et al., 2020; LoSavio et al., 2019). However, most of the research on LC sustainability has been descriptive (Cavaleri et al., 2007; Helseth et al., 2020; LoSavio et al., 2019; Nease et al., 2010; Noroña & Acker, 2016), and only a few studies have examined theory-driven predictors of LC sustainability. Currently, very little is understood about mechanisms by which LCs affect change in sustainability (McLeod et al., 2018). Studies that have applied theory to this question indicate that quality improvement components of the LC such as ongoing use of PDSA cycles, tracking outcomes data over time, and skill-building are positive predictors of sustainability (Ford et al., 2011; Hearld et al., 2016). However, additional research is needed to examine predictors of LC sustainability that are rooted in theory (Hearld et al., 2016).

A Theoretical Model for Studying Learning Collaboratives

The Longitudinal Education for Advancing Practice (LEAP) model (McLeod et al., 2018) proposes multiple mechanisms by which clinical training and subsequent consultation strategies impact the adoption and sustainability of new clinical innovations, drawing from literature on adult learning theory and supervision processes. The LEAP model is helpful for conceptualizing how LCs affect workforce behaviors through training and consultation strategies that promote clinicians’ knowledge and skills in the target innovation. LEAP also describes the importance of attitude- and relationship-related constructs, such as clinician self-efficacy, as factors that motivate long-term learning and utilization of new skill. LEAP posits that the initial learning of a new innovation requires these co-occurring cognitive-, attitudinal- and skill-based mechanisms. Upon acquiring declarative knowledge of a new skill, the learner experiences an increased cognitive load and requires more intensive feedback to recognize mistakes and assess the impact of the skill. During LC initiatives, these initial stages of knowledge and skill acquisition occur when the innovation is first introduced to trainees by the training faculty, along with detailed implementation expectations (i.e., a roadmap for how the trainees should begin to utilize the innovation in their practice). Initially, the training faculty more heavily support decision making and identification of next steps for implementation. Over time, these skills are expected to become more automatic with repetition and feedback as procedural knowledge and metacognition (i.e., trainees’ self-awareness of their own knowledge gaps or training needs) improve. This repetition can occur during each PDSA cycle, where trainees are asked to rehearse new skills they were trained on, monitor their application of skills to identify gaps in implementation, and share feedback with other LC participants during monthly meetings. Meanwhile, cognitive- and skill-related processes are supported by increased positive attitudes towards the innovation and increased self-efficacy regarding their delivering the innovation which comes with increased practice. As a result, learners become more successful at integrating skills, generalizing knowledge, and performing self-assessment (McLeod et al., 2018). Given the relevance of LEAP mechanisms to our understanding of how LC participation affects implementation, this model could be helpful to understand shifts in clinical practice through learner feedback engagement, repetitive practice, self-monitoring, and self-efficacy. To our knowledge, this model has not yet been applied to predict LC sustainability.

Current Study

The current study investigated the sustainability of a 15-month trauma informed care LC (TLC) implemented in 23 rural Pennsylvania counties across a behavioral health system. The TLC was organized and supported by the Behavioral Health Alliance of Rural Pennsylvania (BHARP) and Community Care Behavioral Health Organization of the UPMC Insurance Services Division (Community Care). BHARP (http://www.bharp.org) is comprised of county mental health and intellectual disabilities administrators, human services directors and drug and alcohol Single County Authorities who monitor human services including Pennsylvania’s HealthChoices Medicaid program. Community Care (http://www.ccbh.com) is a nonprofit behavioral health managed care organization that facilitates behavioral health services for Medicaid-enrolled adults and children in 43 of Pennsylvania’s 67 counties.

A full description of TLC design, structure, aims, and sources of funding can be found in Bills et al., (under review) and Minnich et al. (2022). As part of the TLC, provider organizations trained their staff in principles and evidence-based practices related to trauma-informed care. Trainings were based on the trauma-informed organizational practices by Hummer and Dollard (2010) and the dimensional model of trauma-informed care by Harris and Fallot (2001) centering five core values: safety, choice, trust, collaboration, and empowerment. The goal of the TLC was to establish a community of provider organizations that could become self-sufficient in its efforts to be trauma informed. This included establishing an organization in which all staff are knowledgeable about trauma symptoms and their impacts on individuals in service, traumatic stress symptoms and exposure to trauma are regularly assessed and used to inform care, trauma-specific treatments are available and implemented when needed, and improvements to policies and procedures are implemented in response to regular outcomes data collection. An additional component to this initiative was the provision of trauma-informed supervision in which supervisors reflect understanding of trauma exposure on individuals in service as well as their supervisees providing care. A four-day training for supervisors occurred at the start of the TLC and was followed up by a year of monthly consultation calls held via teleconferencing. Consultation meetings were divided into four groups of supervisors based on location, and calls were led by TLC training faculty. The initial training and consultation calls were attended by 31 supervisors, representative of 21 agencies who participated in the TLC (approximately 95% of agencies involved in the TLC); about half of the supervisors in these consultation calls continued this consultation format into the sustainment period.

The target outcomes of the TLC across all provider agencies were regular trauma symptom screening, conducting staff trainings for trauma-informed care with existing on new onboarded staff, and self-reported staff confidence in delivering trauma-informed care. Each of these outcomes improved significantly over the TLC period (Bills et al., under review). The current investigation into sustainability not only assesses change in these outcomes over the span of 3 years, but also the impact of TLC factors related to the LEAP model, such as experience rehearsing trauma-informed care principles by implementing TLC action steps and engagement in feedback through PDSA cycles.

TLC Progress Ratings and Milestones

TLC progress ratings were collected monthly from agencies during active implementation to describe the extent to which agencies felt they had successfully implemented, tested, and sustained their implementation goals. TLC milestones (which were the same for each agency) were tracked on a shared, composite checklist that was divided into 12 categories (e.g., laying the foundation, internal staff training, developing workflows, preparing for sustainability) based on stages of implementation. A total of 78 possible action steps or discussion questions were included in the milestone checklist, and this list was reviewed and addressed by staff at each agency monthly. Both agency-reported progress rating and TLC milestone completion reflected the extent to which trauma-informed care principles had been implemented over time.

Plan-Do-Study-Act Cycles

Each agency engaged in monthly PDSA cycles during active implementation to support TLC goals. PDSAs were individualized to the needs of each agency and provided opportunities to address topics from the TLC milestones such as developing workflows for trauma symptom screening and documentation in the health record, identifying appropriate trauma screeners, and shifting policies and procedures within organizations to build capacity for TIC (e.g., requiring staff to continue education in TIC, establishing agency-wide processes for responding to positive screeners and completing re-assessment). Community Care research staff reviewed data from tracking workbooks for each agency and assigned an overall PDSA quality grade (A +  = outstanding, C− = does not meet criteria) at the end of the TLC. PDSA quality was indicated by a grade that reflected whether the agency had specified actionable objectives, set clear metrics to determine success, completed their PDSA documentation, provided interpretations of their findings, and identified relevant next steps for their organization.

Trauma-Informed Care Center Designation

Since the initial TLC was completed in December 2017, participating agencies have received trainings in evidence-based practices for trauma and become designated by BHARP as a Trauma-Informed Care Center (TICC). Evidence-based practice trainings offered during the sustainment period included Cognitive Processing Therapy (Resick et al., 2016), Trauma-Focused Cognitive Behavioral Therapy (Cohen et al., 2016), Dialectical Behavior Therapy (Linehan, 2018), and Seeking Safety (Najavits, 2002). Clinicians also received additional learning opportunities on topics related to establishing trauma-informed systems of care, such as vicarious trauma and trauma-informed supervision. To become designated as a TICC, agencies were required to provide annual trauma-informed care training to all new and existing staff, implement evidence-based treatments for trauma, implement routine trauma symptom screening, report the specific measurement tools used and the results of these trauma symptom screeners over time in response to trauma-specific treatments, provide trauma-informed supervision for all clinical staff and trauma-informed debriefing at least monthly (or provide plans for improvement if there are deficits in supervision participation), assess consumer feedback on treatment (including symptom, recovery, satisfaction and other surveys), and engage in monthly quality improvement team meetings. The TICC criteria were built upon the TLC aims, and achievement as a TICC is an indicator of sustained trauma-informed care practices and policies within an organization. The TICC designation process required agencies to complete an electronic application form, supply supporting documentation (e.g., participation rates in supervision, attendance, and consultation records for trainings in evidence based interventions) to demonstrate that items on the application were completed and submit an organizational self-assessment. TLC faculty reviewed application materials using an electronic scoring file and rubric which were used by reviewers to designate each agency as “not met” (i.e., temporary practices in place, lack of training), “acceptable” (i.e., trauma-informed care practices fully integrated and spread throughout the organization), or “exemplary” (i.e., integrated trauma-informed care practices with a sustainability plan, consideration of trauma is routinely reflected in treatment planning and outcomes monitoring).

Sustainment During a Global Pandemic

In response to the spread of Coronavirus Disease (COVID), several TICC agencies reported difficulties sustaining trauma-informed care related practices. By summer of 2020, agencies reported they were struggling with coverage due to illness and increased rate of resignations, and many supervisors and clinicians reported the need to cover multiple offices at their agency. In response to COVID, agencies shared ideas on how to continue delivery of trauma treatments via telehealth and how to use telehealth creatively in general (e.g., one provider reported making treatment groups available via telehealth across the TLC network, giving individuals in care access to telehealth services hosted by facilities outside of the county where they reside). Agencies reported initially struggling with the completion of screeners via telehealth. By early 2021, all agencies had arrived at telehealth solutions. In response to resignations, agencies discussed ways to maintain trauma-informed practice changes despite starting over with new staff. Ideas included sharing training materials among agencies, utilizing trainers from other agencies when needed, TICC designated agencies mentoring a new or completely overturned facility, and agencies providing case consultation opportunities for their clinicians for group support and to reinforce and maintain momentum. TLC faculty also continued to provide annual virtual trauma training institutes and access to virtual trainings in evidence-based interventions during this time.

Research Questions

While initial implementation results from the TLC were positive (Bills et al., under review; Minnich et al., 2022), more can be done to understand how the TLC has impacted the sustainment of trauma-informed practices across agencies, and there is limited available knowledge regarding predictors of sustainable LC effects. Furthermore, variables aligned with explanatory mechanisms of training have not been assessed in the TLC. The following questions were established to evaluate the impact of the TLC on long-term changes in trauma-informed care indicators and assess the extent to which LEAP variables (i.e., skill and attitude changes) were associated with sustained trauma-informed practices. Time points are defined as the first month after the TLC concluded, which marked the end of the active implementation period (TA) and three years after the TLC concluded, which marked the end of the sustainment period (TS).

Question 1

To what extent did provider agencies sustain improvements in trauma-informed care quality indicators (i.e., trauma symptom screening, participation in staff trainings on principles of trauma-informed care, staff confidence in delivering trauma-informed care) from TA to TS? We hypothesized that trauma symptom screening rates, percentage of staff trained in trauma-informed care, and staff confidence with using trauma-informed care would be sustained (i.e., stay the same or continue to increase) from TA to TS.

Question 2

To what extent are attitude- and skill-based training variables (i.e., staff confidence delivering trauma-informed care, TLC milestone completion, TLC progress ratings, PDSA quality) associated with TICC status among agencies one and two years after the conclusion of the TLC? We hypothesized that agencies’ TICC status in 2018 (one year post-TLC) and 2019 (two years post-TLC) would be associated with increased TLC milestone completion, higher progress ratings during LC action periods, greater proportions of staff with high levels of confidence using trauma-informed care, and higher objective PDSA quality ratings during the TLC. These hypotheses are supported within the LEAP Model, which posits that clinical trainings are more likely to succeed when increasing skills and fostering positive attitudes towards clinical innovations (McLeod et al., 2018). These hypotheses are also supported by previous research in which improved attitudes towards evidence-based practices were associated with LC participation (Haine-Schlagel et al., 2013), and engagement in LC trainings were associated with improved clinical behaviors (Nadeem et al., 2016).

Method

Participants

The current study included mental health and substance use disorder provider agencies that participated in the TLC (N = 22) across 23 rural counties of the North Central region of Pennsylvania. These agencies represented 29 unique clinic sites that deliver behavioral health services to adults and children. Of the agencies, 12 (13 sites) delivered mental health services, and ten (16 sites) delivered substance use services. Full details regarding recruitment procedures can be found in Bills and colleagues (under review).

Procedures

This research was approved by the UPMC Quality Review Committee as well as the West Virginia University Institutional Review Board. Data included archival records (i.e., TLC tracking workbooks) as well as cross-sectional surveys that were collected for an internal quality review of TLC sustainability.

TLC Data Collection

During the active implementation period, data were collected from each provider, summarized, and disseminated to Community Care and BHARP staff serving as a TLC faculty. Faculty summarized data and presented quarterly updates to the TLC. Each provider received an individual report quarterly on progress towards TLC goals.

Post-TLC Data Collection

Trauma symptom screening rates, staff trauma-informed care training participation, and aggregated staff confidence in delivering trauma-informed care at TS were collected via follow-up survey. This survey was administered via online platform by TLC faculty; 30-min phone interviews to review the questions were scheduled with each provider. Participants who did not complete the online survey were able to provide information during the 30-min phone interviews. All providers that had participated in the TLC were approached during a virtually hosted quarterly review meeting with TLC faculty. Leadership (e.g., clinical directors) of each provider was contacted individually by email, provided a blank copy of the survey questions, and asked to schedule a time with Community Care staff for a phone interview. Providers who did not complete the online survey or respond to the phone interview invitation were sent up to two reminder emails 1 week apart.

Sources of Data

TLC Workbook

Each organization maintained a TLC workbook submitted to the TLC faculty monthly for review. This self-reported workbook recorded the organization’s trauma screening rates, number of staff trained in trauma-informed care, aggregated staff confidence ratings, monthly implementation progress ratings, and the total number of staff monitored for outcomes during that month.

Trauma Screening

Trauma screening rates were calculated as the percentage of individuals in service at each provider organization who received a trauma screener during a visit encounter.

Staff Training

During active implementation, organizations did not directly report of percentage of current staff trained in the TLC workbook. To estimate staff training participation rates, the number of staff trained was divided by the number of staff that were reported by agency leadership as being monitored in the TLC workbook for outcomes that same month.

Staff Confidence

Staff confidence using trauma-informed care was rated by staff at each provider organization on a 10-point, Likert-type scale (0 = not confident, 10 = extremely confident).

TLC Progress

Progress ratings were reported by provider organizations on a Likert-type scale that increased by increments of 0.5 (1 = team established/no work accomplished, 5 = outstanding sustainable results).

TLC Milestones

Milestone completion was calculated as a proportion of action steps completed based on the highest action step indicated by provider organizations on the milestone checklist. The average percentage of milestones completed for each category are summarized in Table 2.

PDSA Quality

For the purposes of these analyses, PDSA quality letter grades were converted to numerical ratings, such that a grade of A + became a score of nine out of nine, and a grade of C- became a score of one out of nine. Thus, higher PDSA quality ratings indicated greater proficiency in a provider’s engagement and utilization of PDSA cycles during the TLC (e.g., goal setting, data collection, goal review, and goal adjustment). A full summary of PDSA grade frequencies is provided in Table 3.

Trauma Informed Care Center (TICC) Organizational Application

At the end of the TLC, agencies were given the opportunity to sustain practices through a designation of “Trauma Informed Care Center” (TICC). The TICC designation process required agencies to complete an electronic application form, supply supporting documentation to corroborate application items, and submit a TICC Organizational Self-Assessment Survey that was based on the Creating Trauma-Informed Care Environments curriculum by Hummer and Dollard (2010). On the TICC designation application form, agencies were asked to report the extent to which they currently screen individuals in service for trauma symptoms, use outcomes monitoring in treatment, train their staff in trauma-informed care, apply trauma-informed supervision and debriefing practices, and measure satisfaction of individuals in care. The TICC Organizational Self-Assessment Survey asked three staff members within each agency (one clinical, one executive, and one non-clinical) to provide rankings on 39 items across three domains of competence using a Likert-type scale (0 = no plan in place, 4 = plan has been implemented and revised based on feedback or data collected regarding initial implementation). The three domains assessed were: Competent Trauma-Informed Organizational Practices (e.g., does the agency provide adequate resources for TIC and data monitoring, does the agency have formal policies and procedures in place that reflect the language and practice of TIC?), Competent Trauma-Informed Clinical and Milieu Practices (e.g., does the agency provide opportunities to staff to recognize and address vicarious traumatization, is the physical environment attuned to safety/calming/de-escalation, does the agency provide trauma-specific, evidence-based, and evidence supported treatments?), and Consumer and Family Engagement in Trauma-Informed Care (e.g., are consumers and families at this agency actively involved in treatment and discharge planning and decisions regarding the transition of care, does the agency survey consumers on satisfaction and use data about their experience in care to make changes to care delivery?). For each item answered, staff were asked to provide the data source used to evidence their response, such as staff interviews, health record reviews, consumer interviews, observation, etc. The TICC Organizational Self-Assessment Survey was collected by agency administrators during the TICC application process. Agencies were given a TICC manual and qualitative scoring rubric upon requesting their application. Faculty from the TLC reviewed the application materials using an electronic scoring file and rubric. Both the total assessment score and rubric were used by faculty to designate each agency as “not met” (i.e., temporary practices, lack of training), “acceptable” (i.e., TIC components fully integrated into care, TIC spread throughout the organization), or “exemplary” (i.e., integrated TIC practices and sustainability plan, TIC principles utilized in treatment planning and outcomes monitoring).

Provider Follow-Up Survey

The investigators developed a follow-up survey for provider organizations at TS (April 2021) to assess the extent to which TLC provider organizations were currently engaging in routine trauma-informed practices (i.e., staff trainings in trauma-informed care, trauma symptom screening rates, and staff confidence in using trauma-informed care). The design of this survey was stakeholder-informed, and questions were drafted and edited by TLC faculty before it was administrated to provider agencies. This follow-up survey combined quantitative estimates of trauma informed care organizational factors (trainings, screenings, staff confidence ratings) with open-ended items to determine agencies’ perspectives of how sustainment had unfolded from 2017 to 2020. These open-ended items were not used to address current research questions and were included by stakeholders to inform internal quality improvement. In addition, this survey collected demographic characteristics that were not accessible through existing TLC tracking materials, such as proportion of staff roles that were clinical vs non-clinical, full-time equivalents for current employees, education level of staff, and workforce turnover concerns in the past year. Survey questions were answered on behalf of all staff currently working at provider organizations at TS, and included new staff hired during the sustainment period.

Data Analysis

Changes across time in trauma-informed care indicators (i.e., trauma symptom screening, training rates, confidence delivering trauma-informed care) were assessed across TS using paired-samples t-tests. Associations between TICC status and LEAP model constructs were assessed using a linear mixed modeling approach. Qualitative data were grouped by common themes as facilitators or challenges for screening, staff training, and delivery of trauma-informed care. Statistical analyses were performed using IBM SPSS Version 26.

Results

Participant Characteristics

A total of 22 provider agencies participated in the TLC (12 mental health agencies and 10 substance use agencies). Across agencies, the average breakdown of staff education was 8.4% high school or GED-level (SD = 9.8%), 39.3% Bachelors-level (SD = 25.7%), 45.5% Masters-level (SD = 22.6%), and 6.9% Doctoral-level (SD = 9.1%). Approximately 73.5% of staff at these agencies served in a clinical role (SD = 25.1%), and 26.5% of staff served in a non-clinical role (SD = 25.1%). Clinical roles included clinicians and clinical supervisors; non-clinical roles included positions such as administrative assistants, upper level administrators, and information technology staff.

Descriptive Statistics

Eighteen of the 22 participating agencies (81.8%) completed their final TLC workbook at TA. Follow-up surveys (either online, phone-based, or both) were collected for 19 out of 21 agencies (90.5%) who were still involved in the trauma-informed care initiative as of April 2021 (TS). Of the two agencies who were not interviewed at TS, one never responded to survey requests, and one cited scheduling barriers but expressed interest in being surveyed later. A total of 16 agencies (72.7%) completed the online version of the follow-up survey, and a total of 15 agencies (68.2%) completed a phone survey. Each agency who completed a follow-up survey answered all questions pertaining to the core analyses of the current study. A full summary of trauma-informed care indicators at TA and TS is provided in Table 1. Breakdowns of score frequencies observed for TLC milestone completion and PDSA quality are reported in Tables 2 and 3, respectively.

Table 1 Descriptive statistics for study outcome variables
Full size table
Table 2 Average TLC milestone completion as a percentage of possible action steps addressed
Full size table
Table 3 PDSA grade and quality rating frequencies
Full size table

TICC Status

At 1 year post-TLC (2018), 21 agencies were still taking part in the trauma-informed care initiative; one agency withdrew due to reduced capacity to provide services. Five of these agencies (23.8%) met “exemplary” TICC criteria, nine (42.9%) met “acceptable” TICC criteria, and seven (33.3%) did not meet criteria for TICC status in 2018. At 2 years post-TLC (2019), one site had merged with another participating agency for a total of 20 remaining agencies. All participating agencies in 2019 qualified for TICC status, with 14 (70.0%) meeting “exemplary” and six (30.0%) meeting “acceptable” criteria.

Three-Year Sustainability

We assessed the sustainment of trauma-informed care indicators at Ts (April 2021) via paired samples t-tests. Dependent variables were normally distributed, unless noted otherwise.

Trauma Symptom Screening

There was no statistically significant change in the percent of individuals who received a trauma symptom screening from TA (M = 92.0%, SD = 10.9) to TS (M = 86.1%, SD = 25.1), t(19) = 0.92, p = 0.368; d = 0.20, indicating that initial gains made during the implementation process were sustained.

Staff Training

There was a statistically significant increase in the estimated percentage of staff trained from TA (M = 62.0%, SD = 34.4) to TS (M = 87.7%, SD = 21.9), t(19) = −2.21, p = 0.041; d = −0.52. These data indicated that the proportion of staff trained as estimated by management increased over time.

Staff Confidence

There was a significant decrease in the percent of individuals who endorsed high levels of confidence in delivering trauma-informed care from TA (M = 76.4%, SD = 17.5) to TS (M = 34.5%, SD = 26.6), t(21) = 7.45, p < 0.001; d = 1.59. The high self-reported ratings of confidence in delivery of trauma informed care observed during the LC were not sustained and reverted toward baseline levels.

Follow-Up Interviews

Of the 19 agency staff members interviewed, 78.9% (n = 15) reported success integrating a standardized process or agency-wide procedure to include trauma-related screeners as part of their intake process. Approximately 52.6% (n = 10) of interviewed agencies had embedded screeners into the electronic health record. Factors identified by agencies as hindering their implementation of trauma symptom screening included clinician burden of too many assessment tools or documentation (42.1%; n = 8) and insufficient time to complete screeners (26.3%, n = 5). Agencies who reported valuing competence in trauma treatment sought opportunities to enhance their trauma training offerings whenever possible (42.1%; n = 8). To better establish trauma-informed practices, some agencies emphasized training staff at all levels, rather than focusing exclusively on clinicians (31.6%; n = 6). Agencies reported that the staff training process was improved by having a standard training process or policies for training staff in trauma-informed care incorporated into pre-existing staff training/on-boarding procedures (31.6%; n = 6), while travel and travel costs were challenges to staff training (36.8%; n = 7). Several agencies indicated that their organization self-identified as trauma-informed and that it was part of their everyday language (36.8%, n = 7). Many agencies reported that learning collaborative participation had resulted in improvements to supervision meetings, which now included modeling trauma-informed care principles for clinicians (36.84%; n = 7). Over half of agencies (57.9%; n = 11) indicated that staff turnover is a concern, and five agencies (26.3%) indicated that turnover is specifically affecting their ability to provide trauma-informed care. Most of the agencies surveyed discussed the ways that the COVID pandemic had adversely affected access to trainings for specific treatment modalities (e.g., Seeking Safety, Eye Movement Desensitization and Reprocessing) as well as interrupted the agency’s own ability to train their staff in trauma-informed care (63.2%; n = 12).

Predictors of TICC Status

We performed linear mixed modeling to assess whether changes in skill- and attitude-related variables over time varied as a function of TICC status in 2018 and in 2019, see Table 4.

Table 4 Fixed effects of time and ticc status on skills and attitudes
Full size table

Staff Confidence

We ran a baseline, unadjusted model to assess changes in the percentage of staff who had high confidence in their ability to use trauma-informed care across time. This model indicated that high confidence rates increased significantly during the TLC, F(14, 116.03) = 10.68, p < 0.001. An adjusted model with 2018 TICC status and time as predictors of high staff confidence did not significantly improve fit compared to the baseline model, X2(2) = 1.03, p = 0.598. When accounting for changes over time, 2018 TICC status was not associated with high staff confidence, F(2, 37.00) = 0.81, p = 0.454. Similarly, the adjusted model including 2019 TICC status and time as predictors did not improve fit compared to the baseline model, X2(2) = 0.07, p = 0.965. When accounting for changes over time, 2019 TICC status was not associated with high staff confidence, F(2, 37.99) = 0.04, p = 0.962.

TLC Milestones

The average percentage of possible milestones that were addressed during the entire TLC by agencies’ quality improvement teams was 77.3% (SD = 13.0%). A baseline, unadjusted model to assess changes in the number of TLC milestones addressed by the quality improvement teams across time indicated that the number of milestones addressed increased significantly during the TLC, F(14, 120.00) = 41.39, p < 0.001. The adjusted model with 2018 TICC status and time as predictors of TLC milestone completion improved fit compared to the baseline model, X2(2) = 7.72, p = 0.021. When accounting for changes over time, 2018 TICC status was significantly associated with the number of milestones addressed, F(2, 43.34) = 4.42, p = 0.018. An examination of parameter estimates revealed that the number of milestones completed was significantly greater for agencies who met “exemplary” status in 2018 compared to agencies who met “acceptable” status (Estimate = −1.37, p = 0.005), and agencies who did not meet TICC standards, (Estimate = −0.98, p = 0.045). The adjusted model with 2019 TICC status and time as predictors did not significantly improve fit compared to the baseline model, X2(2) = 1.24, p = 0.538. When accounting for changes over time, 2019 TICC status was not significantly associated with the number of milestones addressed, F(2, 42.86) = 0.81, p = 0.451.

Progress Ratings

The average agency-reported progress rating was 1.6 out of 5 (SD = 0.3) at the beginning of the TLC and 4.7 out of 5 (SD = 0.4) at TA. A baseline, unadjusted model was run to assess changes in agency-reported progress rating across time. This model indicated that progress ratings increased significantly during the TLC, F(14, 110.66) = 52.99, p < 0.001. The adjusted model with 2018 TICC status and time as predictors of progress rating did not significantly improve fit compared to the baseline model, X2(2) = 1.46, p = 0.481. When accounting for changes over time, 2018 TICC status was not associated with progress ratings, F(2, 56.35) = 0.95, p = 0.394. The adjusted model with 2019 TICC status and time as predictors did not significantly improve fit compared to the baseline model, X2(2) = 4.75, p = 0.093. When accounting for changes over time, 2019 TICC status was not associated with progress ratings, F(2, 55.74) = 2.82, p = 0.068.

PDSA Quality Ratings

Only one PDSA quality rating was assigned to each agency during the TLC; thus, changes in PDSA quality over time were not relevant and the final model was compared to an intercept-only baseline model. The average numerical PDSA quality rating assigned was 5.2 out of 9.0 (SD = 2.2). The adjusted model including 2018 TICC status and PDSA quality did not improve fit compared to the baseline model, X2(2) = 1.43, p = 0.488. This model indicated that 2018 TICC status was not associated with PDSA quality ratings, F(2, 330) = 0.72, p = 0.488. The adjusted model including 2019 TICC status and PDSA quality improved fit compared to the baseline model, X2(2) = 22.57, p < 0.001. This model indicated that 2019 TICC status was associated with PDSA quality ratings, F(2, 330) = 11.68, p < 0.001. An examination of parameter estimates revealed that PDSA quality ratings were higher for agencies who met “exemplary” status in 2019 compared to agencies who met “acceptable” status (Estimate = −1.24, p < 0.001).

Discussion

The current study found that trauma symptom screening rates were sustained, staff trauma-informed care training participation rates improved, and staff-reported confidence regarding their capacity to deliver trauma-informed care decreased during the three-year sustainment study period. Additionally, agency-reported completion of implementation milestones during the TLC was associated with sustained trauma-informed care 1 year after the TLC, while PDSA quality rating was associated with trauma-informed care 2 years after the TLC.

Analyses of outcome sustainability at Ts revealed that the percent of individuals in service who received a trauma screener and the number of staff who received training in principles of trauma-informed care were sustained from TA to TS. This supports the ability of TLC to maintain improvements in behaviors that are important for trauma-informed practice over long periods of time. This finding also provides an important addition to the extant literature, which has previously only demonstrated one- or two-year maintenance of trauma-informed care behaviors following a LC implementation (Cavaleri et al., 2007; Helseth et al., 2020; LoSavio et al., 2019). Indeed, these data suggest that LC approaches could have long-term effects that justify the intensive resources that are needed to design and implement LCs.

TICC status in 2018 (but not 2019) was associated with the number of milestones addressed, and TICC status in 2019 (but not 2018) was associated with the PDSA quality rating assigned by the TLC faculty. No other variables mapping onto the LEAP model were associated with TICC status after the conclusion of the TLC. These findings may speak to skill-related mechanisms that are important for initial implementation and sustainability; namely, the importance of frequent rehearsal and reflection by agencies on the specific areas of improvement needed to address gaps in their capacity to deliver trauma-informed care. Factors and strategies that are needed to initiate implementation (catalysts to change) may be different from the factors and strategies that are important for maintaining gains. Milestone completion, which represented the agencies’ ability to follow the guidance of TLC faculty and adhere to a set of pre-determined action steps toward implementation was associated with more immediate (2018) behavior changes; whereas long-term (2019) TICC status was associated with PDSA quality ratings, which represent the agencies’ ability to engage in iterative feedback cycles (i.e., establish an actionable goal, collect data, review results, and adjust new goal). These variables represent unique but intersecting skillsets. Although both measures reflect maturation in the agencies’ ability to deliver trauma-informed services, PDSA quality reflects a quality improvement competency that transcends specific TLC goals. Capacity to gain knowledge in trauma-informed care and follow guided implementation plans is needed to establish practice changes at the organizational level, while additional competencies in data collection, self-reflection, and goal setting are needed to sustain these change over a longer term. These findings appropriately map onto the LEAP model, which posits that initial learning of a clinical skillset requires procedural knowledge development that is heavily supported by cognitive resources. Milestone completion was supported in the form of a checklist that provided clear instructions to agencies on what they must accomplish in order to progress their implementation goals. In contrast, LEAP posits that sustained implementation (i.e., long-term learning) requires a gradual transfer of knowledge to the organizational environment over time, increased self-awareness regarding skill development, and increased capacity to generalize the skill to new situations. PDSA cycles require agencies to apply knowledge to their specific contexts more independently, and agencies must become competent in assessing their own performance, identifying gaps in their performance or knowledge, and setting actionable goals in order to achieve a high PDSA quality score.

A surprising finding was that agencies’ self-reported confidence in the delivery of trauma informed care decreased from TA to TS while the estimated percentage of current staff trained increased over time. One previous study by Kopelovich and colleagues (2019) applied a LC approach to implement cognitive behavioral therapy for psychosis in another large service system. The authors noted a significant drop in clinicians’ self-perceived skill level from the training period to the initial consultation interval after training. As these authors suggest, it is possible that staff perceptions of self-efficacy naturally decrease as a function of having initial implementation supports removed, and that self-efficacy needs to be slowly reestablished over time as more clinical experiences are gained (Kopelovich et al., 2019). It is also possible that confidence in delivering services was supported by structured, monthly engagement in the TLC and the ability to network or share resources with other providers, which then changed after TA to less frequent meetings held across the entire TLC network. During the follow-up survey, multiple staff reported there were resignations and changes in staff roles noted in response to COVID in the year prior to these analyses. This likely impacted our observation of reduced confidence delivering trauma services, as there was an increased percentage of newer hires reflected in the survey administered at TS. It should be noted that confidence in our study was assessed generally (i.e., not specific to one behavior or aspect of trauma-informed care); thus, a reduction in confidence could be associated with self-efficacy as it pertains to several possible clinical behaviors (e.g., supervision seeking, ongoing training attendance). More information is needed to determine where confidence ratings were impacted the most. While lower confidence during the sustainment period does not necessarily indicate a decreased capacity to deliver trauma-informed care, the LEAP model suggests that attitudes of self-efficacy are expected to promote long-term learning by increasing learners’ willingness to continue their engagement in the implementation of evidence-based practices. Thus, additional research in which confidence growth is not disrupted by turnover or other outside factors would be important for assessing the extent to which self-efficacy predicts long-term engagement with trainings and sustainment of trauma-informed care behaviors.

The spread of COVID in spring of 2020 also influenced the ways that agencies delivered behavioral and substance use services for over a year of the three-year sustainment period. Behavioral health provider organizations and individuals in service have reported increased rates of stress and burnout, which can influence organizational culture, staff turnover, and quality of care (Kelly & Hearld, 2020). The current finding that staff training rates increased during the sustainability period is likely a reflection of increased turnover and difficulties retaining workforce considering provider feedback that these challenges increased during COVID. Given that these findings were likely influenced by COVID and the broader strain it has put on health service systems, future research will be critical to demonstrate whether factors associated with sustainability of trauma informed organizational practices in our study are consistent in different sociomedical contexts.

Limitations

The TLC workbook included several variables that were self-reported and therefore subject to reporter bias. Across workbooks, there was variability in the extent to which agencies described their experiences meeting TLC milestones, setting improvement goals, and collecting data to address these goals. For instance, some agencies recorded extensive detail around these improvement meetings, while others only briefly indicated if an item on the checklist had been discussed or not. Discrepancies in the amount of detail provided in the TLC workbooks may have impacted assignments of PDSA grades such that two agencies holding similar conversations would receive different ratings based on the information that was available to raters. Previous literature has documented that increased administrative burden, such as requesting clinicians to complete multiple measurements at multiple time points, might increase burden on these providers to provide patient care (Lemak et al., 2003) Additionally, training data was reported as an absolute number as opposed to a percentage of total staff trained due to this information being unavailable. Thus, these numbers account for changes in the number of staff but do not provide information about the amount of training saturation. It should also be noted that the data for the TLC workbooks and the follow-up survey were collected through the behavioral health managed care organization (Community Care), which serves as payer for these agencies. Despite efforts on the part of Community Care to foster collaborative and open communication with provider agencies, this existing relationship and an implicit expectation that trauma services are maintained in TICC-designated sites may have influenced responses by placing demand characteristics on agencies to overestimate success using trauma-informed care.

There were also some limitations to the methodology applied for data collection. Specifically, agencies had limited availability to schedule interviews, and these were often capped at 30-min intervals for the convenience of agency staff. The current study is post-hoc and does not directly test skill- and attitude-related variables. Further, the agencies that were selected for participation in our study were not randomized (i.e., there is a likelihood that sites were selected based on administrators’ beliefs that a provider would be better suited to implement trauma-informed care), and due to extensive turnover during the COVID pandemic, the staff members assessed at TS were different from those reflected in the TA assessment. To better understand the role that skills and attitude development play in the implementation process, future research should focus on designing true experiments that manipulate the cultivation of specific skill sets.

Strengths

The current study also contained several important strengths. This is the first study to examine training- and skill-related variables in association with sustainability of outcomes in a LC on trauma-informed care. The Breakthrough Series model, which informed the design of the TLC, has been used extensively for health system improvement initiatives (Brar et al., 2021; Haine-Schlagel et al., 2013; MacDonald-Wilson et al., 2017), and it is therefore likely that these results are applicable to providers applying a similar framework. A large percentage of agencies who initially took part in the TLC responded to the three-year follow-up survey, so attrition was not an issue for these analyses, and results that were collected at TS are likely representative of most of the agencies who participated at TA.

Conclusion

LCs may be a viable means of sustaining trauma-informed care practices, such as trauma symptom screening and staff training in trauma principles. Despite difficulties with staff retention in response to a global pandemic, the LC was able to maintain gains in staff training and trauma screening initiatives while noting decreased staff-perceived confidence in applying the principles of trauma-informed care. To maximize sustainability, LC faculty should encourage providers to address several implementation stages (i.e., milestones) while active supports from the TLC are in place and foster and attend to provider engagement in quality improvement (i.e., PDSA) processes. Our findings suggest that trauma-informed care sustainment can be maximized by attending to both trauma-specific and broader quality improvement competencies. Future research should continue to experimentally examine skill development and staff attitudes as they relate to sustained implementation. Modeling LEAP constructs with a larger sample of organizations could provide important insight into how skills and attitudes differentially influence short- and long-term organizational change. Additionally, future sustainability research in this area should directly measure the impact of LCs that are designed to impact a broader array of organizational practices that form Harris and Fallot’s (2001) theoretical basis for trauma-informed care, such as staff perceptions of safety, staff empowerment in agency decision making, and collaboration among staff.