Introduction

Interventions to support youth with social, emotional, and behavioral challenges in schools have been evaluated in randomized clinical trials with favorable results (e.g., McLeod et al., 2017; Pfiffner et al., 2013). Further, teachers’ implementation of some of these practices can be enhanced via intensive coaching or consultation from the research team (e.g., Bradshaw et al., 2018; Stormont et al., 2015). However, there is little research evaluating teachers’ sustained use of interventions under typical practice conditions after intensive supports are removed.

Sustainability has been identified as “one of the most significant translational research problems of our time” (Proctor et al., 2015, p. 2). Without sustainment, effort and resources spent on intervention development and implementation supports may be in vain. Similarly, others argue that the traditional linear pipeline approach to research where interventions are developed, evaluated, and subsequently deemed as “ready” for dissemination and adoption is too simplistic, and additional research is needed to examine the complex social factors that may facilitate intervention adoption and spread within an organization (Horton et al., 2018).

The Making Socially Accepting and Inclusive Classrooms (MOSAIC) program is a teacher-implemented classroom intervention designed to support children’s academic, social, and behavioral functioning and enhance inclusivity in peer relationships. MOSAIC strategies are infused into daily practices and include universal components for all children and targeted components designed specifically for children at risk of attention-deficit/hyperactivity disorder (ADHD) and peer problems. The first randomized trial revealed that children in MOSAIC classrooms (target and non-target children) showed better teacher-rated social and academic competencies and lower impairment in spring (after statistical control of fall functioning) relative to children in a typical practice control group. Target children in MOSAIC classrooms reported greater support from their teachers than did target children in control classrooms (although the intervention did not improve peer perceptions of target children). Further, greater teacher use of core MOSAIC strategies was positively associated with student outcomes (Mikami et al., 2020, 2022).

School staff in the randomized MOSAIC trial were trained and intensely supported to implement intervention strategies. The transition from the end of the trial to typical practice provides an opportunity to examine important questions: (1) to what extent do MOSAIC-trained teachers sustain strategy use after intensive supports have been withdrawn, (2) under conditions that closely match typical practice (i.e., initial training, peer support), do teachers who had not received MOSAIC training previously adopt the strategies over time (spread), and (3) can existing professional supports (e.g., peer-led professional learning communities; PLCs) support MOSAIC strategy use. Such questions may reveal unique information about what supports are needed to sustain and spread the use of evidence-based practices (EBPs). Given the number of EBPs available and the persistent problem with implementation (Klein & Knight, 2005), research on implementation under typical practice conditions is warranted.

Intervention Sustainability: Definitions and Recommendations

In the first comprehensive, rigorous review of the sustainability literature, Stirman and colleagues (2012) concluded that “an intervention could be considered sustained at a given point in time if, after the initial startup implementation support had been withdrawn, core elements were maintained (e.g., remained recognizable or delivered at a sufficient level of fidelity or intensity to yield the desired health outcomes) and adequate capacity for the continuity of these elements was maintained” (p. 10). Using this definition, they found that partial sustainability (implementation of some but not all components) was more common than full sustainability, even when full implementation was initially achieved. Across studies, the assessment of sustainability ranged from less than 12 months post-implementation (6% of studies) to two or more years past initial implementation (64%). Few studies assessed sustainability using objective measures; yet among those that did, less than half of the implementers maintained high levels of fidelity. Thus, continued supports are likely critical for sustained high quality implementation.

The school setting offers the opportunity to reach large numbers of youth in a non-stigmatizing and accessible way. Of hundreds of studies in the Stirman et al. (2012) review, most were conducted in health care or community settings. The findings of the few school-based studies are consistent with Stirman et al. (2012) in that partial sustainability is more common than full sustainability and the methodological rigor of studies is limited. For example, Cooper et al. (2015) used self-report data to assess sustainability of evidence-based school and community programs two years following grant-funded supports. Although 69% of program leaders self-reported sustained use, most reported implementing at a lower level than during the grant-funded period. Only 31% of leaders of classroom-based programs self-reported implementing strategies at the same or higher level than during the grant-funded year. Leadbeater et al. (2015) conducted qualitative analyses of interview data with staff from eight elementary schools who had been using a program for two years to prevent peer victimization. They found, one year later, staff at 5 of 8 schools (63%) were fully implementing the program, and three had sustained implementation of only the less complex components. Lastly, another study using interview data examined sustainability of an evidence-based school-home program for students at risk of behavior problems (Woodbridge et al., 2014). The randomized clinical trial was conducted in 48 elementary schools (24 in the intervention condition). Despite positive program outcomes, staff at only 5 of 24 schools (21%) continued to implement the program two years later.

Collectively, these studies highlight the nascent nature of the school-based literature on sustainability and suggest low to moderate rates of sustained implementation. The actual rates may be even lower because all studies relied on self-report data to assess sustainment. This is a significant limitation, as teachers tend to report higher strategy use than that documented by observation (McLeod et al., 2022). Consistent with recommendations in previous reviews (Shelton et al., 2018; Stirman et al., 2009), this research indicates that objective measurement of sustained implementation is needed to circumvent bias in self-report data.

Intervention Spread: Definitions and Recommendations

Intervention spread represents the adoption of an intervention by new users or within new contexts over time (Horton et al., 2018). The concepts of spread and sustainment are related and desired if the goal is for the intervention effects to persist in a school over time or across teachers, outside of the intensive research team supports. Given that sustainment research in schools is its infancy, we are unaware of any studies that quantitatively examine new adoption or spread of intervention practices in school buildings where randomized trials have taken place.

Intervention spread falls at the nexus of dissemination research and implementation research (see Baker et al., 2021 for review and distinction). Dissemination of information is intended to increase the recipients’ knowledge, intentions, and motivations, in a way that prepares them for adopting the practice. The result of effective dissemination is stakeholder uptake of practices. Following a randomized clinical trial, dissemination efforts often aim to encourage new stakeholders to use knowledge gained from the trial in their real-world practice.

Horton and colleagues (2018), however, argue that the process of thinking about intervention adoption and spread has been overly simplistic to date. They state that researchers need to give greater attention to the social context (e.g., who is implementing the intervention) and the symbolic context (e.g., what symbols denote that the intervention is a priority in the organization). Further, and importantly, dissemination may only be successful when there is capacity building within the organization to provide the training and supports needed to facilitate intervention adoption and sustainment. Indeed, one foci in the sustainability literature is on building capacity for persistent use of practices (Han & Weiss, 2005; Schell et al., 2013).

Professional Learning Communities as a Context for Sustainability and Spread

The trial of the MOSAIC program (Mikami et al., 2022) involved intensive problem-solving coaching with observation and performance feedback for teachers, based on findings that these supports are effective in improving teachers’ implementation of classroom EBPs (Solomon et al., 2012; Stormont et al., 2015). Positive outcomes of this type of coaching include improvements in implementation quality (Sutherland et al., 2015) and frequency of teacher strategy use (Owens et al., 2017; Pas et al., 2015), as well as reductions in disruptive student behavior (Owens et al., 2020; Reinke et al., 2008). Further, a study by Bierman and colleagues (2013) offers promising evidence that intensive coaching during the implementation year was associated with teachers’ sustained implementation of those classroom-based strategies one year later.

Despite the demonstrated success of this type of intensive coaching, it is costly in time and resources and rarely replicable under typical school conditions. Namely, there are monetary costs to training internal or hiring external coaches and school professionals must adapt their infrastructure and schedule to support teacher–coach meetings. This challenge underscores the importance of building capacity within schools to support intervention sustainment and spread.

One promising approach to establishing this capacity is through the use of teacher professional learning communities (PLCs; Stoll et al., 2006a, 2006b). PLCs are regular meetings of school personnel (usually teachers in the same school, or who teach similar grade levels), where members support one another in the implementation of effective practices to improve student outcomes. PLCs are common in North American schools, suggesting that they are already part of the existing infrastructure. Effective PLCs include self-reflection, collaborative learning, community building, having a shared vision and values (Stoll et al., 2006a, 2006b; Watson, 2014), as well as goal setting, troubleshooting, use of accountability measures, and strategies that reinforce teachers’ confidence (Richmond & Manokore, 2011). Darling-Hammond and colleagues’ (2017) review revealed a connection between PLCs that facilitate ongoing job-embedded learning and positive academic and behavioral student outcomes.

PLCs have the potential to encourage intervention sustainment, as well as adoption and spread of intervention practices by new teachers. PLCs allow for the establishment of shared norms, values, and intentions which are theorized to play a critical role in behavior change (Ajzen, 1991; Rogers, 2010); this process may shape perceptions of ease or difficulty of classroom practices. Crucially, in the existing structure of most PLCs, one member (typically, one teacher) is designated as the group leader. Peer leaders may be critical for both adoption and sustainment, as there is evidence that people may be more persuaded by people like themselves (i.e., teachers to teacher) than by others (e.g., Atkins et al., 2008). Similarly, there is some evidence in health care domains that horizontal support and promotion by peers may be more influential in setting norms than top-down promotion (Dixon-Woods et al., 2011). Collectively, this indicates that PLCs may represent a venue that reflects typical practice in schools and through which teachers’ intensions can be shaped. Teachers previously trained in an intervention can help one another to sustain practices once research team supports are withdrawn. In addition, PLCs may be able to promote spread of intervention practices, as those who are previously trained can support those who are new to the intervention (e.g., communicating about ease of practice and sharing ideas for how to overcome barriers and apply strategies within the context).

There is some literature suggesting that PLCs may be specifically relevant to sustainment of intervention practices that promote social, emotional, and behavioral functioning of students. Andreou and colleagues (2015) found that 88% of participants identified regular team meetings for training, troubleshooting, and accountability as beneficial to sustained implementation of behavioral supports. In addition, 65% reported building a “community of practice” or networking with other implementers of the behavioral interventions as beneficial to sustainment (Andreou et al., 2015). Similarly, McIntosh and colleagues (2015) found that the frequency of team meetings, as well as access to consultation and teams sharing data with all school staff, was related to self-reported sustainability of school-wide positive behavioral intervention and supports.

Current Study

To enhance the quality of practices provided to students in the classroom, it is critical to learn how EBPs can be sustained and spread. Implications can be meaningfully enhanced by research that uses observation to measure teachers’ use of practices in addition to self-report. In this study, we explored the extent to which MOSAIC-trained teachers could sustain practices with reduced support in the year following the randomized trial (sustainment), whether teachers in the same building who did not participate in the trial could adopt the practices under support conditions that align with typical practice (spread), and whether PLCs can enhance strategy use in the follow-up year. We adopted Stirman and colleague’s (2012) definition of sustainability and recommendations for enhancing methodological rigor: (a) measuring strategy use over one year after the randomized trial, (b) assessing core program elements (i.e., MOSAIC strategies most predictive of student outcomes), (c) including both objective and self-report assessments of implementation, (d) using dimensional assessment of outcomes, and (e) supporting organizations’ capacity to continue implementation via PLCs (Moore et al., 2017; Shelton & Lee, 2019). Based on the theory of planned behavior (Ajzen, 1991), we also assessed both prospective self-reported intentions to use the strategies, as well as retrospective self-reported strategy use at the end of every two weeks. We hypothesized that teachers who received intensive coaching to support implementation of the MOSAIC intervention would sustain strategy use during a follow-up year. We hypothesized that under realistic support conditions in the follow-up year, teachers who were new to the MOSAIC intervention would successfully use strategies in the classroom, although not to the extent of teachers who had received intensive coaching the previous year. Finally, we hypothesized that teachers who attended more PLC meetings would have greater strategy use than those who attended fewer PLC meetings.

Method

Participants

The randomized trial of MOSAIC enrolled 34 general education teachers of grades K through 5 (17 in the intervention condition; 17 in the typical practice control condition) equally distributed across two study sites (Authors, in press). We invited all 34 teachers to participate in this follow-up study (the year following the randomized trial), as well as any other teachers in the same building who were interested in implementing MOSAIC strategies. Consented participants included: (a) 13 of the original 17 MOSAIC teachers (76%) who participated in the intervention condition in the randomized trial (MOSAIC group), and (b) seven of the original 17 control teachers (41%) who participated in the control condition, plus 10 new teachers (collectively, the latter two groups represent the new-to-MOSAIC group). Data from MOSAIC group (from the implementation year and follow-up year) allowed us to evaluate sustainment, and data from the new-to-MOSAIC group (follow-up year) allowed us to evaluate the spread of and adoption of MOSAIC strategies among teachers new to MOSAIC.

In the current sample, 97% were female, 94% were non-Hispanic, White, 53% had earned a Master’s degree. Participants had an average of 13.05 years of teaching experience. In the randomized trial sample, 94% were female, 85% were non-Hispanic, White, and 47% had earned a Master’s degree. On average, teachers in the randomized sample had 11.5 years of teaching experience (which would represent 12.5 years of experience at the time of the follow-up year). Chi-square and t-test results indicate that, with the exception of race (i.e., 2 teachers of color were not retained in the current, (X2 (4, n = 20) = 40.0, p < 0.001), there were no statistically significant differences between participants in the randomized sample and the follow-up sample. Thus, the current sample is comparable to and representative of those in the randomized trial. Of note, it does not appear that the four MOSIAC teachers who declined did so primarily out of lack of interest or support for the program (i.e., one moved to a new school; one went on maternity leave; one was taking on a new teacher intern and did not have time for the research project; one did not state a reason other than being too busy).

Procedure

Approval of the randomized trial and current study was granted by the university and school district review boards at each site. In the year following the trial, we approached all teachers who had participated in the trial and other general education teachers at the same schools and invited them to participate in the study. Interested teachers signed consent forms.

MOSAIC Randomized Clinical Trial

In the randomized trial, teachers were randomly assigned within each school to either the MOSAIC condition or the control condition for the 2018–2019 academic year (teachers were aware of their assigned condition). In the spring prior to the 2018–2019 academic year, MOSAIC teachers attended a 2-h orientation in which the research team and other study staff members provided the MOSAIC manual, described the program rationale, and taught the MOSAIC strategies through the use of pictures and video models. Teachers who collaboratively worked with us during the intervention development process recommended that we provide the manual and training in the spring of the year prior to implementation so that teachers could adequately prepare space and materials in their classroom for some of the strategies (e.g., space for classroom expectations, CARE time and take-a-break space; see descriptions below).

Throughout the academic year, MOSAIC teachers were intensively coached by study staff members. On average, teachers received 13.4 coaching sessions (SD = 1.6, range = 9–16) with an average duration of 39 min per session. All sessions were 1:1 between the coach and the teacher. Coaches conducted live classroom observations of teachers twice a month to record implementation integrity and gain information for the coaching sessions. Coaches were aware of teacher condition and conducted observations of teachers both in the MOSAIC group and control group (however, observers who completed inter-rater observations were not aware of teacher condition, mitigating concerns about bias or drift on the part of facilitators). Following each observation, coaches emailed their teachers with performance feedback.

During sessions, coaches reviewed the teacher’s use of strategies (i.e., highlighting strengths and problem-solving challenges), introduced new strategies, and developed a plan with the teacher for future strategy use. MOSAIC strategies aim to: (a) increase students’ social and behavioral competencies and (b) enhance inclusivity of peers (see Appendix for list of strategies). MOSAIC strategies are designed to be infused into daily practices (not at a specific time of day). Teachers were asked to use all strategies for their whole class and to apply some strategies more frequently to target children (students with elevated symptoms of attention-deficit/hyperactivity disorder (ADHD) and peer problems; see Mikami et al. 2022 for details).

Foundational strategies were introduced during the fall (e.g., Reviewing/Reinforcing Expectations for Inclusiveness, Personalized Greetings, Check-Ins), and additional strategies were introduced in the winter (e.g., CARE Time, Highlighting Positive Attributes, Discreet Corrections) and spring (e.g., Take-a-Break Space). Although 26 strategies were implemented by MOSAIC teachers, 10 core strategies that were most predictive of positive student outcomes in the pilot trial were prioritized (Mikami et al., 2020; see Appendix). At the end of the trial year, the research team and study staff members provided the same 2-h MOSAIC orientation training initially provided to MOSAIC teachers to any interested staff in participating schools. This was conducted in the spring as recommended by prior teachers to give them greater time to plan for strategy use from the start of the year. During this training, we asked MOSAIC teachers to share their experiences as peer leaders and influencers. All other teachers were given the MOSAIC manual and encouraged to communicate with MOSAIC teachers about implementation.

PLCs in the Follow-Up Year

Follow-up assessments occurred during the 2019–2020 academic year. All data were collected before the COVID-19 pandemic school closures in March of 2020. During the follow-up year, we held 5 PLC meetings that were facilitated by a MOSAIC teacher with support from a study staff member (3 were held in the fall; 2 were held in the winter prior to COVID-related school closings). We developed suggested agendas and scripts for facilitators designed to create a culture of learning, communication, collaboration, goal setting, accountability, and idea sharing related to MOSAIC strategy implementation and overcoming barriers to use.

To build capacity, PLC members were reminded that the MOSAIC teachers had gained substantial expertise and were valuable resources for troubleshooting challenges. Facilitators encouraged reflections from group members by asking open-ended questions and using a Socratic approach to elicit teachers’ talking about successes or challenges in implementation. Teachers were encouraged to consult with one another during and between meetings. During PLC meetings, teachers shared how they were using the prioritized MOSAIC strategies and creative ways they fit strategy use into their schedule; they also shared challenges and worked as a group to collaboratively develop solutions. If questions were posed to facilitators, they redirected the question to the PLC leader or other MOSAIC teachers before answering, so as to facilitate peer consultation. Principals were encouraged to add a MOSAIC “tip of the month” to their monthly newsletters and to remind teachers of the strategies and the PLC at staff meetings.

Measures

Observed Strategy Use

In the trial year and the follow-up year, live observations were conducted in teachers’ classrooms to record use of each MOSAIC strategy; observers were aware of teacher condition (i.e., MOSAIC versus new-to-MOSAIC, although those conducting inter-rater observations were not). In the trial year, teachers were observed an average of 23.16 times (SD = 9.90). In the follow-up year, teachers were observed an average of 9.68 times (SD = 1.32) between September and March (with no significant differences between MOSAIC and new-to-MOSAIC teachers in number of observations). Observations were 40 min in duration (to corresponded to the length of a period in our school buildings), divided into 8-min blocks. We calculated proportion scores for each time period reported (e.g., monthly or entire year), which represent the number of times the teacher was observed using the strategy divided by the total periods observed. Observers were trained research assistants. We double-coded 34% of observations in the trial year and 38.5% in the follow-up year to calculate inter-rater reliably using inter-class correlation (ICC) coefficients. Using Cicchetti (1994) convention for interpreting ICC for inter-rater reliability ICCs for the strategies described below was excellent in the trial year (≥ 0.90) and the follow-up year (≥ 0.93).

Teacher Self-Report of Strategy Use

To align with previous studies of sustainment, we assessed teacher self-report of strategy use (although we viewed it as a secondary outcome to observation data). We assessed self-report prospectively at the first PLC meeting. All teachers were encouraged to review the MOSAIC manual and develop a plan by specifying which strategies they intended to use, as well as identifying up to five strategies to prioritize. We did this, in part to assess the extent to which teachers in both groups were aware of all strategies and to have a baseline measure of intensions to give context to our interpretation of later self-reported and observed strategy use.

In addition, every two weeks (September through March) all teachers were sent a strategy use survey comprised of a checklist of strategies and asked to check all strategies they implemented in the past two weeks. The strategies on the survey aligned with those coded by observers with two exceptions (a) observers coded Greetings and Check-ins separately, whereas teachers were only asked about Check-ins; (b) observers coded distinct nuances when teachers highlighted a positive attribute (if the attribute was (1) enduring or non-enduring and (2) related to an academic or non-academic characteristic), whereas teachers were simply asked if they highlighted a positive attribute (see Appendix). These modifications to the self-report survey were done to reduce teacher burden when completing the survey.

PLC Attendance

Attendance was taken by facilitators at each meeting. On average, teachers attended 3.77 sessions (range was 2 to 5; SD = 0.82). We categorized teachers into three groups: low attendance (attending 2–3 meetings; n = 12), moderate attendance (attending 4 meetings; n = 12), and high attendance (attending 5 meetings; n = 6). These groups allowed us to examine possible linear associations and to have sufficient and relatively similar cell sizes in each group.

Data Analytic Strategy

We calculated self-reported strategy and observed strategy use (observation data represent the primary outcome measure; self-report data represent a secondary outcome measure) among MOSAIC teachers and new-to-MOSAIC teachers during the follow-up year. To test Aim 1 (sustainment), we calculated strategy use among MOSAIC teachers in the current, follow-up year relative to their respective strategy use in the previous year during the randomized trial. High values of strategy use in the MOSAIC group in the follow-up year that show little drop-off from the trial year would indicate sustainment. To test Aim 2 (spread), we reported descriptive statistics regarding the amount of strategy use the new-to-MOSAIC teachers were doing in the follow-up year, and compared strategy use among the MOSAIC teachers relative to the new-to-MOSAIC teachers. We also compared these values in the follow-up year relative to their respective strategy use in the previous year during the randomized trial (for the teachers in the current study who were in the control group in the randomized trial). Strategy use in the new-to-MOSAIC group in the follow-up year would indicate spread, though lower values of strategy use in the new-to-MOSAIC group relative to the MOSAIC group in the follow-up year were expected.

Because MOSAIC teachers could progressively show less strategy use over time (i.e., sustainment drops off over time), and new-to-MOSAIC teachers could progressively show increased strategy use (i.e., spread increases over time), we also report information about observed strategy use across September and October compared to use across November through February.

Although we conducted independent samples t-tests on strategy use variables, the sample size is underpowered for statistical significance. To reduce interpretation of potentially spurious findings, we report Hedge’s g effect sizes and only interpret findings that are replicated across multiple strategies. For interpretation, we consider effect sizes of 0.24 or lower to represent no group difference, between 0.25 and 0.79 as small to medium effects, and 0.80 or higher as large group differences. In addition, because some MOSAIC strategies (e.g., Greetings, Reviewing/Reinforcing Expectations for Behavior) are not unique to MOSAIC, such that we expected some new-to-MOSAIC teachers to be using them, we report data about individual strategies.

Lastly, in order to explore the extent to which PLCs may have facilitated sustainment and spread (Aim 3), we examined observed strategy use among low, moderate, and high PLC attendance groups and provide effect sizes comparing low (2–3 meetings) to high (4–5 meetings) groups. All statistical analyses were computed using SPSS 27.

Results

Aim 1: Sustainment

Based on self-report data, most MOSAIC teachers reported that they intended to sustain use of all MOSAIC strategies (see Table 1). Over 75% of MOSAIC teachers reported intent to use each strategy. However, there was wide variability in strategies that teachers prioritized as the top five strategies they intended to use (see Table 1). As an indicator of self-reported sustainment, with the exception of CARE Time, MOSAIC teachers reported high use of all strategies over time, i.e., reporting use of the strategies in the last two weeks on over 80% of probes (see Table 2).

Table 1 Percent of Teachers by Group Reporting Intended and Prioritized Use of the Strategy
Full size table
Table 2 Self-Reported Teacher Strategy Use of Core MOSAIC Strategies Across the Year by Group
Full size table

As an indicator of observed sustainment, Table 3 contains the level of observed implementation by the MOSAIC teachers during the randomized trial year (see MOSAIC previous year columns). In addition, % Relative Use MOSAIC column reveals the level of observed implementation among MOSAIC in the follow-up year as compared to the level of observed implementation by MOSAIC teachers during the randomized trial year. For MOSAIC teachers, the percent relative use of strategies ranged from 66% (CARE Time) to 145% (Greetings). Thus, MOSAIC teachers showed good sustainment of 75% of strategies (80% or higher), or greater use of some strategies in the follow-up year as compared to the randomized trial year. Stated differently, MOSAIC teachers demonstrated a less than 20% decline in the use of most strategies, when intensive supports were withdrawn, and they transitioned to support conditions that aligned with typical practice.

Table 3 Observed Teacher Strategy Use of Core MOSAIC Strategies Across the Year by Group
Full size table

Aim 2: Intervention Spread

Most new-to-MOSAIC teachers reported that they intended to use all MOSAIC strategies (see Table 1). With the exception of two strategies (CARE Time and Question of the Day), over 75% of teachers in this group reported that they intended to use each strategy. Again, there was wide variability across teachers in their top five prioritized strategies.

As an indicator of teacher reported strategy use throughout the year, new-to-MOSAIC teachers reported high use (use in the last two weeks on over 80% of probes) of four strategies: Check-Ins, Reviewing and Reinforcing Expectations for Behavior, and Discreet Corrections. They reported less use of strategies that are more unique to MOSAIC (e.g., Reviewing and Reinforcing Expectations for Inclusiveness, CARE time, and Highlighting Positive Attributes).

Collectively, the results of observed strategy use by new-to-MOSAIC teachers and control teachers during the randomized trial year (see Control previous year columns) provide evidence of spread. Namely, bolded values show that, for 6 of 10 (60%) strategies, observed strategy use by new-to-MOSAIC teachers in the follow-up year was higher than that of control teachers in the previous year. In addition, % Relative Use Control column reveals that for new-to-MOSAIC teachers, the percent relative use of strategies ranged from 0% (CARE Time) to 111% (Highlighting Personal Non-Enduring Qualities) of what the MOSAIC teachers achieved during the randomized trial. Collectively, these findings suggest that strategy use did spread among new teachers not previously trained in MOSAIC under support conditions that are more realistic for schools to maintain.

Unsurprisingly, however, the new-to-MOSAIC teachers (who were supported with only PLCs) were not implementing strategies on average to the same extent as the MOSAIC teachers who had previously received intensive coaching during the randomized trial. According to self-report data, MOSAIC teachers self-reported using 8 of 9 (88%) strategies more than new-to-MOSAIC teachers (effect sizes ranged from 0.32 to 1.16; see Table 2), with the difference reaching statistical significance for three of the strategies (Reviewing Expectations for Inclusiveness, Reinforcing Expectations for Inclusiveness, and Highlighting Personal Qualities). According to observed strategy use data, MOSAIC teachers used 6 of 10 (60%) strategies more than new-to-MOSAIC teachers (effect sizes range from 0.25 to 0.90 for these six strategies) (Fig. 1).

Fig. 1
figure 1

Examples of higher observed strategy across four months by MOSAIC than new-to-MOSAIC teachers

Full size image

Aim 3: Strategy Use and PLC Attendance

Figure 2 reveals the pattern of observed MOSAIC core strategy use for the year use among low, moderate, and high attenders of the PLCs. When looking for a linear association, the pattern suggests that attendance was associated with higher strategy use for only 2 of 10 (20%) strategies. However, when comparing low attendance (2–3 meetings) to high attendance (4–5 meetings) groups, the patterns suggest that greater attendance was associated with greater strategy use for 8 strategies: Check-ins (Hedges’ g = 0.66), Reviewing Expectations for Behavior (Hedges’ g = 0.73), Reinforcing Expectations for Behavior (Hedges’ g = 0.35), Reviewing Expectations for Inclusiveness (Hedges’ g = 1.39), Reinforcing Expectations for Inclusiveness (Hedges’ g = 0.47), CARE Time (Hedges’ g = .0.54), Highlighting Personal Enduring Qualities (Hedges’ g = 0.28), and Discreet Corrections Hedges’ g = 1.05). For the non-MOSAIC group, high attenders used Reviewing Expectations for Inclusiveness (t(15) = 3.19, p > 0.01) and Discreet Corrections (t(15) = 3.19, p > 0.01) significantly more than did the low attender group; (t(15) = 2.36, p > 0.01). For the MOSAIC group, high attenders used Reviewing Expectations for Inclusiveness (t(11) = 1.85, p > 0.01) and CARE Time (t(11) = 1.99, p > 0.01) significantly more than did the low attender group. Thus, there is some evidence that a relationship between attendance and observed implementation exists; however, these data cannot support a definitive conclusion in either direction.

Fig. 2
figure 2

Core strategy use among teachers attending PLCs at a low, moderate, high Rate

Full size image

Discussion

To maximize positive outcomes associated with evidence-based school interventions, we need to study intervention strategy use by teachers following training procedures in randomized trials. In this study, we examined the extent to which MOSAIC-trained teachers sustained practices with reduced support in the year following the randomized trial (sustainment), whether teachers in the same building who did not participate in the trial could adopt the practices under support conditions that align with typical practice (spread), and whether participation in PLCs could enhance strategy use in the follow-up year. Under these conditions, we found evidence of sustainment and spread; however, their associations with attendance at PLCs (as measured in this study) were inconsistent and modest.

Sustainment and Spread

The pattern of data in Table 1 suggests that teachers have intentions for sustainment (in the MOSAIC group) and spread (in the new-to-MOSAIC group). This may represent an indicator of teachers’ perceived feasibility and effectiveness of MOSAIC strategies, and/or strategy use that was not detected via observations (e.g., if observations happened in the afternoon, Greetings and/or Community Circle may not have been observed). On the other hand, these data in combination with data in Tables 2 and 3 also highlight that teachers may be overly optimistic about intended use; teacher reports of intended behaviors exceeded reports of observed behaviors, which is consistent with other study findings (e.g., McLeod et al., 2022).

Teachers who received intensive coaching the previous year (MOSAIC group) were observed to use most strategies more often than those who had not (new-to-MOSAIC group). More importantly, MOSAIC teachers’ rates of observed strategy use in the follow-up year showed less than 20% decline relative to their rates of use when receiving intensive coaching. These rates of sustainment are higher than those reported in previous school-based sustainment studies (Cooper et al., 2015; Woodbridge et al., 2014), perhaps reflecting the benefits of the intensity of the initial coaching provided (i.e., year-long observations with email and live performance feedback). In addition, the finding that teachers implemented some strategies (i.e., Greetings, Reinforcing Expectations for Behavior and Inclusiveness) more in the follow-up year than in the randomized trial year may offer evidence that teachers perceive these strategies as easier and/or more effective than others. Lastly, the patterns in Fig. 1 suggest that the sustainment was durable into the second half of the school year.

In addition, we found evidence of successful intervention spread to new teachers. New-to-MOSAIC teachers self-reported using more than half of the strategies of interest (5 of 9) in the past two weeks. For 6 of 10 (60%) strategies, new-to-MOSAIC teachers’ observed implementation in the follow-up year was higher than that of control teachers in the previous year. These observation data offer promise that, among teachers who are interested in implementing the program and who report intentions to implement, initial training and ongoing support from other trained teachers may facilitate their use of strategies. Unsurprisingly, although new-to-MOSAIC teachers appeared to adopt some strategies, they did not do so at the rate of teachers who had previous experience with the intervention.

We acknowledge that some strategies are not unique to MOSAIC; thus, it is not surprising that new-to-MOSAIC teachers performed similar to MOSAIC teachers for some strategies. However, the data in Table 3 suggest that new-to-MOSAIC teachers outperformed previous Control teachers on both general strategies (e.g., Reinforcing Expectations for Behavior) and strategies that are more unique to MOSAIC (e.g., Highlighting Enduring Qualities), suggesting some specificity in the transfer of MOSAIC strategies either via the training and/or trained peer teachers. A strength of this study was connecting new-to-MOSAIC teachers to peers who had previous experience with implementation of the intervention. These findings provide rationale for continuing research on intervention spread in schools, particularly with consideration of context and capacity building for intervention support (Horton et al., 2018).

Professional Learning Communities

We anticipated that attendance at PLCs would serve an important support function and result in greater strategy use among all teachers. We found a small and inconsistent association between PLC attendance and strategy use. The PLC literature for teacher pedagogy (Darling-Hammond et al., 2017) that suggests that PLCs may facilitate implementation of positive behavioral supports; however, because we only found this association for a few strategies, we cannot draw a definitive conclusion about the benefits of PLCs in this study. Our findings do suggest that PLCs without prior intensive coaching, may not be sufficient to bring ‘new’ teachers to the same level of implementation as previously trained teachers. Namely, when considered in combination with implementation rates for new-to-MOSAIC teachers presented in Table 3, PLC attendance was not a sufficient replacement for a year of intensive coaching. Unfortunately, without random assignment to PLC or not, we cannot deduce the impact on sustainment among those who received the initial training.

Limitations

Although there are several strengths to the current study (e.g., sample representative of the randomized trial, assessment over several months, use of observation data), it also has limitations. Given the small sample size, findings should be considered preliminary. Although statistically significant findings are impressive given a small sample size, the absence of statistical significance (where desirable, e.g., indicating equivalency between groups) could be explained by sufficient sustainment or spread, or could be due to a lack of power to detect differences. In addition, the design was naturalistic. Stronger conclusions can be drawn from studies that randomly assign teachers to different fidelity maintenance strategies. Further, although observation data are a strength, the observations were limited to once per month and thus may under or over-represent actual strategy use in either year. All participants volunteered to participate, which may have increased our levels of sustainment and spread. The new-to-MOSAIC group consisted of members in the control group and those who are interested in the program; thus, other factors may have influenced results or introduced a bias toward using the intervention more than the average teacher. We did not explicitly measure teacher perceptions of factors associated with PLCs or facilitators or barriers to implementation and COVID school closures prevented contact at the end of this assessment year. We did not measure fidelity to conversations within the PLCs. Finally, this study did not include student outcome data; although results are promising for indicating sustainment and spread of the MOSAIC intervention with reduced support during the follow-up year, it remains unclear if the levels of sustainment and spread were sufficient to produce positive changes for students.

Implications and Future Directions

Given the nascent state of the literature on sustainment of school-based interventions, our results make an important contribution and offer insights for future study. First, consistent with coaching and consultation studies (Cook et al., 2019; Stormont et al., 2015), intensive problem-solving consultation that includes performance feedback is an effective method for improving teachers’ use of EBPs. This study suggests that the benefits may sustain into the year following intensive training. Although this offers optimism for intensive coaching, it is important to recognize that sustainment may not continue without additional implementation supports and/or accountability for sustainment, such as continued observation and PLC discussions.

Second, our findings also suggest that PLCs are not a sufficient replacement for intensive coaching as part of initial training for a new program, but there are many conceptual reasons why continuing research may be warranted. For the current study, group membership (i.e., MOSAIC or new-to-MOSAIC) may have influenced participation in PLCs, as previous experience with the intervention may have facilitated buy-in and encouraged attendance above and beyond the average teacher who has no experience with the intervention. Future research could experimentally manipulate possible mechanisms of change within PLCs, such as identification of a specific implementation goal or solution, and sharing of data or information at the next meeting to help evaluate of the utility of the solution selected. In addition, it could be valuable to evaluate the potency of PLCS that do or do not include school administrators in the process.

Lastly, our data suggest that other sustainment strategies should be evaluated in combination with PLCs. Namely, teachers’ initial intentions to implement were high. Perhaps these intentions could be harnesses through a self-directed, technology-directed, or peer-directed implementation planning process, wherein teachers set specific implementation goals and plan for potential barriers and solutions, such as that evaluated in consultation studies (e.g., Sanetti et al., 2018). Further, there may be an additional process that administrators or PLCs leaders could use (e.g., short cycle self-assessments) to enhance teacher accountability to these plans.

An additional strength of this study was the multi-modal assessment of intervention strategy use, with both observation and self-report data (Tables 2 and 3). Results indicate that teachers tended to report use of strategies at a much higher rate than what was observed in their classroom. The data appear to support potential bias of self-reported use of strategies, which has previously been identified as a limitation in the research literature. Self-report data were included in the study in alignment with other studies of sustainment and spread, but results indicate that an objective measure of integrity or strategy use, like observation data, may be warranted.

Conclusion

To maximize the impact of evidence-based practices, we must invest in understanding factors associated with teachers’ sustained use of strategies under typical conditions. Our results suggest receiving intensive coaching in the previous year was associated with sustained use of strategies (i.e., less than 20% decline in use across years). This finding suggests the promise of intensive coaching to lead to more durable changes in teacher practices with less support in subsequent years. Further, initial training coupled with PLCs may offer an important foundational mechanisms or social infrastructure to support the spread of interventions from trained teachers to other interested teachers, although additional supports are likely warranted.