Introduction

Although evidence-based programs (EBPs) have significant potential for improving health and educational outcomes, research on their translation into applied settings has yielded disappointing results, frequently due to implementation factors (Durlak & DuPre, 2008; Elliott & Mihalic, 2004; Hallfors & Godette, 2002). Implementation is multifaceted, but “implementation fidelity” generally refers to delivering an intervention in a manner consistent with the protocols establishing its effectiveness (Dane & Schneider, 1998). Implementation may be particularly salient when EBPs are delivered under conditions substantially different from those in the original research (Castro et al., 2014; Kumpfer et al., 2017). In these situations, both planned and unplanned adaptations may occur, leading to uncertainty regarding participant outcomes (Durlak & DuPre, 2008). Despite these challenges, there are many reasons to attempt the translation of EBPs into real-world settings, especially into underserved communities, and to better understand implementation variability. While some studies suggest benefits of adapting interventions for new contexts (Barrera et al., 2017; Brody et al., 2004, 2006), others show that minimal adaptation is needed for success with diverse populations (Chaffin et al., 2012).

The goal of this study was to pilot test the Head Start Research based, Developmentally Informed (REDI) program, an evidence-based school readiness intervention, in two midwestern Head Start programs, one on a Native American reservation and one in an adjacent community. This context varied substantially from that of the original research trials, which occurred in Pennsylvania. We followed an established model for EBP adaptation (Chen et al., 2012), which included gathering systematic stakeholder feedback regarding the need for planned adaptations of the unmodified EBP, both during and after program implementation. Simultaneously, we documented spontaneous adaptations occurring in the new context. Finally, we explored whether certain aspects of the multi-component REDI program were more likely to be changed, and whether changes were related to temporal factors across the school year.

EBP Program Implementation in Real-World Settings

Early scholarship in implementation science emphasized the importance of implementation orthodoxy, noting that programs yielded their best outcomes when implemented with fidelity to the original model (Derzon et al., 2005; Durlak & DuPre, 2008; Wilson et al., 2003). This literature also documented the negative impacts of unplanned changes inconsistent with the intervention logic model (Durlak & DuPre, 2008; Humphrey et al., 2018). Nevertheless, research continued to document high rates of unplanned adaptations to EBPs in applied settings (Moore et al., 2013). Gottfredson (2001) estimated that the typical fidelity of a widely disseminated EBP ranged between 42 and 68%. A review of Blueprint Program implementation found that only 57% of the implementers delivered the complete intervention dosage (Elliott & Mihalic, 2004).

Studies of EBP adaptation have delineated both the kinds of changes made by implementers and the reasons for making them (Moore et al., 2013; Wiltsey Stirman et al., 2013, 2019), with the goal of distinguishing detrimental changes from those that might enhance program fit and effectiveness. In their review of EBP dissemination in Pennsylvania, Moore et al. (2013) found that 44% of implementers changed the intervention in some way. Most adaptations involved changes to the program procedures, dosage, and content, with relatively few (22%) reporting cultural adaptations. The main reasons for these changes included lack of time and resources or staffing challenges. Additional factors included difficulty recruiting and retaining participants and participant dissatisfaction (Moore et al., 2013).

Similarly, Escoffery et al. (2018) reviewed 42 EBPs implementation studies and found that 100% of the implementers made changes to the program. In this study, adaptations primarily related to cultural fit of the intervention (64.3%) and needs of a different target population (59.5%). The most common adaptations identified included changes to program content, with changes to context and delivery also frequently reported. Most changes were not made using any systematic adaptation framework, although many were deliberately planned in consultation with developers and stakeholders. Approximately one-third of the studies reported deleting or shortening program content, raising concerns regarding intervention fidelity (Escoffery et al., 2018).

Wiltsey Stirman et al. (2013) reviewed 32 studies reporting EBP adaptations and coded changes into several categories, including who made modification decisions, which aspects of the intervention were modified, the level of intervention delivery changed, and the nature of the change. This review, which did not distinguish between planned and unplanned changes, found that changes included cultural tailoring and additions and deletions to program content.

Models for Planned Adaptation

Recognizing the ubiquity of EBP adaptation in new contexts, as well as the fidelity threats of unplanned changes and the potential benefits of properly tailored interventions, implementation researchers have proposed several systematic approaches to EBP adaptation. Most of this work is focused on cultural adaptation and emphasizes what should and should not be changed when adapting an EBP (Domenech Rodriguez et al., 2011; Resnicow et al., 2000). Lau (2006) and Barrera and Castro (2006) identified four conditions justifying EBP adaptations: (1) the EBP could not engage the target population; (2) risk and protective factors of the target group differed from those of the research sample; 3) the target group presented unique symptoms of a common disorder; and 4) the original EBP was not efficacious for the target population.

Several models for adapting EBPs have been proposed, most focused specifically on cultural adaptation. While these vary somewhat in their procedures, all require strong partnerships between community stakeholders and researchers. Barrera and Castro (2006) suggested a five-step model that begins with collecting data on risk and protective factors and conducting focus groups to assess stakeholder impressions of the EBP; making preliminary adaptations based on the information gathered; piloting the adapted intervention; refining the modified EBP based on feedback; and evaluating the adapted program. Similarly, Wingood and DeClemente (2008) developed the 8-step ADAPT-ITT model for EBP adaptation, which also involved pre-intervention assessment of risk and protective factors and an opportunity for stakeholders to “preview” the intervention and give feedback.

A somewhat different process was proposed by Chen et al. (2012), who suggested first implementing the unchanged EBP and then collecting feedback from stakeholders regarding recommended adaptations both during and after the program. A drawback of this approach is that it lacks consideration of systematic data on risk and protective factors required by other frameworks. However, we used this model for planned adaptation because a strength of this process is that the feedback is solicited when stakeholders have had actual experience with the EPB and can make informed decisions regarding desired changes.

The Current Study

The current study examined the implementation of an EBP to promote school readiness, the REDI program, in two Head Start programs: one serving Native American children, and one serving both Native and non-Native children. The goal was to determine how the implementation might spontaneously vary in this new context, as well as to determine the degree to which teachers in these new schools found the REDI program appropriate and would recommend planned adaptations. Following the model described by Chen et al. (2012), the current study’s researchers and community stakeholders agreed to pilot REDI “as-is” and to collect implementer feedback regarding future planned adaptations. Mindful of the research indicating frequent spontaneous adaptations to EBPs in applied settings, we also collected data on changes occurring during implementation of the unadapted program. Based on prior studies of EBP adaptation, we hypothesized the following: (1) spontaneous adaptations would occur despite clear implementation protocols and the provision of ongoing technical assistance (TA); (2) the reasons for changing the program would be largely pragmatic; and (3) given the curriculum structure, training and professional development built into the REDI program, implementation fidelity would be high. We were also interested in two implementation questions related to these hypotheses: whether changes more likely to occur at certain times of the school year; and whether some components of the intervention were more likely to be changed than others.

Method

Partnership Development

Prior to program implementation, the research team met with stakeholders in the target communities to discuss a potential research collaboration and how it might align with their priorities. All stakeholders expressed concern about children’s school readiness and were attracted by the data on REDI’s impact on both social-emotional and emergent literacy skills. REDI was designed as an enhancement to High Scope/Perry Preschool Program (Schweinhart et al., 2005) and Creative Curriculum (Dodge et al., 2002), which are two curricula most often used in Head Start programs. Despite their effectiveness at producing long term, positive outcomes for children, these curricula were both developed in the mid-20th century, prior to research demonstrating the strong link between specific social-emotional competencies and emergent literacy skills in preschoolers and educational disparities associated with poverty (Blair, 2002; Ladd & Profilet, 1996). These skills are explicitly targeted by the REDI curriculum. The fact that REDI could be integrated with existing curricula and the availability of free materials, training, and coach support were considered benefits. Based on these discussions and with their review of the curriculum materials and the REDI evaluation research, local stakeholders concluded that REDI was a good fit for their Head Start programs. The community partners also indicated they would likely recommend some adaptations, but given that they were unfamiliar with the program specifics, they could not anticipate what they would change without trying it first. Therefore, it was mutually agreed that teachers would undertake a pilot implementation of REDI without planned changes, but that the data collection would involve continuous feedback on the fit of the program and the need for modifications of specific components.

Once this plan was finalized, it was submitted to the respective IRBs of the tribe and the lead university, which reviewed and approved all procedures. Building and maintaining partnerships with community stakeholders was essential, given that the research team members were non-Native and did not live the target communities. To reduce the possibility of interpretation bias, all data, data analyses, and manuscript drafts were reviewed and approved by members of the tribal IRB. Additionally, with the exception of the post-implementation interviews with the participating teachers, we utilized the same measurement, implementation, and assessment protocols used in the original REDI trial.

Sample

The sample included three Head Start classrooms in two rural midwestern communities. One program was operated by a tribal community on a Native American reservation and served 100% indigenous students. One classroom, led by a female, Native American head teacher participated in the study from this center. The classroom was located in a community building on the reservation. The other two classrooms were in a larger community adjacent to the reservation and served both Native and non-Native students. Two classrooms each with a lead teacher and participated in the study. In one classroom, an assistant who functioned as a co-teacher also participated. At this site, all participating staff were white and female. All students resided in the community in which they attended Head Start. Both settings were more rural than those in Pennsylvania where REDI was initially evaluated. Similarly to the original sample, all children met federal guidelines for Head Start eligibility. Although these programs included both 3- and 4-year-olds, only the 4-year-olds, who would be kindergarten-eligible the following school year, were included in the study sample. Altogether there were 39 study-eligible children in the three classrooms, of which 29 received parental consent to participate. The mean age of the sample was 4.6 years. Parent data indicated that 41% of the children were Native American and 58% were White; 66% percent were female.

Procedures

The Head Start REDI Program

The Head Start REDI classroom program is an evidence-based, multi-component intervention designed to promote school readiness in preschool children (Bierman et al., 2008). It was developed as an enhancement to existing Head Start programs. REDI has two main components: a classroom-based curriculum targeting children’s school readiness, and a training and professional development model for implementing teachers. Within the classroom curriculum, there are five subcomponents. The first is the Preschool PATHS Curriculum (Domitrovich et al., 1999), which targets four social-emotional learning domains: prosocial friendship skills, emotional understanding, inhibitory control, and problem-solving and conflict resolution skills. PATHS includes 33 scripted, weekly lessons that use stories, role play, and animal puppets to illustrate and practice the targeted skills. PATHS also included a number of Extension Activities, which included crafts and games and were designed to reinforce the targeted PATHS skill. There were 2 suggested PATHS Extension Activities per week. The second is a Dialogic Reading Program adapted from the work of Wasik et al. (2006) and Whitehurst et al. (1994), in which teachers read two books per week chosen to align with the social-emotional themes of the corresponding PATHS lessons and used language enrichment techniques. Each book was to be read twice, for a total of four book reading sessions per week. The third is Sound Games (Adams et al., 1998), which included sequenced activities to promote phonological awareness, a critical component of early reading (Hogan et al., 2005). Sound Games were scheduled three times per week for 10 to15 minutes and progressed in difficulty over the course of the year. The fourth is Alphabet Center, which involved 2 to 3 activities per week to promote children’s letter knowledge. These activities were not sequenced by difficulty, but introduced several letters at a time and included an assessment tool for teachers to track each child’s letter knowledge.

Within the REDI training and professional development component, there were a number of subcomponents: (1) three days (18 h) of in-person training, including two days prior to program implementation and a one-day booster halfway through the school year; (2) four one-hour, online, asynchronous modules offered through the Better Kid Care Professional Development system run by Penn State Extension, that outlined and modeled program implementation; and (3) regular sessions with a REDI coach, who visited each classroom approximately biweekly to observe program implementation and offer feedback and support (Bierman et al., 2008). Teachers were compensated $100 per day to attend the training. The REDI coach had a degree in early childhood education and 20 years of experience teaching and mentoring preschool educators at a nearby tribal college. The coach underwent the same training as the teachers, as well as four hours of individualized, supplemental training in coaching and implementation monitoring from one of the REDI developers.

Prior rigorous, longitudinal evaluations of the REDI classroom program have demonstrated its enduring positive impact on children’s school readiness (Bierman et al., 2014; Nix et al., 2013), long-term social-emotional adjustment (Welsh et al., 2020), and resilience in the face of early adversity (Sanders et al., 2020). The following section describes the measures used to track the implementation fidelity of the REDI curricular subcomponents and professional development utilization. This includes measures of dosage, teaching quality, and spontaneous program adaptations which occurred over the school year.

Measures

Dosage

The dosage was tracked by counting the number of activities reported in the Teacher Weekly Implementation Report. The weekly dosage value was created by summing the activities within each component. The maximum dosage values tracked for each component were: PATHS, 33 lessons (1 lesson x 33 weeks); Dialogic Reading, 66 total books read (2 books per week x 33 weeks; the report did not ask teachers to report the second reading of either book); Sound Games, 99 total possible games across 33 weeks (3 lessons each week x 33 weeks); PATHS Extension Activities, 66 activities (2 lessons each week x 33 weeks); and Alphabet Center, 33 lessons (1 session per week). Dosage was regarded as measures of implementation fidelity and a possible area of adaptation and was used to evaluate hypothesis 1.

Participant Engagement

Teachers were asked to report on child engagement by rating each lesson on a three-point scale from “1” (not well, kids were not engaged or did not understand the lesson) to “3” (very well, kids were very engaged, and the lesson was just right for their skill level). Child engagement was regarded as a measure of implementation fidelity and was used to evaluate hypothesis 3.

Global Ratings of Teaching Quality

Three measures of teaching quality, derived from the Classroom Assessment Scoring System (CLASS: Pianta et al., 2008) were measured over time. These were completed by the coach during the classroom visits and assessed the specific program subcomponents the teachers happened to be doing at the time of the observation. All items involved a 5-point scale from “1” (almost never) to “5’ (almost always). Emotion communication and support (3 items, alpha = .84) describes how much teachers encouraged children to communicate their feelings, used self-control techniques to help children regulate emotions, and how often teachers used feeling words and I-statements with children. Sensitivity and responsiveness (2-items, r = .38) describes how much teachers were available to the children, the teacher’s warmth in verbal interactions, and the teacher’s responsiveness to the children’s interests and attention. Richness of talk (4-items, alpha = .85) describes how much teachers used rich and varied vocabulary, elaborated their discussion with complex sentences, challenged the children cognitively, and used decontextualized language. The modified CLASS was completed at each visit took approximately 20 minutes. The global ratings of teaching quality were regarded measures of implementation fidelity, and were used to evaluate hypothesis 3.

Adaptations

Adaptations were recorded in two ways. First, teachers reported changes and reasons for them in their Weekly Implementation Report, responding to the question, “If [activity] was not completed, please state why.” Second, teachers reported their adaptations in the post-program interview, describing “What modifications you made, if any, regarding each [curriculum component]. These two sources of information regarding adaptation were used to test the first hypothesis.

Training and Professional Development

Training and professional development activities were documented as follows. Training attendance was tracked for each teacher at the three in-person sessions. Consultation measured the number of visits the REDI coach had with each teacher; the number of coaching sessions was summed across the observation reports. Module completion was tracked within the online training system; completing one full module = 1.0; a partial completion was given a 0.5; not viewing or starting a module = 0.0. Training and professional development were an integral part of the REDI program. As such, we examined this as an area for possible adaptation in this new setting compared to prior REDI implementations in hypothesis 1.

Evaluation Procedures

Two tools, one completed by the coach and one by the teacher, were used to track program implementation and teaching quality. Teachers completed the Weekly Implementation Report to report the program components they delivered and any adaptations they made to implementation. The coach completed the modified CLASS measure of teaching qualities following each classroom observation. Observations occurred approximately biweekly. During the observation, the coach rated whatever activity the teacher did during the visit, which was typically a PATHS lesson or a book reading session.

Additional information about program implementation and specific adaptations were collected in individualized end of program interviews with all teachers approximately one month after concluding the program. Interviews were conducted by a member of the research team, lasted about one hour, and teachers were compensated $25.

Data Analysis

Because of the small sample, quantitative data analyses were limited to descriptive statistics (means, frequencies, standard deviations). Qualitative data from end of program teacher interviews and teacher implementation reports were analyzed to provide evidence to support or negate each of the study’s hypotheses.

Results

Hypothesis 1: Spontaneous Adaptations Would Occur

The Results for the Multicomponent Intervention

The teacher implementation data confirmed that spontaneous adaptations occurred. We tracked implementation from weeks 5 through 33, which resulted in 29 possible PATHS Lessons, 58 possible Dialogic Reading sessions, 58 possible PATHS Extension Activities, 29 Alphabet Center activities, and 99 possible Sound Games for a total possible 273 Head Start REDI lessons completed across the year (see Table 1). Alphabet Center activities were implemented the least frequently (62.92% of the total possible lessons) whereas PATHS Lessons were implemented most frequently (97.70% of the total possible lessons). The second most frequently implemented was Dialogic Reading, followed by PATHS Extension Activities, and then Sound Games. Overall, 79.74% of the expected curricular components were delivered.

Table 1 Dosage of curriculum components across the full sample
Full size table

Reported adaptations are listed in Table 2. Adaptations were divided into four categories: activity was not completed; teachers added more to the lesson; teachers made logistical changes to the lesson; teachers substituted a different activity. Overall, the most frequently reported adaptation involved dosage; it was not completing one or more activities. Substituting a different activity (such as replacing a program activity with a holiday-themed activity), was also common. Logistical changes to the activity occasionally occurred; for example, when teachers used a water table when teaching letters instead of the recommended material.

Table 2 Frequency of modification types - teacher weekly implementation reports
Full size table

The information gathered from the post-program interviews was consistent with the weekly reports. Teachers indicated that they most frequently skipped the Alphabet Center and Sound Games activities. The main reasons given were a lack of time and a sense that the material was either not engaging (e.g., “too boring”) for the children or too difficult for them; this was particularly true of the Sound Games.

Figure 1a and e display overall dosage of each program component over time across all three classrooms. Visual displays suggest no clear temporal pattern except for perhaps a slight decrease in dosage over time for three out of the five components: PATHS Lessons, Sound Games, Alphabet Center.

Fig. 1
figure 1

Average number of curriculum components completed by lesson number

Full size image

The Results for the Training and Professional Development Model

Spontaneous adaptations also occurred with the training and professional development model (see Table 3). We expected everyone to attend all three training days; however, full attendance did not occur. As planned, the coach completed all four REDI Better Kid Care modules (1–4) and the three Coaching Modules (BKC 1–3). However, teachers completed on average 2.3 of the 4 modules (SD = 0.58). The number of coaching visits also varied across teachers (M = 12; SD = 2.0), for an average of 2.0 visits per month for each teacher (SD = 0.33).

Table 3 – Teacher professional development activities
Full size table

Hypothesis 2: We Expected that Teachers’ Spontaneous Adaptations Would be Driven by Pragmatic Concerns

The results indicated some support for this hypothesis. Teachers’ stated reasons for their adaptations are reported in Table 4. Most often, teachers reported lack of time as the main reason they could not fully implement the curriculum. Next was the need to prioritize holiday-themed activities over those in the curriculum. Several other reasons teachers indicated were that they forgot to do the activity, couldn’t find the activity in the manual or felt that the activity was too similar to others they were already doing. Needing support from a partner teacher was mentioned only for the Sound Games. No adaptations indicated any philosophical difficulties with the REDI curriculum.

Table 4 Reasons for modifications – teacher weekly implementation reports
Full size table

The teacher post-program interviews were consistent with the implementation logs. Teachers frequently mentioned lack of time as their main reason for failing to complete program activities. However, two of the four teachers stated that their students were bored by the second reading of the Dialogic Reading books and they either stopped or shortened these in order to maintain children’s interest. Additionally, two teachers also stated that the difficulty level of the Sound Games activities was too high for their students and they either modified or deleted the activities as a result. None of the teachers indicated making adaptations based on philosophical considerations, such as students’ cultural background, Head Start program philosophy, or local community norms.

When asked how they would propose to modify REDI in the future, all teachers mentioned integration of math and science concepts into the curriculum, and several expressed a preference for using their own books for Dialogic Reading rather than the books provided. All of the teachers indicated that the REDI program overall was a good cultural fit with their communities. However, the Native American teacher in the tribal Head Start classroom also indicated that this cultural fit could be enhanced with the integration of indigenous language and stories throughout the curriculum.

Hypothesis 3: We Expected that Reports of Child Engagement and Global Teaching Quality Would be High and Consistent Over Time

Generally, the results supported this hypothesis. Respectively, Fig. 2a and b show results for ratings of child engagement (teacher-reported) and global teaching quality (coach-reported, modified CLASS). Teacher ratings of student engagement were moderate to high and increased over time. For the PATHS Extension activities, consistent reports above a “2” (children were generally engaged) occurred by week 6. Child engagement in the Alphabet Center took more time to develop, with consistent ratings at a “2” or higher not occurring until halfway through the curriculum.

Fig. 2
figure 2

Quality of implementation as rated by teachers and coaches

Full size image

Figure 2b shows coach reports of global ratings of teaching quality. Overall, teaching quality was reported as high. However, this figure shows three points of interest. First, ratings of all three scales showed a slight decrease in quality in January, when schools resumed following the winter holidays, with emotion communication and support and sensitivity and responsiveness rebounding quickly and maintaining high ratings mostly through the end of the year. Second, the richness of talk dropped in January and never fully recovered. Third, emotion communication and support, sensitivity and responsiveness, and Richness of talk all showed another slight decrease at the end of the school year.

Discussion

This pilot study examined implementation features of a multi-component, evidence-based program delivered in a new context. The original trial of the Head Start REDI program occurred in rural and small urban communities in Pennsylvania and included African-American, Latino, and White children (Bierman et al., 2008). We were interested to see how the implementation would vary when REDI was delivered in rural, midwestern communities in and around a Native American reservation. In addition, we were interested to learn whether some aspects of the intervention were more likely to be adapted than others, and whether changes were more likely to occur at different times throughout the school year. Based on prior implementation research, we expected that spontaneous adaptations to the program would occur, and that most adaptations would be pragmatic. Because REDI provides robust training, professional development and technical assistance infrastructure, we anticipated that adherence, dosage, child engagement, and global teacher quality would be generally high for all program components.

Our hypotheses were largely confirmed. As far as dosage was concerned, on average, 79.74% of all program material was delivered. Given that REDI is a complex, multi-component program that teachers were trying for the first time, this suggests that the implementation was largely successful. There was considerable variation in implementation across program components, however. While teachers reported delivering nearly 98% of the weekly PATHS Lessons, rates were lower for the other subcomponents. Only 73% of Sound Games and 63% of Alphabet Center activities were delivered as designed. Teachers reported a high level of satisfaction with the PATHS curriculum, indicating that its structured format made it easy to use, and that children responded well to many of the lesson activities.

Teachers also expressed some reservations about the Sound Games program. In the post-program interviews, three of the four mentioned that Sound Games quickly got too hard, or that the activities were not engaging. It is possible that the level of structure played a role in implementation, as the most structured activities (PATHS and Dialogic Reading) had the highest dosage ratings, while Alphabet Center (the least structured) had the lowest. With the exception of Sound Games, which showed a drop off in the number of sessions completed toward the end of the school year, there was no temporal effect on dosage detected for any REDI program component.

Coach ratings of teaching quality, which included emotion coaching and support, richness of talk, and sensitivity and responsiveness, were consistently high and did not show much variability except for a brief regression that occurred when school resumed after the winter holidays in January. This seasonal drop in teaching quality has implications for technical assistance. Recognizing that the return to school after a long break may be a challenging time for teachers and students alike, technical assistance providers can increase their support and availability during this transition period.

The high levels of dosage and quality were not surprising given the amount of implementation support provided by the REDI program. Two types of training were provided: the in-person trainings and the four on-line REDI teacher modules. On average, teachers completed about 2/3 of the in person training and a little over one-half of the online modules. These rates are lower than observed for teachers in the second REDI classroom trial (Hunter et al., 2022) the online modules were not available during the first REDI study). However, each teacher received 100% of the expected coach visits. This ongoing support and feedback from the coach may have contributed to high implementation fidelity, even though participation in trainings was lower than expected. One unexpected contributing factor was that teachers seemed reluctant to attend training outside their own communities. The initial training was held in one community and the booster in the other, and for the most part teachers only attended the trainings that were held in their community. This was not an issue in the original REDI trials (Bierman et al., 2008; Hunter et al., 2022), but in this new context travel was a barrier.

When teachers were asked about their adaptations to the REDI curriculum, most indicated that they skipped the program activity. This is concerning, as program deletions are consistently associated with diminished EBP effectiveness (Fixsen et al., 2010). Although our small sample size did not allow us to examine program impacts on child outcomes, we might expect to see reduced effect sizes for REDI in a larger trial showing this implementation pattern. The most common explanation teachers gave for these deletions was a lack of time.

In the post-implementation interviews, all teachers indicated that a shortcoming of REDI was its lack of focus on science and math, making it necessary to forego REDI activities in order to meet Head Start’s STEM requirements. Additionally, teachers indicated that they felt the need to substitute holiday-themed activities for the REDI lessons. This provides valuable guidance for future iterations of the REDI curriculum, which could incorporate STEM activities and provide accommodations for holiday-themed activities that incorporate the program targets for each lesson. Additionally, in the future the REDI coach may assist teachers with time management and carefully adapting program components to accommodate holidays. It is possible that some of these time-related constraints would have diminished as teachers became more familiar with REDI. However, given the many requirements and standards imposed on early childhood educators, it seems reasonable to assume that time pressures would be a common challenge.

On the implementation form, teachers indicated whether they used both Dialogic Reading books, but not if they read each book twice. This was a limitation of our data collection system that should be corrected. Teachers indicated in the post-implementation interviews that they would like more flexibility in book selection. Although the books were chosen to align with the themes of the PATHS Lessons, allowing teachers to select their own books may enhance program uptake and sustainability.

Our results are similar to those of other “real world” implementations of EBPs, finding high rates of spontaneous adaptations related to practical constraints such as time and competing demands (Moore et al., 2013). Additionally, our results affirm the concerns of implementation researchers who note that spontaneous deletions of curriculum material can diminish program effectiveness (Fixsen et al., 2010). Despite this, dosage was high overall.

We hypothesized that implementation would be stable across the school year, but found some interesting variability. Teachers’ ratings of dosage and child engagement increased over time, which probably reflects their growing confidence and familiarity with a new curriculum. This was similar to the findings of the original REDI trial (Domitrovich et al., 2010). This improvement also took longer with some program components than others. For example, it took longer for teachers to reach high levels of child engagement for Alphabet Center, the least structured of the program elements, than it did for the PATHS extension activities, which were highly structured. The reduction in child engagement and teaching quality following the winter holidays probably reflects a readjustment period when children and teachers are settling back into routines after an extended break.

Summary and Implications for Practice

A number of factors emerged from this study that can inform the translation of EBPs into new contexts. First, we examined program implementation fidelity over time and by program element. This allowed us to identify component-level variability in implementation fidelity that might otherwise be obscured by overall averages. Teachers indicated that some aspects of REDI were more challenging to implement than others; these components were more likely to be implemented with low fidelity. This suggests that TA providers should be prepared to offer more support around these components, including suggestions for adapting to specific circumstances (i.e., difficulty level). In this study, teachers indicated that the phonological awareness activities were particularly difficult for their students. This suggests that the phonological awareness program may require refinement and taught in smaller steps when teachers encounter this difficulty.

Second, all teachers indicated that the lack of science and math activities was a shortcoming of REDI. Although REDI was designed to specifically promote language and early literacy skills, its lack of attention to STEM shows a failure to align with contemporary early childhood curriculum standards. Prior to future implementations, REDI should be revised to either incorporate evidence-based STEM practices or align better with existing preschool science and math curricula.

Third, teachers in this context exhibited preference for training in their home communities. Low rates of training participation were related to teachers not traveling elsewhere. This was not observed in the original REDI studies, and illustrates the need for clear communication and planning with local partners. Possibly, the rural context influenced participation. Previous studies on professional development for teachers indicate that rural educators are less likely to travel for professional development (Hunt-Barron et al., 2015; Maher & Prescott, 2017). Consequently, rural teachers may prefer to attend training in their own community.

Teachers also reported relatively low rates of completion for the online training modules. It should be noted that these trainings were completed prior to COVID-19, when online professional development formats were less common. Previous research indicates that interactive materials may boost engagement and completion of online professional development activities, either via engaging software (Mixon et al., 2019) or virtual discussion groups (Holmes et al., 2010; McConnell et al., 2012). Possibly, teachers felt less engaged with the online trainings, which limited their motivation to participate in the modules. Given that all teachers mentioned time management challenges with REDI classroom implementation, it is also possible that time constraints limited module completion.

Finally, when asked how they would adapt REDI to better align with local culture and context, all teachers mentioned that the program was already a good cultural fit and there was little they would change in that regard. This provides some justification for the decision to use Chen et al.’s (2012) model for EBP adaptation rather than one that involved adapting prior to implementation. All teachers acknowledged the challenge of implementing a multi-component intervention for the first time, but indicated that they planned to continue using at least some program components.

The sample of both children and teachers differed substantially from that of the original REDI trial in Pennsylvania (Bierman et al., 2008). The original trial included both Latino and African-American children and teachers, and did not include Native Americans. Additionally, the original sample was less rural and from a different region of the U.S. Therefore, it is possible that ethnic, cultural, and geographic factors influenced our findings. Given the low proportion of research conducted with disadvantaged populations (Biglan et al., 2023), this study helps fill a gap regarding EBP translation into new contexts. Rural and Native American communities have been particularly underrepresented in implementation research. In terms of equity and inclusion, the entire goal of the REDI intervention is to reduce disparities in educational success associated with socioeconomic disadvantage. The communities in this study were pleased to have access to this high-quality program and the support for teachers it included, as well as the opportunity to share their input regarding the need for local adaptations. The communities were also actively engaged in all phases of the research process, including the disssemination of results. These are examples of community engagement strategies that can make intervention research more inclusive, approachable, and relevant to stakeholders.

Future research on REDI should focus on linking variations in implementation factors to children’s learning outcomes in Head Start settings. This includes incorporating adaptations recommended by stakeholders and carefully tracking their effects on the school readiness outcomes achieved in the original research.