Supporting providers in delivering evidence-based interventions with high fidelity can not only increase intervention access in the community, but also ensure that these services are producing similar outcomes as research-based trials (Durlak and DuPre 2008; Dusenbury et al. 2003; Chiapa et al. 2015). However, fidelity can be compromised in several ways as providers are trained in a new intervention. For one, without a strong understanding or “buy-in,” providers may “drift” in their implementation of the intervention over time (Elliott and Mihalic 2004) by delivering the intervention in a way that the developer had not intended (e.g., selecting specific components of the intervention to deliver, instead of the entire model, including information outside of the scope of the model; Lau et al. 2017; Marques et al. 2019; Wiltsey-Stirman et al. 2015). Additionally, fatigue and burnout can pose a challenge during dissemination. Providers experiencing a range of risks for burnout including compassion fatigue, emotional exhaustion, higher caseload and younger provider age may be less likely to implement an intervention with fidelity than individuals without these experiences (Aarons, Aarons et al. 2009; Aarons et al. 2011). Finally, meeting with treatment developers for supervision can pose additional stress when trying to manage job time constraints with achieving high model fidelity and adherence (Lyon et al. 2018). Such barriers to high fidelity are problematic because decreased fidelity has been found to be associated with reduced treatment effects in community settings (Hulleman and Cordray 2009). Thus, it is important to understand how provider fidelity can change over time as individuals are trained in a new model of an intervention.

One way that intervention developers can achieve this goal is to identify fidelity requirements that are clear, consistent, and can be replicated over time (Hulleman and Cordray 2009; Fairburn and Cooper 2011; Roben et al. 2017; Schoenwald et al. 2011). An advantage of such clear delineation is that these requirements can enable treatment developers to collect data that can be measured and monitored on a regular (i.e., weekly or monthly) basis. Evaluators can then assess changes in fidelity data over time in a number of ways, including before and after training, or even “real time,” assessing week-to-week or month-to-month changes. Additionally, the ability to measure change in fidelity over time can be used to track changes in fidelity that may arise as providers learn new skills or aspects of an intervention. Ultimately, these data then can be used to support providers in supervision or consultation, either during the training period, or after providers are certified in the model (Costello et al. 2019; Travis and Brestan-Knight 2013). Given strong intervention fidelity is clearly linked with positive outcomes (Dusenbury et al. 2003), it appears critical that intervention developers gain a rich understanding of these data. However, despite the utility of such data, few evidence-based interventions, including early interventions, adequately measure intervention fidelity (Durlak and DuPre 2008) and even fewer track these data longitudinally (Chiapa et al. 2015). Therefore, as a first step in better understanding evidence-based intervention fidelity, it may be warranted to look at how fidelity can change over time, especially as providers learn new and different skills.

Attachment and Biobehavioral Catch-Up

Attachment and Biobehavioral Catch-up (ABC) is a nationally- and internationally-disseminated evidence-based early parenting intervention aimed at improving attachment quality and self-regulation, of which there is an infant (ages 6–24 months) and toddler (ages 24–48 months) model. Currently, there are ABC providers in 21 states in the United States and in eight countries internationally. ABC is unique in that it is one of the few early evidence-based interventions to measure provider fidelity across a large and diverse group of providers, as well as track these fidelity data over the course of one year of supervision. Not only does ABC measure and monitor fidelity, but staff do so in a rigorous way, using microanalytic behavioral coding from video recorded intervention sessions.

The ABC intervention consists of 10 sessions delivered by parent coaches who meet with parents and their children in their homes. The core component of the intervention is considered the “in-the-moment” comments, or real-time feedback, delivered by coaches as parents engage in ABC-targeted behaviors (e.g., nurturance, following the lead with delight, avoiding intrusive and frightening behaviors) with their children (Dozier and Bernard 2019). The frequency and quality of parent coach in-the-moment comments is associated with the degree of change in targeted parent behaviors (Caron et al. 2016a). In addition to making in-the-moment comments, coaches also deliver manualized session content.

The ABC toddler model (ABC-T) builds from the ABC infant model (ABC-I) and initially was developed to address emerging behavioral and emotional dysregulation in young children, particularly children who experienced early adversity. Therefore, a key target of ABC-T is supporting parents in addressing child dysregulation by serving as a strong co-regulator. Parents are supported in this role by responding to their children using the “calming” intervention target, and are taught several strategies to stay both physically and psychologically available during times of child frustration and over-excitement. These strategies include remaining calm during child dysregulation, avoiding power struggles (e.g., lecturing the child or saying, “no” to behaviors that do not need to be limited), and staying close and providing support to their child.

In randomized clinical trials, ABC-I, as compared to a control intervention, was efficacious in improving parental sensitivity and decreasing parental intrusiveness (Berlin et al. 2014; Caron et al. 2016a; Caron et al. 2016b) as well as decreasing child abuse potential and parenting stress (Sprang 2009). Additionally, children whose parents received ABC, compared to children receiving a control intervention, were classified as having secure attachments more frequently (Bernard et al. 2012), and had a more normative diurnal pattern of cortisol production, with results lasting approximately three years after the intervention (Bernard et al. 2015a; Bernard et al. 2015b). ABC has also demonstrated positive outcomes for children across executive functioning skills (Lewis-Morrarty et al. 2012) and emotion regulation (Lind, Bernard et al. 2014).

The ABC-T model has also been efficacious in improving both child and parent outcomes. In two studies conducted using randomized clinical trials with children in foster care, ABC-T was found to enhance executive functioning capabilities, reduce attention problems, (Lind et al. 2017) and increase receptive vocabulary abilities (Raby et al. 2018). Parents who received ABC-T also demonstrated increased sensitivity to their children from pre- to post-intervention (Raby et al. 2018).

Across both models, ABC fidelity is clearly defined and measured using the in-the-moment coding system. The in-the-moment coding system consists of two parts: coding for both parent behaviors as well as parent coach responses toward those behaviors. First, parent behaviors that align with intervention targets (e.g., nurturance, following the lead, delight, avoiding frightening behavior, calming) are coded. Second, parent coach responses or lack of response toward those same parent behaviors are coded. As part of the ABC training process, both the fidelity-focused supervisor and the parent coach code a randomly-selected 5-min clip from the coach’s hour-long video recorded ABC session. The in-the-moment coding system then produces fidelity statistics that provide information on commenting rate, quality, and specificity.

Training as a parent coach in the ABC intervention is a two-step process. Coaches are first trained in the infant model (for parents and infants aged 6–24 months). During training, coaches receive weekly fidelity-focused supervision for one year. In these meetings the fidelity-focused supervisor reviews the coded clip and the weekly fidelity statistics are tracked over time. In order to be certified in the ABC-I model, coaches’ coded data from their final ten sessions must meet the specific fidelity requirements outlined in the model (e.g., comment quantity and quality; Dozier and Bernard 2019). We have found that, on average, coaches’ fidelity during ABC-I training, measured by their rate of commenting, improves from the beginning to the end of the training year; however, with individual variation in commenting. Specifically, some coaches start with and maintain a high rate of commenting across the entire year, while other coaches provide limited commenting, gradually increasing their rate over the course of the year (Costello et al. 2019).Upon certification in the infant model, coach fidelity data and effectiveness in delivering ABC (e.g., adherence to the manualized content; case conceptualization skills) are reviewed, and individuals who are determined to be highly skilled in these areas can elect to be trained in the toddler model (for parents and children aged 24–48 months). Coaches are typically considered to be highly skilled in the ABC-I in-the-moment commenting when they begin the ABC-T training process.

The same commenting strategy that is used in ABC-I is used in ABC-T to support parents in responding in nurturing and sensitive ways with their children, but coaches also comment on parents’ calming behaviors with their children during moments of dysregulation. Additionally, in ABC-T there is an emphasis on the use of “advanced comments” (i.e., comments which provide constructive feedback to change parent behavior). Especially during periods of child dysregulation, coaches are encouraged to use these advanced comments to support parents in engaging in calming strategies. Commenting during child dysregulation, using both positive and “advanced” comments is thought to be more complex than commenting during nurturance, following the lead with delight, and avoiding frightening behavior. Therefore, coaches are required to become certified in ABC-I before moving onto training in ABC-T. Similar to the ABC infant training process, coaches receive one year of weekly fidelity-focused supervision, and their fidelity data, as measured by their in-the-moment commenting, are tracked, with data from their final ten sessions used to determine certification. Thus, the fidelity monitoring over both training periods (infant and toddler) offers a unique opportunity to track changes in coach skill over time, especially as providers move to learning a new and often complex model.

Current Study

It is well established that high fidelity to evidence-based interventions is critical in producing positive outcomes for children and families in the community (Dusenbury et al. 2003). However, due to a number of provider factors, including lack of “buy-in,” stress and burnout, and difficulty managing training requirements with other job responsibilities, provider fidelity to an intervention may be compromised (Aarons et al. 2009; Elliott and Mihalic 2004; Lyon et al. 2018). Unfortunately, few evidence-based interventions measure intervention fidelity, especially longitudinally (Chiapa et al. 2015; Durlak and DuPre 2008). Therefore, an important first step for treatment developers is to understand how provider fidelity can change over time, with the ultimate goal of using these data to support individuals in the community as they work with families to achieve outcomes similar to research-based trials. Attachment and Biobehavioral Catch-up (ABC) is one evidence-based early intervention that has clearly defined fidelity requirements that are measured and monitored over time, using microanalytic behavioral coding from video recorded sessions. This allows program evaluators the opportunity to understand how provider fidelity can change (e.g., either “drift” or increase), particularly as providers move from one model (e.g., infant) to learning a new model (e.g., toddler).

To this point, the current study examined changes in parent coach fidelity as individuals became certified in the ABC infant model and learned the ABC toddler model (ABC-T). Coaches in this sample represented seven states and fourteen organizations, allowing us to understand changes in fidelity over a diverse group of community-based providers. Fidelity was measured by several indicators of the in-the-moment commenting system. These included frequency of comments, percentage of on-target comments, and number of information components used in comments. It was predicted that there would be an initial drop in commenting fidelity at the start of ABC-T, as coaches may have struggled with making comments related to a new intervention target (i.e., calming children during child dysregulation). However, given that coaches were also receiving intensive supervision for one year post-training, it was also predicted that coaches’ commenting would return to or surpass fidelity requirements once they had the opportunity to master the new intervention target.

Method

Participants

Data for this project were collected from seven different cohorts of parent coaches (representing 34 individuals and 14 organizations total) who received training in the ABC-T model between the months of June 2016 and February 2018. Two of the cohorts came from a non-profit organization, trained over the course of two years, in which coaches delivered ABC full-time. One cohort was made up of coaches from multiple agencies who were participating in a statewide dissemination of ABC, and delivered ABC in addition to other job responsibilities. The four remaining cohorts were made up of small groups of coaches (i.e., 1–2 individuals per cohort) who also balanced ABC with additional job responsibilities.

The majority (93%) of coaches identified as female. 43.8% percent of coaches identified as White, 40.6% of coaches identified as Black/African American, 3.1% of coaches identified as Asian, 3.1% of coaches identified as Haitian, and 9.4% of coaches identified as Other. The mean age of coaches was 33 years old (SD = 8.01, range 22–52 years old). The majority (46.7%) of individuals completed a Masters-level education, with education level ranging from completion of high school through doctoral-level education. Coaches had an average of 9.29 years of experience (SD = 7.93) working in the psychology/social work/mental health field. Table 1 summarizes the demographic information for the coaches in this study.

Table 1 Demographic characteristics of the ABC parent coaches
Full size table

Measures

Fidelity Data

Fidelity data, as measured by commenting rate, number of components, and percentage of on-target comments, were collected using the in-the-moment coding system. The in-the-moment coding system both identifies ABC-relevant parent behaviors (e.g., following the lead, nurturance, frightening behavior, and calming) and parent coach comments (or lack of comments) related to the ABC-relevant behaviors (Caron et al. 2016a). This system is used to monitor fidelity to ABC, which is measured by the quantity and quality of parent coach comments (Bernard et al. 2013; Dozier and Bernard 2019). After each session is coded, the fidelity-focused supervisor works with the parent coach to select a 5-min clip. In the ABC-Infant model, all clips are randomly selected using a random number generator (www.random.org). This is meant to encourage coaches to meet ABC fidelity criteria at any point during their hour-long session. Given coaches have already been certified in ABC-I before they get trained in ABC-T, and thus have demonstrated the ability to meet session fidelity requirements predictably and consistently, the clip selection in ABC-T differs from ABC-I. In the ABC-T program, if child dysregulation occurs in the session, then the parent coach informs the supervisor of times during the session when the dysregulation occurs. The supervisor then reviews the suggested coding times and makes the final decision about which 5-min clip to code. Typically, if there is more than one moment of child dysregulation, the supervisor will choose the most challenging 5-min clip for coding, and this ensures the parent coach is receiving helpful coaching and feedback to use in the next session. Thus, the goal of the fidelity-focused supervision in ABC-T is to focus specifically on supporting coaches in learning and practicing making comments related to child dysregulation and associated parental calming and not calming behaviors.

After the 5-min clip is coded, a summary of fidelity statistics is calculated from the coding sheet. These summary statistics are then reviewed during fidelity-focused supervision and used to determine certification to ABC. In order for coaches to be certified in ABC, they need to attend a two-day, in-person training (for ABC-I) or a half-day training (for ABC-T), complete a minimum of three cases and attend fidelity-focused supervision for one year. Most importantly, certification for both programs requires coaches to meet several commenting-specific criteria in their coded clips, including an average of one comment per minute, at least 80% on-target comments, and an average of one component per comment.

Commenting Rate

Commenting rate is defined as the number of on-target comments per minute that coaches make in a 5-min clip, and calculated by dividing the total number of parental behaviors by the total number of on-target coach comments.

Number of Components

Commenting components are defined as descriptive statements that coaches use when providing feedback to parents. Three different components can be included in a comment: parents’ responses to their children are described (e.g., “He handed you the car and you took it.”), the intervention target is labeled (e.g., “Great job following the lead.”), and a research-supported outcome of the ABC-targeted behavior is identified (“This shows him he has an effect on the world.”). The number of components is calculated by adding up each of the above pieces of information, with a range of 0 (no components) to 3 (all three components).

Percent of on-Target Comments

On-target comments occur when coaches’ comments are solely focused on the ABC-targeted behaviors. Comments that deviate from ABC behaviors or identify something outside of the scope of ABC would be coded as off-target (e.g., “Nice job teaching him to share.”), as would comments that mis-identify behaviors with the wrong target (e.g., “Great example of following the lead.” when the parent calmed the child). The percentage of on-target comments is calculated by dividing the total number of comments by the total number of on-target comments in the clip.

Procedure

Parent coaches in each cohort attended a half-day in-person or video conference training led by an ABC trainer with expertise in supervising both the ABC infant and toddler models. The training for each cohort followed the same protocol: coaches were introduced to the new calming behavior target, given a rationale as to why addressing calming during child dysregulation is important, and were provided case examples illustrating both commenting on calming behaviors and addressing these topics within the manualized content. Additionally, coaches learned how to code calming and not calming behaviors, as well as differentiate the calming behavior target from the other, related infant model behavior targets (e.g., nurturance). After completing the initial training, coaches received weekly individual fidelity-focused and small-group clinical supervision, which occurred over video conferencing, for approximately one year.

In their delivery of ABC-T, coaches video recorded each of their sessions. The coaches then uploaded their session video to a HIPAA-compliant server for review. Prior to meeting for supervision, supervisors and coaches independently coded a 5-min video clip from the longer session video that included child dysregulation and entered these data onto a fidelity spreadsheet. During supervision, the fidelity-focused supervisor reviewed feedback specifically directed to the coach’s commenting around the calming behavior target as well as on the coach’s coding of the clip, with particular emphasis on how the coach identified calming and not calming behaviors. Coding feedback was determined through visual inspection and comparison between the supervisor’s and coach’s coding sheet. Coaches received a copy of their supervisor’s coding sheet before every meeting. Coaches were not required to meet an interrater reliability cutoff with their supervisor. Only the fidelity data from the supervisor, and not the coach, from these weekly sessions were stored and tracked across the year.

Supervision sessions were led by eight research staff who were determined to be expert coders in the fidelity coding system. All eight supervisors identified as female and were either undergraduate research assistants or research staff who had been selected by a laboratory coordinator to train in the ITM coding system. Supervisors first attended an intensive, 3-month long training in the ABC-Infant model coding system, occurring between the spring of 2012 through the fall of 2016. This consisted of individuals completing weekly coding practice exercises and meeting with an expert ITM coder to discuss their coding. Reliability was determined by achieving 70% or greater agreement with a “master” coder on several parent behavior and coach commenting codes across ten “test” videos, administered at the end of the coding training. After becoming reliable in the ABC-I coding system, each fidelity-focused supervisor then supervised clinicians in the infant model for at least 2 to 3 years before being trained in the ABC-T coding system. During this time, supervisors met for weekly consultation meetings, along with their fellow supervisors, with an expert ITM coder to review their coding and supervision questions to prevent drift.

The supervisors then attended an additional 4-week training to become reliable in ABC-Toddler coding. These training sessions occurred between September 2016–August 2017. This training consisted of supervisors learning how to code calming and not calming behaviors and coach comments using video review, practice coding, and a training curriculum. Reliability in this system was determined by the supervisors’ ability to correctly identify and code calming behaviors and comments (which was reviewed by a co-developer of the ABC toddler model), as well as how to provide feedback to parent coaches. Fidelity-focused supervisors all followed the same general structure for supervision sessions, whereby they reviewed their coding sheet, as compared to the coach’s coding sheet and discussed the coach’s commenting relative to the ABC fidelity criteria. Supervisors were trained to engage in skill building exercises, such as engaging in role-plays and “live” commenting of clips with their coaches during the weekly meetings. Supervisors engaged in the same consultation process as they did for the ABC-I model, meeting weekly with an expert ITM coder and their fellow supervisors, to review coding and supervision.

Data included in this study were collected from three different time points, reflecting three discrete phases of the coaches’ training: time 1 (T1) indicated coaches’ fidelity data (i.e., commenting rate, percent on-target, number of components) at ABC-I certification, which were taken from an average of their final ten sessions; time 2 (T2) was the data point in ABC-T at which coaches first had an opportunity to comment on dysregulation; finally, time 3 (T3) indicated fidelity data from coaches’ final ABC-T supervision session that had dysregulation, typically occurring at the end of the training year. Given child dysregulation did not occur in all ABC-T sessions, T2 and T3 were chosen to reflect coaches’ ability to address calming and not calming behaviors. All data analyzed in this study were originally collected for program evaluation purposes, and thus considered exempt by [omitted for review] Institutional Review Board, as they were archived, deidentified, and used for program evaluation.

Data Analytic Plan

All analyses were completed using SPSS 25.0 (IBM Corp. Released 2017. IBM SPSS Statistics for Windows, Version 25.0. Armonk, NY: IBM Corp). A repeated measures analyses of variance was used to assess change in commenting rate, missed opportunities, percent on-target, and number of components and three time points: T1: certification in the ABC-I model, T2: during the first ABC -T session when the coach commented on child dysregulation, and T3: during the last ABC-T session when the coach commented on child dysregulation.

Results

Preliminary Analyses

Table 2 demonstrates fidelity data averaged across all cohorts of coaches at each time point. At infant certification, on average, all coaches were well above the fidelity requirements of an average of 1.0 comments per minute, 80% or greater on-target comments, and an average of 1.0 components per comment. Notably, on average, coaches across all cohorts maintained a commenting rate above 1.0 comments per minute through their toddler training (even as other commenting fidelity fluctuated over time).

Table 2 Average fidelity data from the training cohorts across each time point
Full size table

Primary Analyses

Commenting Rate

There was a main effect of time on coach commenting rate, F (2, 66) = 6.81, p = .002. Post hoc tests using a Bonferroni correction indicated a significant decrease in commenting rate from T1 (M = 1.61) to T2 (M = 1.23), p = .03 as well as a significant increase in commenting rate from T2 to T3 (M = 1.81), p = .01. Figure 1 indicates these changes over time. There was no significant difference between T1 and T3, p = .55.

Fig. 1
figure 1

Average fidelity data across each time point relative to certification requirement. Note. Dashed line in each graph represents the ABC certification requirement.

Full size image

Percent on-Target Comments

Mauchly’s test indicated that the assumption of sphericity had been violated (ε = 0.55). Using a Greenhouse-Geisser correction, there was a main effect of time on percentage of on-target comments, F (1.11, 36.64) = 9.89, p = .003. Post hoc tests using a Bonferroni correction indicated a significant decrease in percentage of on-target comments from T1 (M = 94.7) to T2 (M = 75.6), p = .011, as well as a significant increase in percentage of on-target comments from T2 to T3 (M = 96.0), p = .008. Figure 1 indicates these changes over time. There was no significant difference between T1 and T3, p = 1.0.

Number of Components

There was a main effect of time on number of components, F (2, 64) = 8.54, p = .001. Post hoc tests using a Bonferroni correction indicated a significant decrease in number of components from T1 (M = 1.45) to T2 (M = 1.11), p = .04, as well as a significant decrease in number of components from T1 to T3 (M = .991), p = .000. Figure 1 indicates these changes over time. There was no significant difference between T2 and T3, p = .98.

Discussion

High fidelity to evidence-based interventions has consistently been found to be a key component of achieving community outcomes that replicate research-based trials (Durlak and DuPre 2008). However, few interventions measure and monitor intervention fidelity, and even fewer do so over time (Chiapa et al. 2015; Durlak and DuPre 2008). Given the importance of maintaining high intervention fidelity during community dissemination, the current study examined changes in provider fidelity data while training in the Attachment and Biobehavioral Catch-up (ABC) intervention.

In the current study, change in fidelity data, measured by several indicators of the in-the-moment commenting system (e.g., commenting rate, number of components, and percentage of on-target comments) from ABC “parent coaches” was assessed as coaches moved from the ABC infant model certification to the toddler model certification. Given that coaches were required to learn a more complex, specialized skill in ABC-T as compared to ABC-I, it was expected that there would be an initial drop in commenting frequency, quality, and descriptiveness at the start of the toddler model but that these levels would return to or surpasses fidelity requirements over the course of the training year.

Consistent with the study hypotheses, a main effect of time on coach fidelity data was found. Both commenting rate and percentage of on-target comments decreased between ABC-I certification and early in the ABC-T training process. However, coaches were able to increase both their commenting frequency and on-target comments by the end of the toddler training year. Interestingly, it was found that coaches struggled over time with comment descriptiveness, as demonstrated by a decrease in average information components over time.

Given the demand on coaches learning new information, the initial decrease in both commenting rate and percentage of on-target comments at the start of the ABC-T training could be explained by a number of factors. One, coaches may not have felt as confident in delivering the new model as they did at the end of their infant intervention training. Within ABC, commenting during the toddler model is thought to be more complex than in the infant model, as toddler dysregulation (and supporting parents in serving as strong co-regulators) can often be overwhelming and challenging, and require a thoughtful and skilled approach. For example, in ABC-T, coaches are not only responsible for providing a high level of positive feedback, but also rely heavily on providing “advanced comments” (i.e., comments which provide constructive feedback to help parents to engage in an ABC-targeted behavior), to support parents in addressing child dysregulation using the calming skills. Providing these advanced comments is typically challenging for coaches, especially in the moment, and requires support from their fidelity-focused supervisor. Thus, coaches may have initially struggled with feeling confident in making these complex comments.

Additionally, it could have been initially challenging for coaches to identify child dysregulation. In ABC-T, child dysregulation is defined as frustration, anger, and associated behavioral difficulties (e.g., hitting, biting). These behaviors are contrasted with child distress (e.g., sad, sick, hurt, scared) that require a parent to respond in nurturing ways. These often-nuanced differences can become quite challenging for coaches to distinguish in an often fast-paced session. Therefore, coaches may have had difficulty identifying what to say during such moments as they arose in session.

Finally, about half of the coaches in the sample were previously trained in other interventions, including some coaches trained in behavioral parent training and methods that may have focused less on promoting co-regulation than what is addressed in ABC-T. Therefore, coaches may have had difficulty with “buy-in” to ABC’s goals, potentially initially feeling conflicted about the effectiveness of the ABC strategies. Similarly, coaches may have struggled with interference of knowledge from other interventions while delivering ABC.

However, despite the initial drop, coach commenting rate and percentage of on-target comments increased over time and met fidelity requirements at the ABC-T certification. One hallmark of the ABC training model is that coach fidelity is monitored over time, and coaches meet for weekly fidelity-focused supervision. Thus, coaches’ commenting rate and percentage of on-target comments could have improved due to this ongoing support. Although the specific content of these meetings was not tracked for the current study, in general, supervisors dedicated these meetings to discussing child dysregulation cues (including differentiating dysregulation versus distress cues) and parent calming or not calming behaviors. Coaches also had the opportunity to practice making comments they could use during sessions. Finally, if coaches were struggling with buy-in or interference from interventions, this could be addressed during these weekly meetings. Therefore, these meetings may have helped coach confidence, skill, and buy-in to improve over time, which affected a return to infant fidelity rates and quality.

Interestingly, the descriptiveness of comments, as measured by the number of information components, decreased from the infant model certification through the ABC-T certification. This decrease over time was initially surprising because comment descriptiveness was expected to follow a similar pattern as the other commenting fidelity data (e.g., frequency and percent on-target). However, given comments regarding child dysregulation could be challenging to make, especially in a busy and potentially chaotic session, coaches may have had trouble balancing making frequent comments with including adequate detail in these comments. Additionally, given video clips that included supporting parents through dysregulation were prioritized for coding and supervision, these clips may have included more opportunities for coaches to deliver the “advanced” comments than comments addressing positive parent behaviors. Because these advanced comments are typically framed as suggestions of things parents could try differently, coaches are encouraged not to add a high level of detail to their feedback. The emphasis on making these kinds of suggesting comments, which is typically higher in ABC-T than it is in the ABC-I, may have influenced comment descriptiveness across infant to toddler model delivery. However, despite the decrease over time, comment descriptiveness still either reached or was very close to certification requirements at each time point. Thus, coaches were still able to maintain a high level of detail in their commenting, even with the added responsibility of making the advanced suggesting comments.

Limitations

The current study is characterized by several limitations. First, the study is limited by a small sample size, with data from 34 parent coaches included in analyses. Additionally, the coaches came from several different training cohorts over a 2-year period. As a result, there may have been trainer or supervisor effects that could have influenced changes in coach fidelity. For example, there could have been varied content in what was covered in fidelity-focused supervision, and supervisors could have used practice strategies that were specific to individual coach needs, instead of keeping strategies constant across all coaches. Therefore, it may be warranted for future studies to assess possible trainer and/or supervisor effects across coaches (Caron 2017).

Second, this study did not explore other variables that may have affected coach commenting fidelity. For example, because coaches came from different organizations and agencies, there could have been organizational factors (e.g., varying degrees of support from the local agency) for training in and delivering an evidence-based intervention (e.g., Aarons et al. 2011; Fixsen et al. 2005). Similarly, there may have been notable differences in patterns of fidelity for coaches who carried a large ABC-T caseload or delivered ABC-T full time compared to coaches delivered the intervention part time while balancing additional work responsibilities. Additionally, this study did not explore coaches’ own traits and experiences working with families that may have impacted how they engaged with their ABC families including delivering the in-the-moment commenting. Future studies could benefit from investigating individual patterns and predictors of fidelity, particularly using more sophisticated statistical analyses (e.g., growth mixture modeling) than what were used in the current study. Ultimately, it would be helpful to develop a nuanced understanding of coach fidelity to inform training and supervision within ABC.

Finally, given the current study used data from a community-based implementation, the design did not include a control group. It would be interesting to assess potential differences in fidelity between parent coaches who receive supervision and those who practice without supervision, over a training year. Despite these limitations, the current study was still successful in demonstrating change in commenting fidelity over time within a community-based implementation from a small and diverse sample of ABC parent coaches.

Conclusion

The findings from the current study point to the potential of using extended fidelity monitoring and supervision, post-training, for community-based providers who are learning a new evidence-based skill or intervention model. Notably, within the current study, although provider commenting fidelity typically remained at certification levels through the training year, there was a decrease from ABC-I certification to beginning the ABC-T training, with an eventual increase in most aspects of commenting at the end of the ABC-T training year. Therefore, although not specifically assessed in this study, the ongoing support coaches received through ABC supervision and fidelity-focused feedback may have helped them to feel confident and competent in delivering the ABC-T model. In general, researchers have noted that providing clearly delineated fidelity requirements that are predictable and consistent is an important step as individuals are learning complex evidence-based interventions (e.g., Travis and Brestan-Knight 2013). Therefore, ongoing supervision, post-training, could allow trainees delivering an evidence-based intervention the opportunity to problem-solve potential barriers to delivering an intervention with fidelity, including the chance to practice new or challenging techniques, with the goal of honing provider skill and confidence during the learning of a complex intervention.

Supporting fidelity monitoring and supervision, both during and after training, has been found to be beneficial for several reasons. For one, researchers have demonstrated decreased staff turnover when providers are given fidelity-focused feedback (e.g., Aarons et al. 2009). Such support may also help to increase or enhance provider buy-in and commitment to an intervention, especially if intervention fidelity requirements are clearly defined and communicated (Schoenwald et al. 2011). Finally, fidelity monitoring and ongoing supervision could be considered an investment in promoting strong outcomes in families receiving community-based services, given the well documented impact of high fidelity on successful intervention dissemination (Durlak and DuPre 2008; Dusenbury et al. 2003).

Providing ongoing fidelity and supervision does not have to be as rigorous (i.e., via behavioral coding of videorecorded sessions) and frequent (i.e., weekly) as what was described in this study. Indeed, for many agencies, it may not be feasible, across cost, time allotment, and staffing needs to provide this kind of support or partner with treatment developers for an extended period of time. However, the results from this study lend support to incorporating fidelity monitoring and feedback that can be implemented on a regular basis. For example, strategies such as completing session checklists, reporting on the use of treatment components, or participation in bi-weekly or monthly telephone consultation calls are common methods of monitoring fidelity in community settings (Nelson et al. 2012; Schoenwald et al. 2011). These methods can be beneficial to community-based agencies and providers because they often require fewer resources or demands than weekly video review and coding. Perhaps most importantly, they still meet the goal of setting up an opportunity for regular communication and support between providers and treatment developer staff. Ultimately, investing in fidelity monitoring and feedback can provide positive outcomes across the agency, provider, and family level. Thus, it is warranted for intervention developers to consider incorporating such systems into their training process.