The findings from this study revealed that the SARE Trial led by Senn et al. (2015, 2017) showed high intervention fidelity overall, particularly in intervention design and interventionist training. Multiple measures used in the RCT converged to support the latter conclusion and scores on the Gearing et al. (2011) CIFG buttress this conclusion. In designing the intervention, careful attention was given to creating a theoretically driven, empirically based program supported by highly scripted intervention manuals and an implementation guide. Detailed and extensive training protocols were developed and implemented. New facilitators underwent 9 days of intensive training including dress rehearsals of program sessions critiqued by the developer in the presence of a mock audience. As mentioned previously, providing interventionists with the opportunity for behavioral rehearsal has been identified as a best practice in intervention science for promoting successful training and maximizing intervention fidelity (Beidas et al., 2014, Cross et al., 2011). Intervention delivery was intentionally monitored, and fidelity measured. Self-reported adherence checks were employed in the form of Fidelity Issues Forms (FIFs). Intervention sessions were audio-recorded, and a subsample of these recordings scored for intervention fidelity by two independent raters. Facilitators’ willingness to comply with intervention fidelity measures was moderate to high as indicated by the number of FIFs completed (67%) and program sessions recorded (99%).
Facilitator training was supplemented with session by session supervision with the developer. These meetings resulted in prompts to carry out protocol adherence or correct protocol deviations. They also provided the opportunity for facilitators to create and participate in a community of practice of sorts that prompted peer to peer support and sharing of lessons learned, all of which helped to enhance their competencies in the facilitation process. There were other benefits to supplementing self-report adherence checks with live supervisory meetings including the discovery that one activity was carried out incorrectly by a facilitator at one site throughout the first year of the trial. Perfection in intervention delivery is an elusive outcome. If a facilitator does not see their mistake, they will not record or discuss it. Multiple checks on intervention fidelity are, therefore, recommended. The highly scripted nature of this intervention, the inclusion of troubleshooting tips, and the extensive training and ongoing supervision likely contributed to the high intervention fidelity scores obtained overall.
It is worth noting that in the SARE Trial, all four components of intervention fidelity recommended by Gearing et al. (2011) were assessed, a rarity in the literature. The program was subjected to evaluation from the beginning of its development, demonstrating that an early commitment to intervention fidelity was made by the team.
Despite having multiple outcome measures, participants were not asked to complete assessments of their understanding or recollection of what they learned in a way that was specifically designed to assess this learning. They were asked, at each follow-up time point, what if any strategies from EAAA they subsequently put into practice to resist sexual coercion (the results from this research will be published in a forthcoming paper), but these were intended as outcome measures rather than measures of implementation receipt. We recognize that including measures of implementation receipt would have been beneficial from the standpoint of assessing intervention fidelity; however, any additional assessment steps would have increased the already high research burden for participants. Balancing the benefits of implementation measurement with the risk of participant attrition due to burden are key features of consideration when designing longitudinal intervention research. Our ability to retain research participants over time (95% retention rate across 12 months; Senn et al., 2015) suggests this was a good compromise.
Limitations of the Study
The main limitation of this study is that the data and other documents supporting the extent to which intervention fidelity was maximized were compiled and reviewed retrospectively by the first two authors on this paper, both of whom were involved in the original RCT, which may have biased their ratings on the CIFG. The CIFG offers some objectivity to assessment of intervention fidelity, as shown in the range of scores among raters. Inclusion of the third author, who was not involved in the original trial, helped to mitigate but not eliminate potential biases of interpretation of all results. Fortunately, the results concerning attendance collected by several facilitators and the independently rated IFCs are consistent with the scores on the CIFG.
Further, additional steps could have been taken to increase intervention fidelity in the EAAA trial. As is commonly reported in other implementation research (Gearing et al., 2011), there were no objective measures to assess the impact of training on the facilitators’ knowledge or skill level, although the program developer informally monitored the impact of training on facilitators’ competence during the initial training and throughout the trial. Having recordings of the program sessions, a sample of which could be rated on an Intervention Fidelity Checklist was a reasonable replacement. In addition, because facilitators worked in pairs, it is possible that learning deficits in one or the other were compensated for by her colleague.
It would have been helpful to include measures of facilitators’ engagement with and enthusiasm for the program. Some indication of facilitators’ compliance can be gleaned from the number of audio recordings collected and FIFs received, although the number of FIFs completed was lower than expected. Even after we explained to facilitators that the FIFs were not intended to be punitive as variations in program delivery are inevitable, expected, and at times unavoidable (e.g., resulting from a fire alarm or power disruption), our reassurances did little to improve facilitators’ compliance. Further, although facilitators were directed to maintain journal notes after each program session they facilitated and to bring these notes to their supervisory meetings, in an effort to protect their privacy, these journals were never collected. As a result, the extent to which facilitators complied with this direction cannot be determined. Based on our observations, facilitators appeared to be compliant with and attentive to the procedures around recording of program sessions and the transfer of these recordings to the Co-ordinating Centre. They also seemed eager for and receptive to feedback on their delivery of EAAA from the developer, although there were occasions when facilitators expressed frustration with the level of critique they received during dress rehearsals. In fact, some veteran facilitators (i.e., those returning for a second year of program delivery) expressed reluctance and dismay at having to continue to perform critiqued dress rehearsals in front of the program developer or their site coordinator/investigator, arguing that because of their level of expertise these requirements were unnecessary and overly onerous. Their overconfidence was always made apparent during these rehearsals when at least one, if not several, activities these facilitators were certain they had mastered completely were delivered incorrectly after the 6-month hiatus between the end of program delivery in the previous academic year and the fall semester of the subsequent year. Perceptions aside, it is likely that facilitators varied in their willingness to comply with procedures put in place to maximize fidelity, but without an objective indicator of compliance we cannot say for certain.
The findings stemming from assessment of intervention fidelity can aid in more nuanced interpretation of the results of outcome or efficacy research (Greenberg & Barnow, 2014; Rychetnik et al., 2002) particularly when implementation is conducted across multiple sites, as was the case with the EAAA trial (Greenberg & Barnow, 2014; Ofek, 2016). Applying the components of the Gearing et al. (2011) intervention fidelity framework to RCTs such as the one discussed here can help enhance the quality of the design and provide insight into the interpretation of the outcomes.
Future research could examine implementation in a more naturally occurring setting than an RCT where there may be greater variance in levels of intervention fidelity, allowing for the identification of a threshold of fidelity monitoring that yields positive outcomes in delivery of the intervention. This is worthwhile to explore given the resource intense nature of fidelity monitoring. Of note, innovative methods to capture and rate fidelity are beginning to be studied. For example, Caperton et al. (2018) compared the fidelity monitoring of partial sessions (of a psychosocial intervention) with full sessions against interrater agreement and found that approximately a third of a session had sufficient agreement to approach interrater levels. The results from such implementation research measurements can inform the development of feasible and cost-effective ways to modify intervention delivery and help ensure scale up. The findings from the present study show how methods that are recommended to enhance intervention fidelity optimized the efficacy of an evidence-based sexual assault resistance education program.