Adherence and Delivery: Implementation Quality and Program Outcomes for the Seventh-Grade keepin’ it REAL Program
- 574 Downloads
Poor implementation quality (IQ) is known to reduce program effects making it important to consider IQ for evaluation and dissemination of prevention programs. However, less is known about the ways specific implementation variables relate to outcomes. In this study, two versions of keepin’ it REAL, a seventh-grade drug prevention intervention, were implemented in 78 classrooms in 25 schools in rural districts in Pennsylvania and Ohio. IQ was measured through observational coding of 276 videos. IQ variables included adherence to the curriculum, teacher engagement (attentiveness, enthusiasm, seriousness, clarity, positivity), student engagement (attention, participation), and a global rating of teacher delivery quality. Factor analysis showed that teacher engagement, student engagement, and delivery quality formed one factor, which was labeled delivery. A second factor was adherence to the curriculum. Self-report student surveys measured substance use, norms (beliefs about prevalence and acceptability of use), and efficacy (beliefs about one’s ability to refuse substance offers) at two waves (pretest, immediate posttest). Mixed model regression analysis which accounted for missing data and controlled for pretest levels examined implementation quality’s effects on individual level outcomes, statistically controlling for cluster level effects. Results show that when implemented well, students show positive outcomes compared to students receiving a poorly implemented program. Delivery significantly influenced substance use and norms, but not efficacy. Adherence marginally significantly predicted use and significantly predicted norms, but not efficacy. Findings underscore the importance of comprehensively measuring and accounting for IQ, particularly delivery, when evaluating prevention interventions.
KeywordsImplementation quality Program evaluation Substance use Adolescents
This publication was supported by Grant Number R01DA021670 from the National Institute on Drug Abuse to The Pennsylvania State University (Michael Hecht, Principal Investigator). Its contents are solely the responsibility of the authors and do not necessarily represent the official views of the National Institutes of Health. (NIH Manuscript # NIHMS272843). Portions of this paper were presented at the 2013 annual meeting of the Society of Prevention Research.
- Arthur, M. W., Hawkins, J. D., Pollard, J. A., Catalano, R. F., & Baglioni, A. J. (2002). Measuring risk and protective factors for use, delinquency, and other adolescent problem behaviors the communities that care youth survey. Evaluation Review, 26, 575–601. doi:10.1177/0193841X0202600601.PubMedGoogle Scholar
- Choi, H. J., Krieger, J. L., & Hecht, M. L. (2013). Reconceptualizing efficacy in substance use prevention research: refusal response efficacy and drug resistance self-efficacy in adolescent substance use. Health Communication, 28, 40–52. doi:10.1080/10410236.2012.720245.PubMedCentralCrossRefPubMedGoogle Scholar
- Colby, M., Hecht, M. L., Miller-Day, M., Krieger, J. R., Syvertsen, A. K., Graham, J. W., et al. (2013). Adapting school-based substance use prevention curriculum through cultural grounding: a review and exemplar of adaptation processes for rural schools. American Journal of Community Psychology, 1–2, 190–205. doi:10.1007/s10464-012-9524-8.CrossRefGoogle Scholar
- Graczyk, P. A., Domitrovich, C. E., & Zins, J. E. (2002). Facilitating the implementation of evidence-based prevention and mental health promotion efforts in schools. Handbook of School Mental Health Advancing Practice and Research, 301–318. doi:10.1007/978-0-387-73313-5_21
- Ho, A. D. & Kane, T. J. (2013). The reliability of classroom observations by school personnel Retrieved from http://metproject.org/downloads/MET_Reliability_of_Classroom_Observations_Research_Paper.pdf
- Knoche, L. L., Sheridan, S. M., Edwards, C. P., & Osborn, A. Q. (2010). Implementation of a relationships-based school readiness intervention: a multidimensional approach to fidelity measurement for early childhood. Early Childhood Research Quarterly, 25, 299–313. doi:10.1016/j.ecresq.2009.05.003.PubMedCentralCrossRefPubMedGoogle Scholar
- Marsiglia, F. F., Kulis, S., Yabiku, S. T., Nieri, T. A., & Coleman, E. (2011). When to intervene: elementary school, middle school or both? Effects of keepin’ it REAL on substance use trajectories of Mexican heritage youth. Prevention Science, 12, 48–62. doi:10.1007/s11121-010-0189-y.PubMedCentralCrossRefPubMedGoogle Scholar
- Miller-Day, M., Pettigrew, J., Hecht, M. L. M., Shin, Y., Graham, J., & Krieger, J. (2013). How prevention curricula are taught under real-world conditions: types of and reasons for teacher curriculum adaptations in 7th grade drug prevention curriculum. Health Education, 113, 324–344. doi:10.1108/09654281311329259.PubMedCentralCrossRefPubMedGoogle Scholar
- Odom, S. L., Fleming, K., Diamond, K., Lieber, J., Hanson, M., Butera, G., et al. (2010). Examining different forms of implementation and in early childhood curriculum research. Early Childhood Research Quarterly, 25, 314–328. doi:10.1016/j.ecresq.2010.03.001.PubMedCentralCrossRefPubMedGoogle Scholar
- Pettigrew, J., Miller-Day, M., Shin, Y., Hecht, M. L., Krieger, J. R., & Graham, J. W. (2013). Describing teacher-student interactions: a qualitative assessment of teacher implementation of the 7th grade keepin’ it REAL substance use intervention. American Journal of Community Psychology. doi:10.1007/s10464-012-9539-1.Google Scholar
- Shin, Y., Miller-Day, M., Pettigrew, J., & Hecht, M. L. (2011). Qualitative approach to implementer typologies: how teachers and students interact in implementation of school based prevention intervention. Washington DC: Paper presented at the meeting of the Society for Prevention Research.Google Scholar