While the measurement of treatment integrity is important to determine how much, and how well, interventions are delivered in schools, the science of treatment integrity is not well developed in education research. The purpose of this paper is to describe a program of research that has developed treatment integrity measures over the past 10 years to assess teacher delivery of an indicated program targeting reductions in problem behavior in early childhood and elementary school classrooms. Specifically, this paper will highlight the importance of active use of conceptual models to guide treatment integrity measure development, multidimensional assessment of treatment integrity and training procedures for observers, using several studies to illustrate the evolution and refinement of our measurement approach. Recommendations for researchers developing and evaluating interventions in schools are provided, as are recommendations to help the field move toward a more rigorous science of treatment integrity.
This is a preview of subscription content, access via your institution.
Buy single article
Instant access to the full article PDF.
Tax calculation will be finalised during checkout.
Barber, J. P., Mercer, D., & Krakauer, I. (1996). Development of an adherence/competence rating scale for individual drug counseling. Drug and Alcohol Dependence, 43, 125–132.
Berkel, C., Mauricio, A. M., Schoenfelder, E., & Sandler, I. N. (2011). Putting the pieces together: An integrated model of program implementation. Prevention Science, 12(1), 23–33. https://doi.org/10.1007/s11121-010-0186-1.
Botvin, G. J., & Griffin, K. W. (2004). Life skills training: Empirical findings and future directions. Journal of Primary Prevention, 25(4), 211–232.
Bronfenbrenner, U. (1979). The ecology of human development. Cambridge: Harvard University Press.
Carroll, K. M., Nich, C., Sifry, R. L., Nuro, K. F., Frankforter, T. L., Ball, S. A., Fenton, L., & Rounsaville, B. J. (2000). A general system for evaluating therapist adherence and competence in psychotherapy research in the addictions. Drug and Alcohol Dependence, 57(3), 225–238. https://doi.org/10.1016/s0376-8716(99)00049-6.
Chorpita, B. F., & Daleiden, E. L. (2009). Mapping evidence-based treatments for children and adolescents: Application of the distillation and matching model to 615 treatments from 322 randomized trials. Journal of Consulting and Clinical Psychology, 77(3), 566. https://doi.org/10.1037/a0014565.
Cicchetti, D. V. (1994). Guidelines, criteria, and rules of thumb for evaluating normed and standardized assessment instruments in psychology. Psychological Assessment, 6(4), 284. https://doi.org/10.1037/1040-3522.214.171.1244.
Conroy, M. A., Sutherland, K. S., Algina, J., Ladwig, C., Werch, B., Martinez, J., et al. (2019). Outcomes of the BEST in CLASS intervention on teachers’ use of effective practices, self-efficacy, and classroom quality. School Psychology Review, 48, 31–45. https://doi.org/10.17105/SPR-2018-0003.V48-1.
Conroy, M. A., Sutherland, K. S., Algina, J., Werch, B., & Ladwig, C. (2018). Prevention and treatment of externalizing problem behaviors in young children: Clinical implications from a randomized controlled trial of BEST in CLASS. AERA Open, 4(1), 1–16. https://doi.org/10.1177/2332858417750376.
Conroy, M. A., Sutherland, K. S., Granger, K. L., Marcolouides, K. M., Feil, E., Huang, K., & Montession, A. (2020). Preliminary study of the effects of BEST in CLASS-Web on young children’s social-emotional and behavioral outcomes. Manuscript submitted for publication.
Conroy, M. A., Sutherland, K. S., Wilson, R. E., Martinez, J., Whalon, K. J., & Algina, J. (2015). Measuring teacher implementation of the BEST in CLASS intervention program and corollary child outcomes. Journal of Emotional and Behavioral Disorders, 23, 144–155. https://doi.org/10.1177/1063426614532949.
Domitrovich, C. E., Bradshaw, C. P., Poduska, J. M., Hoagwood, K., Buckley, J. A., Olin, S., et al. (2008). Maximizing the implementation quality of evidence-based preventive interventions in schools: A conceptual framework Advances in School Mental. Health Promotion, 1(3), 6–28. https://doi.org/10.1080/1754730X.2008.9715730.
Durlak, J. A. (2010). The importance of doing well in whatever you do: A commentary on the special section, “Implementation research in early childhood education.” Early Childhood Research Quarterly, 25(3), 348–357. https://doi.org/10.1016/j.ecresq.2010.03.003.
Durlak, J. A. (2015). Studying program implementation is not easy but it is essential. Prevention Science, 16(8), 1123–1127. https://doi.org/10.1007/s11121-015-0606-3.
Evans, M., Piasecki, J., Kriss, M., & Hollon, S. (1984). Rater’s manual for the collaborative study psychotherapy rating scale- form 6 (CSPRS-6). University of Minnesota and St. Paul-Tamsey Medical Center. Unpublished manuscript.
Han, S. S., & Weiss, B. (2005). Sustainability of teacher implementation of school-based mental health programs. Journal of Abnormal Child Psychology, 33(6), 665–679. https://doi.org/10.1007/s10802-005-7646-2.
Hogue, A., Dauber, S., Lichvar, E., Bobek, M., & Henderson, C. E. (2014). Validity of therapist self-report ratings of fidelity to evidence-based practices for adolescent behavior problems: Correspondence between therapists and observers. Administration and Policy in Mental Health and Mental Health Services Research, 42(2), 229–243. https://doi.org/10.1007/s10488-014-0548-2.
Hogue, A., Liddle, H. A., & Rowe, C. (1996). Treatment adherence process research in family therapy: A rationale and some practical guidelines. Psychotherapy Theory, Research, Practice, Training, 33(2), 332. https://doi.org/10.1037/0033-3126.96.36.1992.
Hollon, S., Evans, M., Auerbach, A., DeRubeus, R., Elkin, I., Lowerry, A., et al. (1988). Development of a system for rating therapies for depression: Differentiating cognitive therapy, interpersonal psychotherapy, and clinical management pharmacotherapy. Vanderbilt University, Nashville, TN. Unpublished manuscript.
Greenberg, L. S. (1986). Change process research. Journal of Consulting and Clinical Psychology, 54, 4–9.
Hogue, A., Dauber, S., Chinchilla, P., Fried, A., Henderson, C., Inclan, J., Reiner, R. H., & Liddle, H. A. (2008). Assessing fidelity in individual and family therapy for adolescent substance abuse. Journal of Substance Abuse Treatment, 35(2), 137–147.
McLeod, B. D., Kunemund, R., Nemer, S. L., & Lyon, A. R. (2020). Leveraging Implementation Science and Practice to Support the Delivery of Evidence-Based Practices in Services for Youth with Emotional and Behavioral Disorders. In T. Farmer, M. Conroy, E. Farmer, & K. Sutherland (Eds.), Handbook of Research on Emotional and Behavioral Disorders: Interdisciplinary Developmental Perspectives on Children and Youth (pp. 417–432). New York: Routledge/Taylor & Francis.
McLeod, B. D., Southam-Gerow, M. A., Tully, C. B., Rodriguez, A., & Smith, M. M. (2013). Making a case for treatment integrity as a psychosocial treatment quality indicator for youth mental health care. Clinical Psychology: Science and Practice, 20(1), 14–32. https://doi.org/10.1111/cpsp.12020.
McLeod, B. D., Southam-Gerow, M. A., & Weisz, J. R. (2009). Conceptual and methodological issues in treatment integrity measurement. School Psychology Review, 38(4), 541.
McLeod, B. D., Sutherland, K. S., Martinez, R. G., Conroy, M. A., Snyder, P. A., & Southam-Gerow, M. A. (2017). Identifying common practice elements to improve social, emotional, and behavioral outcomes of young children in early childhood classrooms. Prevention Science, 18(2), 204–213. https://doi.org/10.1007/s11121-016-0703-y.
McLeod, B. D., & Weisz, J. R. (2010). The therapy process observational coding system for child psychotherapy strategies scale. Journal of Clinical Child & Adolescent Psychology, 39(3), 436–443. https://doi.org/10.1080/15374411003691750.
Perepletchikova, F., & Kazdin, A. E. (2005). Treatment integrity and therapeutic change: Issues and research recommendations. Clinical Psychology: Science and Practice, 12(4), 365–383. https://doi.org/10.1093/clipsy.bpi045.
Proctor, E., Silmere, H., Raghavan, R., Hovmand, P., Aarons, G., Bunger, A., et al. (2011). Outcomes for implementation research: conceptual distinctions, measurement challenges, and research agenda. Administration and Policy in Mental Health and Mental Health Services Research, 38(2), 65–76. https://doi.org/10.1007/s10488-010-0319-7.
Sanetti, L. M. H., Charbonneau, S., Knight, A., Cochrane, W. S., Kulcyk, M. C. M., & Kraus, K. E. (2020). Treatment fidelity reporting in intervention outcome studies in the school psychology literature from 2009 to 2016. Psychology in the Schools, 57, 901–922. https://doi.org/10.1002/pits.22364.
Sanetti, L. M. H., Dobey, L. M., & Gritter, K. L. (2012). Treatment integrity of interventions with children in the Journal of Positive Behavior Interventions from 1999–2009. Journal of Positive Behaviour Interventions, 14, 29–46. https://doi.org/10.1177/1098300711405853.
Sanetti, L. M. H., Gritter, K. L., & Dobey, L. M. (2011). Treatment fidelity of interventions with children in the school psychology literature from 1995–2008. School Psychology Review, 40, 72–84.
Sanetti, L. M. H., & Kratochwill, T. R. (2008). Treatment integrity in behavioral consultation: Measurement, promotion, and outcomes. International Journal of Behavioral Consultation and Therapy, 4(1), 95–114. https://doi.org/10.1037/h0100835.
Sanetti, L. M. H., & Kratochwill, T. R. (2009). Toward developing a science of treatment integrity: Introduction to the special series. School Psychology Review, 38(4), 445–459.
Schoenwald, S. K. (2011). It’s a bird, it’s a plane, it’s… fidelity measurement in the real world. Clinical Psychology: Science and Practice, 18(2), 142–147. https://doi.org/10.1111/j.1468-2850.2011.01245.x.
Schoenwald, S. K., Mchugh, R. K., & Barlow, D. H. (2012). The science of dissemination and implementation. In R. K. McHugh & D. H. Barlow (Eds.), Dissemination and implementation of evidence-based psychological interventions (pp. 16–42). Oxford: Oxford University Press.
Sutherland, K. S., Conroy, M. A., Abrams, L., Vo, A., & Ogston, P. (2013a). An initial evaluation of the Teacher-Child Interaction Direct Observation System: Measuring teacher-child interaction behaviors in classroom settings. Assessment for Effective Intervention, 39, 12–23. https://doi.org/10.1177/1534508412463814.
Sutherland, K. S., Conroy, M. A., Algina, J., Ladwig, C., Jessee, G., & Gyure, M. (2018a). Reducing child problem behaviors and improving teacher-child interactions and relationships: A randomized controlled trial of BEST in CLASS. Early Childhood Research Quarterly, 42, 31–43. https://doi.org/10.1016/j.ecresq.2017.08.001.
Sutherland, K. S., Conroy, M. A., & Granger, K. L. (2020). BEST in CLASS A Tier-2 program for children with and at risk for emotional/behavioral disorders. In T. W. Farmer, M. A. Conroy, E. M. Z. Farmer, & K. S. Sutherland (Eds.), Handbook of Research on Emotional and Behavioral Disorders (pp. 214–226). Abingdon: Routledge.
Sutherland, K. S., Conroy, M. A., McLeod, B. D., Algina, J., & Kunemund, R. L. (2018b). Factors associated with teacher delivery of a classroom-based Tier 2 prevention program. Prevention Science, 19(2), 186–196. https://doi.org/10.1007/s11121-017-0832-y.
Sutherland, K. S., Conroy, M. A., McLeod, B. D., Algina, J., & Wu, E. (2018c). Teacher competence of delivery of BEST in CLASS as a mediator of treatment effects. School Mental Health, 10(3), 214–225. https://doi.org/10.1007/s12310-017-9224-5.
Sutherland, K. S., Conroy, M. A., McLeod, B. D., Kunemund, R., & McKnight, K. (2019). Common practice elements for improving social, emotional, and behavioral outcomes of young elementary school students. Journal of Emotional and Behavioral Disorders, 27(2), 76–85. https://doi.org/10.1177/1063426618784009.
Sutherland, K. S., McLeod, B. D., Conroy, M. A., Abrams, L. M., & Smith, M. M. (2014). Preliminary psychometric properties of the BEST in CLASS Adherence and Competence Scale. Journal of Emotional and Behavioral Disorders, 22(4), 249–259. https://doi.org/10.1177/1063426613497258.
Sutherland, K. S., McLeod, B. D., Conroy, M. A., & Cox, J. R. (2013b). Measuring implementation of evidence-based programs targeting young children at risk for emotional/behavioral disorders: Conceptual issues and recommendations. Journal of Early Intervention, 35(2), 129–149. https://doi.org/10.1177/1053815113515025.
Vroom, E. B., Massey, O. T., Yampolskaya, S., & Levin, B. L. (2020). The Impact of Implementation Fidelity on Student Outcomes in the Life Skills Training Program. School Mental Health, 12(1), 113–123. https://doi.org/10.1007/s12310-019-09333-1.
Waltz, J., Addis, M. E., Koerner, K., & Jacobson, N. S. (1993). Testing the integrity of a psychotherapy protocol: assessment of adherence and competence. Journal of consulting and clinical psychology, 61(4), 620. https://doi.org/10.1037/0022-006X.61.4.620.
Williford, A. P., Wolcott, C. S., Whittaker, J. V., & Locasale-Crouch, J. (2015). Program and teacher characteristics predicting the implementation of Banking Time with preschoolers who display disruptive behaviors. Prevention Science, 16(8), 1054–1063. https://doi.org/10.1007/s11121-015-0544-0.
Yoder, P., & Symons, F. (2010). Observational measurement of behavior. Berlin: Springer Publishing Company.
This research was supported by Grants from the U.S. Department of Education’s Institute for Education Sciences (R324A11017, R305A140487, R305A150246, R324A160158 and R305A180182). The opinions expressed by the authors are not necessarily reflective of the position of or endorsed by the U.S. Department of Education.
Conflict of interest
The authors declare that they have no conflict of interest.
All study procedures involving human participants were in accordance with the ethical standards of the researchers’ institutional review boards and with the 1964 Helsinki Declaration and its later amendments of comparable ethical standards.
Informed consent was obtained from all individual participants included in the study.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Sutherland, K.S., McLeod, B.D., Conroy, M.A. et al. Developing Treatment Integrity Measures for Teacher-Delivered Interventions: Progress, Recommendations and Future Directions. School Mental Health (2021). https://doi.org/10.1007/s12310-021-09423-z