Skip to main content

Developing Treatment Integrity Measures for Teacher-Delivered Interventions: Progress, Recommendations and Future Directions

Abstract

While the measurement of treatment integrity is important to determine how much, and how well, interventions are delivered in schools, the science of treatment integrity is not well developed in education research. The purpose of this paper is to describe a program of research that has developed treatment integrity measures over the past 10 years to assess teacher delivery of an indicated program targeting reductions in problem behavior in early childhood and elementary school classrooms. Specifically, this paper will highlight the importance of active use of conceptual models to guide treatment integrity measure development, multidimensional assessment of treatment integrity and training procedures for observers, using several studies to illustrate the evolution and refinement of our measurement approach. Recommendations for researchers developing and evaluating interventions in schools are provided, as are recommendations to help the field move toward a more rigorous science of treatment integrity.

This is a preview of subscription content, access via your institution.

Fig. 1

References

  1. Barber, J. P., Mercer, D., & Krakauer, I. (1996). Development of an adherence/competence rating scale for individual drug counseling. Drug and Alcohol Dependence, 43, 125–132.

  2. Berkel, C., Mauricio, A. M., Schoenfelder, E., & Sandler, I. N. (2011). Putting the pieces together: An integrated model of program implementation. Prevention Science, 12(1), 23–33. https://doi.org/10.1007/s11121-010-0186-1.

    Article  PubMed  PubMed Central  Google Scholar 

  3. Botvin, G. J., & Griffin, K. W. (2004). Life skills training: Empirical findings and future directions. Journal of Primary Prevention, 25(4), 211–232.

    Article  Google Scholar 

  4. Bronfenbrenner, U. (1979). The ecology of human development. Cambridge: Harvard University Press.

    Google Scholar 

  5. Carroll, K. M., Nich, C., Sifry, R. L., Nuro, K. F., Frankforter, T. L., Ball, S. A., Fenton, L., & Rounsaville, B. J. (2000). A general system for evaluating therapist adherence and competence in psychotherapy research in the addictions. Drug and Alcohol Dependence, 57(3), 225–238. https://doi.org/10.1016/s0376-8716(99)00049-6.

  6. Chorpita, B. F., & Daleiden, E. L. (2009). Mapping evidence-based treatments for children and adolescents: Application of the distillation and matching model to 615 treatments from 322 randomized trials. Journal of Consulting and Clinical Psychology, 77(3), 566. https://doi.org/10.1037/a0014565.

    Article  PubMed  Google Scholar 

  7. Cicchetti, D. V. (1994). Guidelines, criteria, and rules of thumb for evaluating normed and standardized assessment instruments in psychology. Psychological Assessment, 6(4), 284. https://doi.org/10.1037/1040-3590.6.4.284.

    Article  Google Scholar 

  8. Conroy, M. A., Sutherland, K. S., Algina, J., Ladwig, C., Werch, B., Martinez, J., et al. (2019). Outcomes of the BEST in CLASS intervention on teachers’ use of effective practices, self-efficacy, and classroom quality. School Psychology Review, 48, 31–45. https://doi.org/10.17105/SPR-2018-0003.V48-1.

    Article  Google Scholar 

  9. Conroy, M. A., Sutherland, K. S., Algina, J., Werch, B., & Ladwig, C. (2018). Prevention and treatment of externalizing problem behaviors in young children: Clinical implications from a randomized controlled trial of BEST in CLASS. AERA Open, 4(1), 1–16. https://doi.org/10.1177/2332858417750376.

    Article  Google Scholar 

  10. Conroy, M. A., Sutherland, K. S., Granger, K. L., Marcolouides, K. M., Feil, E., Huang, K., & Montession, A. (2020). Preliminary study of the effects of BEST in CLASS-Web on young children’s social-emotional and behavioral outcomes. Manuscript submitted for publication.

  11. Conroy, M. A., Sutherland, K. S., Wilson, R. E., Martinez, J., Whalon, K. J., & Algina, J. (2015). Measuring teacher implementation of the BEST in CLASS intervention program and corollary child outcomes. Journal of Emotional and Behavioral Disorders, 23, 144–155. https://doi.org/10.1177/1063426614532949.

    Article  Google Scholar 

  12. Domitrovich, C. E., Bradshaw, C. P., Poduska, J. M., Hoagwood, K., Buckley, J. A., Olin, S., et al. (2008). Maximizing the implementation quality of evidence-based preventive interventions in schools: A conceptual framework Advances in School Mental. Health Promotion, 1(3), 6–28. https://doi.org/10.1080/1754730X.2008.9715730.

    Article  Google Scholar 

  13. Durlak, J. A. (2010). The importance of doing well in whatever you do: A commentary on the special section, “Implementation research in early childhood education.” Early Childhood Research Quarterly, 25(3), 348–357. https://doi.org/10.1016/j.ecresq.2010.03.003.

    Article  Google Scholar 

  14. Durlak, J. A. (2015). Studying program implementation is not easy but it is essential. Prevention Science, 16(8), 1123–1127. https://doi.org/10.1007/s11121-015-0606-3.

    Article  PubMed  Google Scholar 

  15. Evans, M., Piasecki, J., Kriss, M., & Hollon, S. (1984). Rater’s manual for the collaborative study psychotherapy rating scale- form 6 (CSPRS-6). University of Minnesota and St. Paul-Tamsey Medical Center. Unpublished manuscript.

  16. Han, S. S., & Weiss, B. (2005). Sustainability of teacher implementation of school-based mental health programs. Journal of Abnormal Child Psychology, 33(6), 665–679. https://doi.org/10.1007/s10802-005-7646-2.

    Article  Google Scholar 

  17. Hogue, A., Dauber, S., Lichvar, E., Bobek, M., & Henderson, C. E. (2014). Validity of therapist self-report ratings of fidelity to evidence-based practices for adolescent behavior problems: Correspondence between therapists and observers. Administration and Policy in Mental Health and Mental Health Services Research, 42(2), 229–243. https://doi.org/10.1007/s10488-014-0548-2.

    Article  Google Scholar 

  18. Hogue, A., Liddle, H. A., & Rowe, C. (1996). Treatment adherence process research in family therapy: A rationale and some practical guidelines. Psychotherapy Theory, Research, Practice, Training, 33(2), 332. https://doi.org/10.1037/0033-3204.33.2.332.

    Article  Google Scholar 

  19. Hollon, S., Evans, M., Auerbach, A., DeRubeus, R., Elkin, I., Lowerry, A., et al. (1988). Development of a system for rating therapies for depression: Differentiating cognitive therapy, interpersonal psychotherapy, and clinical management pharmacotherapy. Vanderbilt University, Nashville, TN. Unpublished manuscript.

  20. Greenberg, L. S. (1986). Change process research. Journal of Consulting and Clinical Psychology, 54, 4–9.

    Article  Google Scholar 

  21. Hogue, A., Dauber, S., Chinchilla, P., Fried, A., Henderson, C., Inclan, J., Reiner, R. H., & Liddle, H. A. (2008). Assessing fidelity in individual and family therapy for adolescent substance abuse. Journal of Substance Abuse Treatment, 35(2), 137–147.

    Article  Google Scholar 

  22. McLeod, B. D., Kunemund, R., Nemer, S. L., & Lyon, A. R. (2020). Leveraging Implementation Science and Practice to Support the Delivery of Evidence-Based Practices in Services for Youth with Emotional and Behavioral Disorders. In T. Farmer, M. Conroy, E. Farmer, & K. Sutherland (Eds.), Handbook of Research on Emotional and Behavioral Disorders: Interdisciplinary Developmental Perspectives on Children and Youth (pp. 417–432). New York: Routledge/Taylor & Francis.

    Chapter  Google Scholar 

  23. McLeod, B. D., Southam-Gerow, M. A., Tully, C. B., Rodriguez, A., & Smith, M. M. (2013). Making a case for treatment integrity as a psychosocial treatment quality indicator for youth mental health care. Clinical Psychology: Science and Practice, 20(1), 14–32. https://doi.org/10.1111/cpsp.12020.

    Article  Google Scholar 

  24. McLeod, B. D., Southam-Gerow, M. A., & Weisz, J. R. (2009). Conceptual and methodological issues in treatment integrity measurement. School Psychology Review, 38(4), 541.

    Google Scholar 

  25. McLeod, B. D., Sutherland, K. S., Martinez, R. G., Conroy, M. A., Snyder, P. A., & Southam-Gerow, M. A. (2017). Identifying common practice elements to improve social, emotional, and behavioral outcomes of young children in early childhood classrooms. Prevention Science, 18(2), 204–213. https://doi.org/10.1007/s11121-016-0703-y.

    Article  PubMed  Google Scholar 

  26. McLeod, B. D., & Weisz, J. R. (2010). The therapy process observational coding system for child psychotherapy strategies scale. Journal of Clinical Child & Adolescent Psychology, 39(3), 436–443. https://doi.org/10.1080/15374411003691750.

    Article  Google Scholar 

  27. Perepletchikova, F., & Kazdin, A. E. (2005). Treatment integrity and therapeutic change: Issues and research recommendations. Clinical Psychology: Science and Practice, 12(4), 365–383. https://doi.org/10.1093/clipsy.bpi045.

    Article  Google Scholar 

  28. Proctor, E., Silmere, H., Raghavan, R., Hovmand, P., Aarons, G., Bunger, A., et al. (2011). Outcomes for implementation research: conceptual distinctions, measurement challenges, and research agenda. Administration and Policy in Mental Health and Mental Health Services Research, 38(2), 65–76. https://doi.org/10.1007/s10488-010-0319-7.

    Article  PubMed  PubMed Central  Google Scholar 

  29. Sanetti, L. M. H., Charbonneau, S., Knight, A., Cochrane, W. S., Kulcyk, M. C. M., & Kraus, K. E. (2020). Treatment fidelity reporting in intervention outcome studies in the school psychology literature from 2009 to 2016. Psychology in the Schools, 57, 901–922. https://doi.org/10.1002/pits.22364.

    Article  Google Scholar 

  30. Sanetti, L. M. H., Dobey, L. M., & Gritter, K. L. (2012). Treatment integrity of interventions with children in the Journal of Positive Behavior Interventions from 1999–2009. Journal of Positive Behaviour Interventions, 14, 29–46. https://doi.org/10.1177/1098300711405853.

    Article  Google Scholar 

  31. Sanetti, L. M. H., Gritter, K. L., & Dobey, L. M. (2011). Treatment fidelity of interventions with children in the school psychology literature from 1995–2008. School Psychology Review, 40, 72–84.

    Article  Google Scholar 

  32. Sanetti, L. M. H., & Kratochwill, T. R. (2008). Treatment integrity in behavioral consultation: Measurement, promotion, and outcomes. International Journal of Behavioral Consultation and Therapy, 4(1), 95–114. https://doi.org/10.1037/h0100835.

    Article  Google Scholar 

  33. Sanetti, L. M. H., & Kratochwill, T. R. (2009). Toward developing a science of treatment integrity: Introduction to the special series. School Psychology Review38(4), 445–459.

  34. Schoenwald, S. K. (2011). It’s a bird, it’s a plane, it’s… fidelity measurement in the real world. Clinical Psychology: Science and Practice, 18(2), 142–147. https://doi.org/10.1111/j.1468-2850.2011.01245.x.

    Article  Google Scholar 

  35. Schoenwald, S. K., Mchugh, R. K., & Barlow, D. H. (2012). The science of dissemination and implementation. In R. K. McHugh & D. H. Barlow (Eds.), Dissemination and implementation of evidence-based psychological interventions (pp. 16–42). Oxford: Oxford University Press.

    Google Scholar 

  36. Sutherland, K. S., Conroy, M. A., Abrams, L., Vo, A., & Ogston, P. (2013a). An initial evaluation of the Teacher-Child Interaction Direct Observation System: Measuring teacher-child interaction behaviors in classroom settings. Assessment for Effective Intervention, 39, 12–23. https://doi.org/10.1177/1534508412463814.

    Article  Google Scholar 

  37. Sutherland, K. S., Conroy, M. A., Algina, J., Ladwig, C., Jessee, G., & Gyure, M. (2018a). Reducing child problem behaviors and improving teacher-child interactions and relationships: A randomized controlled trial of BEST in CLASS. Early Childhood Research Quarterly, 42, 31–43. https://doi.org/10.1016/j.ecresq.2017.08.001.

    Article  Google Scholar 

  38. Sutherland, K. S., Conroy, M. A., & Granger, K. L. (2020). BEST in CLASS A Tier-2 program for children with and at risk for emotional/behavioral disorders. In T. W. Farmer, M. A. Conroy, E. M. Z. Farmer, & K. S. Sutherland (Eds.), Handbook of Research on Emotional and Behavioral Disorders (pp. 214–226). Abingdon: Routledge.

    Chapter  Google Scholar 

  39. Sutherland, K. S., Conroy, M. A., McLeod, B. D., Algina, J., & Kunemund, R. L. (2018b). Factors associated with teacher delivery of a classroom-based Tier 2 prevention program. Prevention Science, 19(2), 186–196. https://doi.org/10.1007/s11121-017-0832-y.

    Article  PubMed  PubMed Central  Google Scholar 

  40. Sutherland, K. S., Conroy, M. A., McLeod, B. D., Algina, J., & Wu, E. (2018c). Teacher competence of delivery of BEST in CLASS as a mediator of treatment effects. School Mental Health, 10(3), 214–225. https://doi.org/10.1007/s12310-017-9224-5.

    Article  Google Scholar 

  41. Sutherland, K. S., Conroy, M. A., McLeod, B. D., Kunemund, R., & McKnight, K. (2019). Common practice elements for improving social, emotional, and behavioral outcomes of young elementary school students. Journal of Emotional and Behavioral Disorders, 27(2), 76–85. https://doi.org/10.1177/1063426618784009.

    Article  Google Scholar 

  42. Sutherland, K. S., McLeod, B. D., Conroy, M. A., Abrams, L. M., & Smith, M. M. (2014). Preliminary psychometric properties of the BEST in CLASS Adherence and Competence Scale. Journal of Emotional and Behavioral Disorders, 22(4), 249–259. https://doi.org/10.1177/1063426613497258.

    Article  Google Scholar 

  43. Sutherland, K. S., McLeod, B. D., Conroy, M. A., & Cox, J. R. (2013b). Measuring implementation of evidence-based programs targeting young children at risk for emotional/behavioral disorders: Conceptual issues and recommendations. Journal of Early Intervention, 35(2), 129–149. https://doi.org/10.1177/1053815113515025.

    Article  Google Scholar 

  44. Vroom, E. B., Massey, O. T., Yampolskaya, S., & Levin, B. L. (2020). The Impact of Implementation Fidelity on Student Outcomes in the Life Skills Training Program. School Mental Health, 12(1), 113–123. https://doi.org/10.1007/s12310-019-09333-1.

    Article  Google Scholar 

  45. Waltz, J., Addis, M. E., Koerner, K., & Jacobson, N. S. (1993). Testing the integrity of a psychotherapy protocol: assessment of adherence and competence. Journal of consulting and clinical psychology, 61(4), 620. https://doi.org/10.1037/0022-006X.61.4.620.

    Article  PubMed  Google Scholar 

  46. Williford, A. P., Wolcott, C. S., Whittaker, J. V., & Locasale-Crouch, J. (2015). Program and teacher characteristics predicting the implementation of Banking Time with preschoolers who display disruptive behaviors. Prevention Science, 16(8), 1054–1063. https://doi.org/10.1007/s11121-015-0544-0.

    Article  PubMed  Google Scholar 

  47. Yoder, P., & Symons, F. (2010). Observational measurement of behavior. Berlin: Springer Publishing Company.

    Google Scholar 

Download references

Funding

This research was supported by Grants from the U.S. Department of Education’s Institute for Education Sciences (R324A11017, R305A140487, R305A150246, R324A160158 and R305A180182). The opinions expressed by the authors are not necessarily reflective of the position of or endorsed by the U.S. Department of Education.

Author information

Affiliations

Authors

Corresponding author

Correspondence to Kevin S. Sutherland.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Ethical approval

All study procedures involving human participants were in accordance with the ethical standards of the researchers’ institutional review boards and with the 1964 Helsinki Declaration and its later amendments of comparable ethical standards.

Informed consent

Informed consent was obtained from all individual participants included in the study.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Sutherland, K.S., McLeod, B.D., Conroy, M.A. et al. Developing Treatment Integrity Measures for Teacher-Delivered Interventions: Progress, Recommendations and Future Directions. School Mental Health (2021). https://doi.org/10.1007/s12310-021-09423-z

Download citation