School Mental Health

, Volume 10, Issue 3, pp 214–225 | Cite as

Teacher Competence of Delivery of BEST in CLASS as a Mediator of Treatment Effects

  • Kevin S. SutherlandEmail author
  • Maureen A. Conroy
  • Bryce D. McLeod
  • James Algina
  • Eleanor Wu
Original Paper


This study investigated if training and practice-based coaching in an evidence-based program was associated with higher observed treatment integrity (adherence and competence) and if these treatment integrity components were associated with teacher report of child behavioral outcomes in the BEST in CLASS efficacy trial. Participants were 462 children (M = 4.32 years, SD = 0.53; 65% male; 17.0% Caucasian, 66.0% African-American, 5.0% Hispanic, and 12.0% other) identified as having problem behavior and their 185 teachers (M = 12.09 years teaching experience; 99% female; 47.0% Caucasian, 48.0% African-American, 1.0% Asian/Pacific Islander, 3.0% Hispanic and 1.0% other). Teachers and focal children were randomly assigned to the intervention (teacher n = 92, children n = 230) or control condition (teacher n = 93, child n = 232). Results of a multilevel mediation analysis indicated that the BEST in CLASS intervention had a positive effect on teacher report of child problem behavior (SSIS-RS) and externalizing problems (C-TRF), as well as having a positive effect on teachers’ adherence and competence of delivery of the intervention. There was an indirect effect through competence of delivery for externalizing problems, but not problem behavior. No indirect effects for adherence were found. Implications of these findings and directions for future research are discussed.


Treatment integrity Teacher delivery Problem behavior Evidence-based program 



This research was supported by a Grant (R324A110173) from the U. S. Department of Education, Institute for Education Sciences, with additional support from an NIH/NCATS Clinical and Translational Science Award to the University of Florida UL1 TR000064. The opinions expressed by the authors are not necessarily reflective of the position of or endorsed by the U. S. Department of Education.

Compliance with Ethical Standards

Conflict of interest

The authors declare that they have no conflict of interest.

Ethical Approval

All study procedures involving human participants were in accordance with the ethical standards of the researchers’ Institutional Review Boards and with the 1964 Helsinki declaration and its later amendments of comparable ethical standards.

Informed Consent

Informed consent was obtained from all individual participants included in the study.


  1. Abry, T., Rimm-Kaufman, S. E., Larsen, R. A., & Brewer, A. J. (2013). The influence of fidelity of implementation on teacher–student interaction quality in the context of a randomized controlled trial of the Responsive Classroom approach. Journal of School Psychology, 51(4), 437–453. doi: 10.1016/j.jsp.2013.03.001.PubMedCrossRefGoogle Scholar
  2. Achenbach, T. M., & Rescorla, L. A. (2000). ASEBA preschool forms & profiles. Burlington, VT: University of Vermont, Research Center for Children, Youth and Families.Google Scholar
  3. Allen, J. D., Linnan, L. A., & Emmons, K. M. (2012). Fidelity and its relationship to implementation effectiveness, adaptation, and dissemination. Dissemination and Implementation Research in Health: Translating Science to Practice. doi: 10.1093/acprof:oso/9780199751877.003.0014.Google Scholar
  4. Barbarin, O. A. (2007). Mental health screening of preschool children: validity and reliability of ABLE. American Journal of Orthopsychiatry, 77(3), 402–408. doi: 10.1037/0002-9432.77.3.402.PubMedCrossRefGoogle Scholar
  5. Barber, J. P., Crits-Christoph, P., & Luborsky, L. (1996). Effects of therapist adherence and competence on patient outcome in brief dynamic therapy. Journal of Consulting and Clinical Psychology, 64(3), 619–622.PubMedCrossRefGoogle Scholar
  6. Berry, V., Axford, N., Blower, S., Taylor, R. S., Edwards, R. T., Tobin, K., et al. (2016). The effectiveness and micro-costing analysis of a universal, school-based, social–emotional learning programme in the UK: A cluster-randomised controlled trial. School Mental Health, 8(2), 238–256. doi: 10.1007/s12310-015-9160-1.CrossRefGoogle Scholar
  7. Bierman, K. L., Coie, J., Dodge, K., Greenberg, M., Lochman, J., McMohan, R., et al. (2013). School outcomes of aggressive–disruptive children: Prediction from kindergarten risk factors and impact of the fast track prevention program. Aggressive Behavior, 39(2), 114–130. doi: 10.1002/ab.21467.PubMedCrossRefGoogle Scholar
  8. Brauner, C. B., & Stephens, C. B. (2006). Estimating the prevalence of early childhood serious emotional/behavioral disorders: Challenges and recommendations. Public Health Reports, 121(3), 303–310. doi: 10.1177/003335490612100314.PubMedPubMedCentralCrossRefGoogle Scholar
  9. Brennan, E. M., Bradley, J. R., Allen, M. D., & Perry, D. F. (2008). The evidence base for mental health consultation in early childhood settings: Research synthesis addressing staff and program outcomes. Early Education and Development, 19(6), 982–1022. doi: 10.1080/10409280801975834.CrossRefGoogle Scholar
  10. Bulotsky-Shearer, R. J., Bell, E. R., & Domínguez, X. (2012). Latent profiles of problem behavior within learning, peer, and teacher contexts: Identifying subgroups of children at academic risk across the preschool year. Journal of School Psychology, 50(6), 775–798. doi: 10.1016/j.jsp.2012.08.001.PubMedCrossRefGoogle Scholar
  11. Carroll, K. M., Nich, C., Sifry, R. L., Nuro, K. F., Frankforter, T. L., Ball, S. A., et al. (2000). A general system for evaluating therapist adherence and competence in psychotherapy research in the addictions. Drug and Alcohol Dependence, 57(3), 225–238. doi: 10.1016/S0376-8716(99)00049-6.PubMedCrossRefGoogle Scholar
  12. Cicchetti, D. V. (1994). Guidelines, criteria, and rules of thumb for evaluating normed and standardized assessment instruments in psychology. Psychological Assessment, 6(4), 284–290. doi: 10.1037/1040-3590.6.4.284.CrossRefGoogle Scholar
  13. Conroy, M. A., & Sutherland, K. S. (2011). Promoting social, emotional, and behavioral competence in young high-risk children: A preventative classroom-based early intervention model. Unpublished manual, Virginia Commonwealth University, Richmond, VA.Google Scholar
  14. Conroy, M. A., Sutherland, K. S., Algina, J. J., Wilson, R. E., Martinez, J. R., & Whalon, K. J. (2015). Measuring teacher implementation of the intervention program and corollary child outcomes. Journal of Emotional and Behavioral Disorders, 23(3), 144–155.Google Scholar
  15. Conroy, M. A., Sutherland, K. S., Vo, A. K., Carr, S., & Ogston, P. L. (2014). Early childhood teachers’ use of effective instructional practices and the collateral effects on young children’s behavior. Journal of Positive Behavior Interventions, 16(2), 81–92. doi: 10.1177/1098300713478666.
  16. Crits-Christoph, P., Siqueland, L., Chittams, J., Barber, J. P., Beck, A. T., Frank, A., et al. (1998). Training in cognitive, supportive-expressive, and drug counseling therapies for cocaine dependence. Journal of Consulting and Clinical Psychology, 66(3), 484–492. doi: 10.1037//0022-006X.66.3.484.PubMedCrossRefGoogle Scholar
  17. Domitrovich, C. E., Cortes, R. C., & Greenberg, M. T. (2007). Improving young children’s social and emotional competence: A randomized trial of the preschool “PATHS” curriculum. The Journal of Primary Prevention, 28(2), 67–91. doi: 10.1007/s10935-007-0081-0.PubMedCrossRefGoogle Scholar
  18. Domitrovich, C. E., Gest, S. D., Jones, D., Gill, S., & DeRousie, R. M. S. (2010). Implementation quality: Lessons learned in the context of the Head Start REDI trial. Early Childhood Research Quarterly, 25(3), 284–298. doi: 10.1016/j.ecresq.2010.04.001.PubMedPubMedCentralCrossRefGoogle Scholar
  19. Doumen, S., Verschueren, K., Buyse, E., Germeijs, V., Luyckx, K., & Soenens, B. (2008). Reciprocal relations between teacher–child conflict and aggressive behavior in kindergarten: A three-wave longitudinal study. Journal of Clinical Child & Adolescent Psychology, 37(3), 588–599. doi: 10.1016/j.jsp.2007.06.009.CrossRefGoogle Scholar
  20. Durlak, J. A. (2010). The importance of doing well in whatever you do: A commentary on the special section, “Implementation research in early childhood education”. Early Childhood Research Quarterly, 25(3), 348–357. doi: 10.1016/j.ecresq.2010.03.003.CrossRefGoogle Scholar
  21. Durlak, J. A. (2015). Studying program implementation is not easy but it is essential. Prevention Science, 16(8), 1123–1127. doi: 10.1007/s11121-015-0606-3.PubMedCrossRefGoogle Scholar
  22. Durlak, J. A., & DuPre, E. P. (2008). Implementation matters: A review of research on the influence of implementation on program outcomes and the factors affecting implementation. American Journal of Community Psychology, 41(3–4), 327–350. doi: 10.1007/s10464-008-9165-0.PubMedCrossRefGoogle Scholar
  23. Durlak, J. A., Weissberg, R. P., Dymnicki, A. B., Taylor, R. D., & Schellinger, K. B. (2011). The impact of enhancing students’ social and emotional learning: A meta-analysis of school-based universal interventions. Child Development, 82(1), 405–432. doi: 10.1111/j.1467-8624.2010.01564.x.PubMedCrossRefGoogle Scholar
  24. Feeley, M., DeRubeis, R. J., & Gelfand, L. A. (1999). The temporal relation of adherence and alliance to symptom change in cognitive therapy for depression. Journal of Consulting and Clinical Psychology, 67(4), 578–582.PubMedCrossRefGoogle Scholar
  25. Feil, E. G., Severson, H. H., & Walker, H. M. (1998). Screening for emotional and behavioral delays the Early Screening Project. Journal of Early Intervention, 21(3), 252–266. doi: 10.1177/105381519802100306.CrossRefGoogle Scholar
  26. Feil, E. G., Small, J. W., Forness, S. R., Serna, L. A., Kaiser, A. P., Hancock, T. B., et al. (2005). Using different measures, informants, and clinical cut-off points to estimate prevalence of emotional or behavioral disorders in preschoolers: Effects on age, gender, and ethnicity. Behavioral Disorders, 30(4), 375–391.CrossRefGoogle Scholar
  27. Fox, L., Hemmeter, M. L., Snyder, P., Binder, D. P., & Clarke, S. (2011). Coaching early childhood special educators to implement a comprehensive model for promoting young children’s social competence. Topics in Early Childhood Special Education, 31(3), 178–192. doi: 10.1177/0271121411404440.CrossRefGoogle Scholar
  28. Gilliam, W. S., & Marchesseault, C. M. (2005). From capitols to classrooms, policy to practice: State funded Pre-k at the classroom level. New Haven, CT: Yale University Child Study Center.Google Scholar
  29. Goncy, E. A., Sutherland, K. S., Farrell, A. D., Sullivan, T. N., & Doyle, S. T. (2015). Measuring teacher implementation in delivery of a bullying prevention program: The impact of instructional and procedural adherence and competence on student responsiveness. Prevention Science, 16(3), 440–450. doi: 10.1007/s11121-014-0508-9.PubMedCrossRefGoogle Scholar
  30. Greenberg, M. T. (2010). School-based prevention: Current status and future challenges. Effective Education, 2(1), 27–52. doi: 10.1080/19415531003616862.CrossRefGoogle Scholar
  31. Gresham, F., & Elliott, S. N. (2008). Social skills improvement system (SSIS) rating scales. Bloomington, MN: Pearson Assessments.Google Scholar
  32. Hamre, B. K., Justice, L. M., Pianta, R. C., Kilday, C., Sweeney, B., Downer, J. T., et al. (2010). Implementation fidelity of My Teaching Partner literacy and language activities: Association with preschoolers’ language and literacy growth. Early Childhood Research Quarterly, 25(3), 329–347. doi: 10.1016/j.ecresq.2009.07.002.CrossRefGoogle Scholar
  33. Hamre, B. K., & Pianta, R. C. (2001). Early teacher–child relationships and the trajectory of children’s school outcomes through eighth grade. Child Development, 72(2), 625–638. doi: 10.1111/1467-8624.00301.PubMedCrossRefGoogle Scholar
  34. Harn, B., Parisi, D., & Stoolmiller, M. (2013). Balancing fidelity with flexibility and fit: What do we really know about fidelity of implementation in schools? Exceptional Children, 79(2), 181–193. doi: 10.1177/001440291307900204.CrossRefGoogle Scholar
  35. Hedges, L. V. (2007). Effect sizes in cluster-randomized designs. Journal of Educational and Behavioral Statistics, 32(4), 341–370. doi: 10.3102/1076998606298043.CrossRefGoogle Scholar
  36. Hogue, A., Henderson, C. E., Dauber, S., Barajas, P. C., Fried, A., & Liddle, H. A. (2008). Treatment adherence, competence, and outcome in individual and family therapy for adolescent behavior problems. Journal of Consulting and Clinical Psychology, 76(4), 544–555. doi: 10.1037/0022-006X.76.4.544.PubMedPubMedCentralCrossRefGoogle Scholar
  37. LoCasale-Crouch, J., Cabell, S. Q., Jimenez, M., & Taylor, M. (2014). Research on consultation in early childhood programs. Handbook of research in school consultation. doi: 10.4324/9780203133170.ch19.Google Scholar
  38. Lüdtke, O., Marsh, H. W., Robitzsch, A., Trautwein, U., Asparouhov, T., & Muthen, B. (2008). The multilevel latent covariate model: A new, more reliable approach to group-level effects in contextual studies. Psychological Methods, 13(3), 203–229. doi: 10.1037/a0012869.PubMedCrossRefGoogle Scholar
  39. McHugh, R. K., & Barlow, D. H. (2010). The dissemination and implementation of evidence-based psychological treatments: A review of current efforts. American Psychologist, 65(2), 73–84. doi: 10.1037/a0018121.PubMedCrossRefGoogle Scholar
  40. McLeod, B. D., Southam-Gerow, M. A., Rodriguez, A., Quinoy, A., Arnold, C., Kendall, P. C., et al. (2016). Development and initial psychometrics for a therapist competence instrument for CBT for youth anxiety. Journal of Clinical Child and Adolescent Psychology. doi: 10.1080/15374416.2016.1253018.PubMedGoogle Scholar
  41. Mendel, P., Meredith, L. S., Schoenbaum, M., Sherbourne, C. D., & Wells, K. B. (2008). Interventions in organizational and community context: A framework for building evidence on dissemination and implementation in health services research. Administration and Policy in Mental Health and Mental Health Services Research, 35(1–2), 21–37. doi: 10.1007/s10488-007-0144-9.PubMedCrossRefGoogle Scholar
  42. Muthén, L. K. & Muthén, B. O. (1998–2015). Mplus user’s guide (7th ed.). Los Angeles, CA: Muthén & Muthén.Google Scholar
  43. Newborg, J. (2005). Battelle developmental inventory, Second edition: Examiner’s manual. Itasca, IL: Riverside.Google Scholar
  44. Pituch, K. A., Stapleton, L. M., & Kang, J. Y. (2006). A comparison of single sample and bootstrap methods to assess mediation in cluster randomized trials. Multivariate Behavioral Research, 41(3), 367–400. doi: 10.1207/s15327906mbr4103_5.PubMedCrossRefGoogle Scholar
  45. Proctor, E., Silmere, H., Raghavan, R., Hovmand, P., Aarons, G., Bunger, A., et al. (2011). Outcomes for implementation research: Conceptual distinctions, measurement challenges, and research agenda. Administration and Policy in Mental Health and Mental Health Services Research, 38(2), 65–76. doi: 10.1007/s10488-010-0319-7.PubMedCrossRefGoogle Scholar
  46. Qi, C. H., & Kaiser, A. P. (2003). Behavior problems of preschool children from low-income families: Review of the literature. Topics in Early Childhood Special Education, 23(4), 188–216. doi: 10.1177/02711214030230040201.CrossRefGoogle Scholar
  47. Raudenbush, S. W., & Bryk, A. S. (2002). Hierarchical linear models: Applications and data analysis methods (Vol. 1). Thousand Oaks: Sage.Google Scholar
  48. Roorda, D. L., Verschueren, K., Vancraeyveldt, C., Van Craeyevelt, S., & Colpin, H. (2014). Teacher–child relationships and behavioral adjustment: Transactional links for preschool boys at risk. Journal of School Psychology, 52(5), 495–510. doi: 10.1016/j.jsp.2014.06.004.PubMedCrossRefGoogle Scholar
  49. Sameroff, A. J., & Mackenzie, M. J. (2003). Research strategies for capturing transactional models of development: The limits of the possible. Development and Psychopathology, 15(03), 613–640. doi: 10.1017/S0954579403000312.PubMedCrossRefGoogle Scholar
  50. Schoenwald, S. K. (2012). The transport and diffusion of multisystemic therapy. In Dissemination and implementation of evidence-based psychological interventions (pp. 227–246). doi: 10.1093/med:psych/9780195389050.003.0012.
  51. Schoenwald, S. K., Garland, A. F., Chapman, J. E., Frazier, S. L., Sheidow, A. J., & Southam-Gerow, M. A. (2011). Toward the effective and efficient measurement of implementation fidelity. Administration and Policy in Mental Health and Mental Health Services Research, 38(1), 32–43. doi: 10.1007/s10488-010-0321-0.PubMedCrossRefGoogle Scholar
  52. Simons, A. D., Padesky, C. A., Montemarano, J., Lewis, C. C., Murakami, J., Lamb, K., et al. (2010). Training and dissemination of cognitive behavior therapy for depression in adults: A preliminary examination of therapist competence and client outcomes. Journal of Consulting and Clinical Psychology, 78(5), 751. doi: 10.1037/a0020569.PubMedCrossRefGoogle Scholar
  53. Snyder, P. A., Hemmeter, M. L., & Fox, L. (2015). Supporting implementation of evidence-based practices through practice-based coaching. Topics in Early Childhood Special Education, 35(3), 133–143. doi: 10.1177/0271121415594925.CrossRefGoogle Scholar
  54. Snyder, P., Hemmeter, M. L., Meeker, K. A., Kinder, K., Pasia, C., & McLaughlin, T. (2012). Characterizing key features of the early childhood professional development literature. Infants & Young Children, 25(3), 188–212. doi: 10.1097/IYC.0b013e31825a1ebf.CrossRefGoogle Scholar
  55. Stapleton, L. M., Pituch, K. A., & Dion, E. (2015). Standardized effect size measures for mediation analysis in cluster-randomized trials. Journal of Experimental Education, 83(4), 547–582. doi: 10.1080/00220973.2014.919569.CrossRefGoogle Scholar
  56. Sutherland, K. S., Conroy, M. A., Algina, J., Ladwig, C., Jessee, G., & Gyure, M. (2017). Reducing child problem behaviors and improving teacher-child interactions and relationships: A randomized controlled trial of BEST in CLASS. Manuscript submitted for publication. Google Scholar
  57. Sutherland, K. S., Conroy, M. A., Vo, A., & Ladwig, C. (2015). Implementation integrity of practice-based coaching: Preliminary results from the BEST in CLASS efficacy trial. School Mental Health, 7(1), 21–33.Google Scholar
  58. Sutherland, K. S., McLeod, B. D., Conroy, M. A., Abrams, L. M., & Smith, M. M. (2014). Preliminary psychometric properties of the BEST in CLASS adherence and competence scale. Journal of Emotional and Behavioral Disorders, 22(4), 249–259.Google Scholar
  59. Sutherland, K. S., McLeod, B. D., Conroy, M. A., & Cox, J. R. (2013). Measuring implementation of evidence-based programs targeting young children at risk for emotional/behavioral disorders: Conceptual issues and recommendations. Journal of Early Intervention, 35(2), 129–149. doi: 10.1177/1053815113515025.CrossRefGoogle Scholar
  60. Tofighi, D., & MacKinnon, D. P. (2011). RMediation: An R package for mediation analysis confidence intervals. Behavior Research Methods, 43(3), 692–700. doi: 10.3758/s13428-011-0076-x.PubMedPubMedCentralCrossRefGoogle Scholar
  61. Wanless, S. B., & Domitrovich, C. E. (2015). Readiness to implement school-based social–emotional learning interventions: Using research on factors related to implementation to maximize quality. Prevention Science, 16(8), 1037–1043. doi: 10.1007/s11121-015-0612-5.PubMedCrossRefGoogle Scholar
  62. Webb, C. A., DeRubeis, R. J., & Barber, J. P. (2010). Therapist adherence/competence and treatment outcome: A meta-analytic review. Journal of Consulting and Clinical Psychology, 78(2), 200–211. doi: 10.1037/a0018912.PubMedPubMedCentralCrossRefGoogle Scholar
  63. Webster-Stratton, C. (1997). From parent training to community building. Families in Society: The Journal of Contemporary Social Services, 78(2), 156–171. doi: 10.1606/1044-3894.755.CrossRefGoogle Scholar
  64. Webster-Stratton, C., Reid, M. J., & Stoolmiller, M. (2008). Preventing conduct problems and improving school readiness: Evaluation of the incredible years teacher and child training programs in high-risk schools. Journal of Child Psychology and Psychiatry, 49(5), 471–488. doi: 10.1111/j.1469-7610.2007.01861.x.PubMedCrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC 2017

Authors and Affiliations

  • Kevin S. Sutherland
    • 1
    Email author
  • Maureen A. Conroy
    • 2
  • Bryce D. McLeod
    • 1
  • James Algina
    • 2
  • Eleanor Wu
    • 1
  1. 1.Virginia Commonwealth UniversityRichmondUSA
  2. 2.University of FloridaGainesvilleUSA

Personalised recommendations