Teacher Competence of Delivery of BEST in CLASS as a Mediator of Treatment Effects
This study investigated if training and practice-based coaching in an evidence-based program was associated with higher observed treatment integrity (adherence and competence) and if these treatment integrity components were associated with teacher report of child behavioral outcomes in the BEST in CLASS efficacy trial. Participants were 462 children (M = 4.32 years, SD = 0.53; 65% male; 17.0% Caucasian, 66.0% African-American, 5.0% Hispanic, and 12.0% other) identified as having problem behavior and their 185 teachers (M = 12.09 years teaching experience; 99% female; 47.0% Caucasian, 48.0% African-American, 1.0% Asian/Pacific Islander, 3.0% Hispanic and 1.0% other). Teachers and focal children were randomly assigned to the intervention (teacher n = 92, children n = 230) or control condition (teacher n = 93, child n = 232). Results of a multilevel mediation analysis indicated that the BEST in CLASS intervention had a positive effect on teacher report of child problem behavior (SSIS-RS) and externalizing problems (C-TRF), as well as having a positive effect on teachers’ adherence and competence of delivery of the intervention. There was an indirect effect through competence of delivery for externalizing problems, but not problem behavior. No indirect effects for adherence were found. Implications of these findings and directions for future research are discussed.
KeywordsTreatment integrity Teacher delivery Problem behavior Evidence-based program
This research was supported by a Grant (R324A110173) from the U. S. Department of Education, Institute for Education Sciences, with additional support from an NIH/NCATS Clinical and Translational Science Award to the University of Florida UL1 TR000064. The opinions expressed by the authors are not necessarily reflective of the position of or endorsed by the U. S. Department of Education.
Compliance with Ethical Standards
Conflict of interest
The authors declare that they have no conflict of interest.
All study procedures involving human participants were in accordance with the ethical standards of the researchers’ Institutional Review Boards and with the 1964 Helsinki declaration and its later amendments of comparable ethical standards.
Informed consent was obtained from all individual participants included in the study.
- Abry, T., Rimm-Kaufman, S. E., Larsen, R. A., & Brewer, A. J. (2013). The influence of fidelity of implementation on teacher–student interaction quality in the context of a randomized controlled trial of the Responsive Classroom approach. Journal of School Psychology, 51(4), 437–453. doi: 10.1016/j.jsp.2013.03.001.PubMedCrossRefGoogle Scholar
- Achenbach, T. M., & Rescorla, L. A. (2000). ASEBA preschool forms & profiles. Burlington, VT: University of Vermont, Research Center for Children, Youth and Families.Google Scholar
- Berry, V., Axford, N., Blower, S., Taylor, R. S., Edwards, R. T., Tobin, K., et al. (2016). The effectiveness and micro-costing analysis of a universal, school-based, social–emotional learning programme in the UK: A cluster-randomised controlled trial. School Mental Health, 8(2), 238–256. doi: 10.1007/s12310-015-9160-1.CrossRefGoogle Scholar
- Bierman, K. L., Coie, J., Dodge, K., Greenberg, M., Lochman, J., McMohan, R., et al. (2013). School outcomes of aggressive–disruptive children: Prediction from kindergarten risk factors and impact of the fast track prevention program. Aggressive Behavior, 39(2), 114–130. doi: 10.1002/ab.21467.PubMedCrossRefGoogle Scholar
- Brennan, E. M., Bradley, J. R., Allen, M. D., & Perry, D. F. (2008). The evidence base for mental health consultation in early childhood settings: Research synthesis addressing staff and program outcomes. Early Education and Development, 19(6), 982–1022. doi: 10.1080/10409280801975834.CrossRefGoogle Scholar
- Bulotsky-Shearer, R. J., Bell, E. R., & Domínguez, X. (2012). Latent profiles of problem behavior within learning, peer, and teacher contexts: Identifying subgroups of children at academic risk across the preschool year. Journal of School Psychology, 50(6), 775–798. doi: 10.1016/j.jsp.2012.08.001.PubMedCrossRefGoogle Scholar
- Carroll, K. M., Nich, C., Sifry, R. L., Nuro, K. F., Frankforter, T. L., Ball, S. A., et al. (2000). A general system for evaluating therapist adherence and competence in psychotherapy research in the addictions. Drug and Alcohol Dependence, 57(3), 225–238. doi: 10.1016/S0376-8716(99)00049-6.PubMedCrossRefGoogle Scholar
- Conroy, M. A., & Sutherland, K. S. (2011). Promoting social, emotional, and behavioral competence in young high-risk children: A preventative classroom-based early intervention model. Unpublished manual, Virginia Commonwealth University, Richmond, VA.Google Scholar
- Conroy, M. A., Sutherland, K. S., Algina, J. J., Wilson, R. E., Martinez, J. R., & Whalon, K. J. (2015). Measuring teacher implementation of the intervention program and corollary child outcomes. Journal of Emotional and Behavioral Disorders, 23(3), 144–155.Google Scholar
- Conroy, M. A., Sutherland, K. S., Vo, A. K., Carr, S., & Ogston, P. L. (2014). Early childhood teachers’ use of effective instructional practices and the collateral effects on young children’s behavior. Journal of Positive Behavior Interventions, 16(2), 81–92. doi: 10.1177/1098300713478666.
- Crits-Christoph, P., Siqueland, L., Chittams, J., Barber, J. P., Beck, A. T., Frank, A., et al. (1998). Training in cognitive, supportive-expressive, and drug counseling therapies for cocaine dependence. Journal of Consulting and Clinical Psychology, 66(3), 484–492. doi: 10.1037//0022-006X.66.3.484.PubMedCrossRefGoogle Scholar
- Doumen, S., Verschueren, K., Buyse, E., Germeijs, V., Luyckx, K., & Soenens, B. (2008). Reciprocal relations between teacher–child conflict and aggressive behavior in kindergarten: A three-wave longitudinal study. Journal of Clinical Child & Adolescent Psychology, 37(3), 588–599. doi: 10.1016/j.jsp.2007.06.009.CrossRefGoogle Scholar
- Durlak, J. A., Weissberg, R. P., Dymnicki, A. B., Taylor, R. D., & Schellinger, K. B. (2011). The impact of enhancing students’ social and emotional learning: A meta-analysis of school-based universal interventions. Child Development, 82(1), 405–432. doi: 10.1111/j.1467-8624.2010.01564.x.PubMedCrossRefGoogle Scholar
- Feil, E. G., Small, J. W., Forness, S. R., Serna, L. A., Kaiser, A. P., Hancock, T. B., et al. (2005). Using different measures, informants, and clinical cut-off points to estimate prevalence of emotional or behavioral disorders in preschoolers: Effects on age, gender, and ethnicity. Behavioral Disorders, 30(4), 375–391.CrossRefGoogle Scholar
- Fox, L., Hemmeter, M. L., Snyder, P., Binder, D. P., & Clarke, S. (2011). Coaching early childhood special educators to implement a comprehensive model for promoting young children’s social competence. Topics in Early Childhood Special Education, 31(3), 178–192. doi: 10.1177/0271121411404440.CrossRefGoogle Scholar
- Gilliam, W. S., & Marchesseault, C. M. (2005). From capitols to classrooms, policy to practice: State funded Pre-k at the classroom level. New Haven, CT: Yale University Child Study Center.Google Scholar
- Goncy, E. A., Sutherland, K. S., Farrell, A. D., Sullivan, T. N., & Doyle, S. T. (2015). Measuring teacher implementation in delivery of a bullying prevention program: The impact of instructional and procedural adherence and competence on student responsiveness. Prevention Science, 16(3), 440–450. doi: 10.1007/s11121-014-0508-9.PubMedCrossRefGoogle Scholar
- Gresham, F., & Elliott, S. N. (2008). Social skills improvement system (SSIS) rating scales. Bloomington, MN: Pearson Assessments.Google Scholar
- Hamre, B. K., Justice, L. M., Pianta, R. C., Kilday, C., Sweeney, B., Downer, J. T., et al. (2010). Implementation fidelity of My Teaching Partner literacy and language activities: Association with preschoolers’ language and literacy growth. Early Childhood Research Quarterly, 25(3), 329–347. doi: 10.1016/j.ecresq.2009.07.002.CrossRefGoogle Scholar
- Hogue, A., Henderson, C. E., Dauber, S., Barajas, P. C., Fried, A., & Liddle, H. A. (2008). Treatment adherence, competence, and outcome in individual and family therapy for adolescent behavior problems. Journal of Consulting and Clinical Psychology, 76(4), 544–555. doi: 10.1037/0022-006X.76.4.544.PubMedPubMedCentralCrossRefGoogle Scholar
- McLeod, B. D., Southam-Gerow, M. A., Rodriguez, A., Quinoy, A., Arnold, C., Kendall, P. C., et al. (2016). Development and initial psychometrics for a therapist competence instrument for CBT for youth anxiety. Journal of Clinical Child and Adolescent Psychology. doi: 10.1080/15374416.2016.1253018.PubMedGoogle Scholar
- Mendel, P., Meredith, L. S., Schoenbaum, M., Sherbourne, C. D., & Wells, K. B. (2008). Interventions in organizational and community context: A framework for building evidence on dissemination and implementation in health services research. Administration and Policy in Mental Health and Mental Health Services Research, 35(1–2), 21–37. doi: 10.1007/s10488-007-0144-9.PubMedCrossRefGoogle Scholar
- Muthén, L. K. & Muthén, B. O. (1998–2015). Mplus user’s guide (7th ed.). Los Angeles, CA: Muthén & Muthén.Google Scholar
- Newborg, J. (2005). Battelle developmental inventory, Second edition: Examiner’s manual. Itasca, IL: Riverside.Google Scholar
- Proctor, E., Silmere, H., Raghavan, R., Hovmand, P., Aarons, G., Bunger, A., et al. (2011). Outcomes for implementation research: Conceptual distinctions, measurement challenges, and research agenda. Administration and Policy in Mental Health and Mental Health Services Research, 38(2), 65–76. doi: 10.1007/s10488-010-0319-7.PubMedCrossRefGoogle Scholar
- Raudenbush, S. W., & Bryk, A. S. (2002). Hierarchical linear models: Applications and data analysis methods (Vol. 1). Thousand Oaks: Sage.Google Scholar
- Roorda, D. L., Verschueren, K., Vancraeyveldt, C., Van Craeyevelt, S., & Colpin, H. (2014). Teacher–child relationships and behavioral adjustment: Transactional links for preschool boys at risk. Journal of School Psychology, 52(5), 495–510. doi: 10.1016/j.jsp.2014.06.004.PubMedCrossRefGoogle Scholar
- Schoenwald, S. K. (2012). The transport and diffusion of multisystemic therapy. In Dissemination and implementation of evidence-based psychological interventions (pp. 227–246). doi: 10.1093/med:psych/9780195389050.003.0012.
- Schoenwald, S. K., Garland, A. F., Chapman, J. E., Frazier, S. L., Sheidow, A. J., & Southam-Gerow, M. A. (2011). Toward the effective and efficient measurement of implementation fidelity. Administration and Policy in Mental Health and Mental Health Services Research, 38(1), 32–43. doi: 10.1007/s10488-010-0321-0.PubMedCrossRefGoogle Scholar
- Simons, A. D., Padesky, C. A., Montemarano, J., Lewis, C. C., Murakami, J., Lamb, K., et al. (2010). Training and dissemination of cognitive behavior therapy for depression in adults: A preliminary examination of therapist competence and client outcomes. Journal of Consulting and Clinical Psychology, 78(5), 751. doi: 10.1037/a0020569.PubMedCrossRefGoogle Scholar
- Sutherland, K. S., Conroy, M. A., Algina, J., Ladwig, C., Jessee, G., & Gyure, M. (2017). Reducing child problem behaviors and improving teacher-child interactions and relationships: A randomized controlled trial of BEST in CLASS. Manuscript submitted for publication. Google Scholar
- Sutherland, K. S., Conroy, M. A., Vo, A., & Ladwig, C. (2015). Implementation integrity of practice-based coaching: Preliminary results from the BEST in CLASS efficacy trial. School Mental Health, 7(1), 21–33.Google Scholar
- Sutherland, K. S., McLeod, B. D., Conroy, M. A., Abrams, L. M., & Smith, M. M. (2014). Preliminary psychometric properties of the BEST in CLASS adherence and competence scale. Journal of Emotional and Behavioral Disorders, 22(4), 249–259.Google Scholar
- Sutherland, K. S., McLeod, B. D., Conroy, M. A., & Cox, J. R. (2013). Measuring implementation of evidence-based programs targeting young children at risk for emotional/behavioral disorders: Conceptual issues and recommendations. Journal of Early Intervention, 35(2), 129–149. doi: 10.1177/1053815113515025.CrossRefGoogle Scholar
- Webster-Stratton, C., Reid, M. J., & Stoolmiller, M. (2008). Preventing conduct problems and improving school readiness: Evaluation of the incredible years teacher and child training programs in high-risk schools. Journal of Child Psychology and Psychiatry, 49(5), 471–488. doi: 10.1111/j.1469-7610.2007.01861.x.PubMedCrossRefGoogle Scholar