Prevention Science

, Volume 19, Issue 2, pp 186–196 | Cite as

Factors Associated with Teacher Delivery of a Classroom-Based Tier 2 Prevention Program

  • Kevin S. SutherlandEmail author
  • Maureen A. Conroy
  • Bryce D. McLeod
  • James Algina
  • Rachel L. Kunemund


Teachers sometimes struggle to deliver evidence-based programs designed to prevent and ameliorate chronic problem behaviors of young children with integrity. Identifying factors associated with variations in the quantity and quality of delivery is thus an important goal for the field. This study investigated factors associated with teacher treatment integrity of BEST in CLASS, a tier-2 prevention program designed for young children at risk for developing emotional/behavioral disorders. Ninety-two early childhood teachers and 231 young children at-risk for emotional/behavioral disorders participated in the study. Latent growth curve analyses indicated that both adherence and competence of delivery increased across six observed time points. Results suggest that teacher education and initial levels of classroom quality may be important factors to consider when teachers deliver tier-2 (i.e., targeted to children who are not responsive to universal or tier-1 programming) prevention programs in early childhood settings. Teachers with higher levels of education delivered the program with more adherence and competence initially. Teachers with higher initial scores on the Emotional Support subscale of the Classroom Assessment Scoring System (CLASS) delivered the program with more competence initially and exhibited higher growth in both adherence and competence of delivery across time. Teachers with higher initial scores on the Classroom Organization subscale of the CLASS exhibited lower growth in adherence across time. Contrary to hypotheses, teacher self-efficacy did not predict adherence, and teachers who reported higher initial levels of Student Engagement self-efficacy exhibited lower growth in competence of delivery. Results are discussed in relation to teacher delivery of evidence-based programs in early childhood classrooms.


Early childhood Teacher delivery Problem behavior 



This research was supported by a grant (R324A110173) from the U. S. Department of Education, Institute for Education Sciences, with additional support from an NIH/NCATS Clinical and Translational Science Award to the University of Florida UL1 TR000064 and Grant H325H140001 from the U.S. Department of Education, Office of Special Education Programs. The opinions expressed by the authors are not necessarily reflective of the position of or endorsed by the U. S. Department of Education.

Compliance with Ethical Standards

Funding Information

This research was supported by a grant (R324A110173) from the US Department of Education, Institute for Education Sciences, with additional support from an NIH/NCATS Clinical and Translational Science Award to the University of Florida UL1 TR000064. The opinions expressed by the authors are not necessarily reflective of the position of or endorsed by the US Department of Education.

Conflict of Interest

The authors declare that they have no conflict of interest.

Ethical Approval

All study procedures involving human participants were in accordance with the ethical standards of the researchers’ Institutional Review Boards and with the 1964 Helsinki Declaration and its later amendments of comparable ethical standards.

Informed Consent

Informed consent was obtained from all individual participants included in the study.


  1. Achenbach, T. M., & Rescorla, L. A. (2000). CBCL/1, 5-5 & C-TRF/1, 5–5 profiles. Burlington: Research Center for Children, Youth, and Families.Google Scholar
  2. Baker, C. N., Kupersmidt, J. B., Voegler-Lee, M. E., Arnold, D., & Willoughby, M. T. (2010). Predicting teacher participation in a classroom-based, integrated preventive intervention for preschoolers. Early Childhood Research Quarterly, 25, 270–283. doi: 10.1016/j.ecresq.2009.09.005.CrossRefPubMedPubMedCentralGoogle Scholar
  3. Barber, J. P., Crits-Christoph, P., & Luborsky, L. (1996). Effects of therapist adherence and competence on patient outcome in brief dynamic therapy. Journal of Consulting and Clinical Psychology, 64, 619–622. doi: 10.1037/0022-006X.64.3.619.CrossRefPubMedGoogle Scholar
  4. Berlin, L. J., Brooks-Gunn, J., McCarton, C., & McCormick, M. C. (1998). The effectiveness of early intervention: Examining risk factors and pathways to enhanced development. Preventive Medicine, 27, 238–245. doi: 10.1006/pmed.1998.0282.CrossRefPubMedGoogle Scholar
  5. Bronfenbrenner, U. (1979). The ecology of human development: Experiments by nature and design. Cambridge: Harvard University Press.Google Scholar
  6. Bronfenbrenner, U. (2005). Making human beings human: Bioecological perspectives on human development. Thousand Oaks: Sage Publications.Google Scholar
  7. Bruhn, A.L., Lane, K.L., Hirsch, S.E. (2013). A Review of Tier 2 Interventions Conducted Within Multitiered Models of Behavioral Prevention. Journal of Emotional and Behavioral Disorders, 22, 171–189Google Scholar
  8. Carroll, K. M., & Nuro, K. F. (2002). One size cannot fit all: A stage model for psychotherapy manual development. Clinical Psychology: Science and Practice, 9, 396–406. doi: 10.1093/clipsy.9.4.396.Google Scholar
  9. Carroll, K. M., Nich, C., Sifty, R. L., Nuro, K. F., Frankforter, T. L., Ball, S. A., et al. (2000). A general system for evaluating therapist adherence and competence in psychotherapy research. Drug and Alcohol Dependence, 57, 225–238. doi: 10.1016/S0376-8716(99)00049-6.CrossRefPubMedGoogle Scholar
  10. Carter, A. S., Wagmiller, R. J., Gray, S. A., McCarthy, K. J., Horwitz, S. M., & Briggs-Gowan, M. J. (2010). Prevalence of DSM-IV disorder in a representative, healthy birth cohort at school entry: Sociodemographic risks and social adaptation. Journal of the American Academy of Child & Adolescent Psychiatry, 49, 686–698. doi: 10.1016/j.jaac.2010.03.018.Google Scholar
  11. Cicchetti, D. V. (1994). Guidelines, criteria, and rules of thumb for evaluating normed and standardized assessment instruments in psychology. Psychological Assessment, 6, 284–290. doi: 10.1037/1040-3590.6.4.284.CrossRefGoogle Scholar
  12. Conroy, M. A., Sutherland, K. S., Algina, J., Ladwig, C., Werch, B., Martinez, J., Jessee, G., & Gyure, M. (2017). BEST in CLASS: A professional development intervention fostering high quality classroom experiences for young children with problem behavior. Manuscript submitted for publication.Google Scholar
  13. Conroy, M. A., Sutherland, K. S., Algina, J. J., Wilson, R. E., Martinez, J. R., & Whalon, K. J. (2015). Measuring teacher implementation of the BEST in CLASS intervention program and corollary child outcomes. Journal of Emotional and Behavioral Disorders, 23(3), 144–155.CrossRefGoogle Scholar
  14. Conroy, M. A., Sutherland, K. S., Vo, A. K., Carr, S. E., & Ogston, P. (2014). Early childhood teachers’ use of effective instructional practices and the collateral effects on young children’s behavior. Journal of Positive Behavior Interventions, 16, 81–92. doi: 10.1177/1098300713478666.CrossRefGoogle Scholar
  15. Dane, A. V., & Schneider, B. H. (1998). Program integrity in primary and early secondary prevention: Are implementation effects out of control. Clinical Psychology Review, 18, 23–45. doi: 10.1016/S0272-7358(97)00043-3.CrossRefPubMedGoogle Scholar
  16. Domitrovich, C. E., Bradshaw, C. P., Poduska, J. M., Hoagwood, K., Buckley, J. A., Olin, S., Hunter Romanelli, L., Leaf, P. J., Greenberg, M. T., & Ialongo, N. S. (2008). Maximizing the implementation quality of evidence-based preventive interventions in schools: A conceptual framework. Advances in School Mental Health Promotion, 1, 6–28.CrossRefPubMedPubMedCentralGoogle Scholar
  17. Domitrovich, C. E., Cortes, R. C., & Greenberg, M. T. (2007). Improving young children’s social and emotional competence: A randomized trial of the preschool “PATHS” curriculum. The Journal of Primary Prevention, 28, 67–91. doi: 10.1007/s10935-007-0081-0.CrossRefPubMedGoogle Scholar
  18. Domitrovich, C. E., Gest, S. D., Jones, D., Gill, S., & DeRousie, R. M. S. (2010). Implementation quality: Lessons learned in the context of the head start REDI trial. Early Childhood Research Quarterly, 25, 284–298. doi: 10.1016/j.ecresq.2010.04.001.CrossRefPubMedPubMedCentralGoogle Scholar
  19. Domitrovich, C. E., Pas, E. T., Bradshaw, C. P., Becker, K. D., Keperling, J. P., Embry, D. D., & Ialongo, N. (2015). Individual and school organizational factors that influence implementation of the PAX Good Behavior Game Intervention. Prevention Science, 16(8), 1064–1074. doi: 10.1007/s11121-015-0557-8.CrossRefPubMedGoogle Scholar
  20. Durlak, J. A. (2015). Studying program implementation is not easy but it is essential. Prevention Science, 16(8), 1123–1127.CrossRefPubMedGoogle Scholar
  21. Durlak, J. A. (2010). The importance of doing well in whatever you do: A commentary on the special section, “Implementation research in early childhood education.”. Early Childhood Research Quarterly, 25, 348–357. doi: 10.1016/j.ecresq.2010.03.003.CrossRefGoogle Scholar
  22. Durlak, J. A., Weissberg, R. P., Dymnicki, A. B., Taylor, R. D., & Schellinger, K. B. (2011). The impact of enhancing students’ social and emotional learning: A meta-analysis of school-based universal interventions. Child Development, 82, 405–432. doi: 10.1111/j.1467-8624.2010.01564.x.CrossRefPubMedGoogle Scholar
  23. Feil, E. G., Severson, H. H., & Walker, H. M. (1998). Screening for emotional and behavioral delays: The Early Screening Project. Journal of Early Intervention, 21, 252–266. doi: 10.1177/105381519802100306.CrossRefGoogle Scholar
  24. Fixsen, D. L., Naoom, S. F., Blasé, K. A., Friedman, R. M., & Wallace, F. (2005). Implementation research: A synthesis of the literature. Tampa: University of South Florida, Louis de la Parte Florida Mental Health Institute, The National Implementation Research Network (FMHI Publication #231).Google Scholar
  25. Guo, Y., Connor, C. M., Yang, Y., Roehrig, A. D., & Morrison, F. J. (2012). The effects of teacher qualification, teacher self-efficacy, and classroom practices on fifth graders' literacy outcomes. The Elementary School Journal, 113, 3–24. doi: 10.1086/665816.CrossRefGoogle Scholar
  26. Han, S. S., & Weiss, B. (2005). Sustainability of teacher implementation of school-based mental health programs. Journal of Abnormal Child Psychology, 33, 665–679. doi: 10.1007/s10802-005-7646-2.CrossRefPubMedGoogle Scholar
  27. Hemmeter, M. L., Corso, R., & Cheatham, G. (2006). A national survey of early childhood educators: Training needs and strategies. Paper presented at the Conference on Research Innovations in Early Intervention, San Diego, CA.Google Scholar
  28. Hogue, A., Liddle, H. A., & Rowe, C. (1996). Treatment adherence process research in family therapy: A rationale and some practical guidelines. Psychotherapy: Theory, Research, Practice, Training, 33, 332–345. doi: 10.1037/0033-3204.33.2.332.CrossRefGoogle Scholar
  29. Hogue, A., Henderson, C. E., Dauber, S., Barajas, P. C., Fried A., Liddle, H. A. (2008) Treatment adherence, competence, and outcome in individual and family therapy for adolescent behavior problems. Journal of Consulting and Clinical Psychology, 76, 544–555Google Scholar
  30. Little, M. A., Sussman, S., Sun, P., & Rohrbach, L. A. (2013). The effects of implementation fidelity in the Towards No Drug Abuse dissemination trial. Health Education, 113, 281–296. doi: 10.1108/09654281311329231.CrossRefPubMedCentralGoogle Scholar
  31. McCabe, P. C., & Altamura, M. (2011). Empirically valid strategies to improve social and emotional competence of preschool children. Psychology in the Schools, 48, 513–540. doi: 10.1002/pits.20570.CrossRefGoogle Scholar
  32. McLeod, B. D., Southam-Gerow, M. A., Rodriguez, A., Quinoy, A., Arnold, C., Kendall, P. C., & Weisz, J., R. (2016). Development and initial psychometrics for a therapist competence instrument for CBT for youth with anxiety. Journal of Clinical and Adolescent Psychology. Advance online publication. doi: 10.1080/15374416.2016.1253018.
  33. McLeod, B. D., Southam-Gerow, M. A., Tully, C. B., Rodriguez, A., & Smith, M. M. (2013). Making a case for treatment integrity as a psychological treatment quality indicator. Clinical Psychology: Science and Practice, 20, 14–32. doi: 10.1111/cpsp.12020.Google Scholar
  34. Mendel, P., Meredith, L. S., Schoenbaum, M., Sherbourne, C. D., & Wells, K. B. (2008). Interventions in organizational and community context: A framework for building evidence on dissemination and implementation in health services research. Administration and Policy in Mental Health and Mental Health Services Research, 35(1–2), 21–37. doi: 10.1007/s10488-007-0144-9.CrossRefPubMedGoogle Scholar
  35. Mitchell, B. S., Stormont, M. A., & Gage, N. A. (2011). Tier 2 interventions within the context of school-wide positive behavior support. Behavioral Disorders, 36, 241–261.Google Scholar
  36. Muthén, L. K., & Muthén, B. O. (1998-2017). Mplus user’s guide (Eighth ed.). Los Angeles: Muthén & Muthén.Google Scholar
  37. Nelson, J. R., Stage, S., Duppong-Hurley, K., Synhorst, L., & Epstein, M. H. (2007). Risk factors predictive of the problem behavior of children at risk for emotional and behavioral disorders. Exceptional Children, 73, 367–379. doi: 10.1177/001440290707300306.CrossRefGoogle Scholar
  38. Newborg, J. (2005). Battelle developmental inventory, 2nd edition, examiner’s manual. Rolling Meadows: Riverside Publishing.Google Scholar
  39. Olejnik, S., & Algina, J. (2000). Measures of effect size for comparative studies: Applications, interpretations, and limitations. Contemporary Educational Psychology, 25, 241–286.CrossRefPubMedGoogle Scholar
  40. Pas, E. T., Bradshaw, C. P., & Hershfeldt, P. A. (2012). Teacher-and school-level predictors of teacher efficacy and burnout: Identifying potential areas for support. Journal of School Psychology, 50, 129–145. doi: 10.1016/j.jsp.2011.07.003.CrossRefPubMedGoogle Scholar
  41. Pas, E. T., Waasdorp, T. E., & Bradshaw, C. P. (2015). Examining contextual influences on classroom-based implementation of Positive Behavior Support strategies: Findings from a randomized controlled effectiveness trial. Prevention Science, 16(8), 1096–1106. doi: 10.1007/s11121-014-0492-0.CrossRefPubMedGoogle Scholar
  42. Pianta, R., Howes, C., Burchinal, M., Bryant, D., Clifford, R., Early, D., & Barbarin, O. (2005). Features of pre-kindergarten programs, classrooms, and teachers: Do they predict observed classroom quality and child-teacher interactions? Applied Developmental Science, 9(3), 144–159. doi: 10.1207/s1532480xads0903_2.CrossRefGoogle Scholar
  43. Pianta, R. C., Karen, M., Paro, L., & Hamre, B. K. (2008). Classroom assessment scoring system (CLASS) manual, pre-K. Baltimore: Paul H. Brookes Publishing Company.Google Scholar
  44. Proctor, E., Silmere, H., Raghavan, R., Hovmand, P., Aarons, G., Bunger, A., Griffey, R., & Hensley, M. (2011). Outcomes for implementation research: Conceptual distinctions, measurement challenges, and research agenda. Administration and Policy in Mental Health, 38, 65–76.CrossRefPubMedGoogle Scholar
  45. Sameroff, A. J. (1995). General systems theories and developmental psychopathology. In E. D. Cicchetti & D. J. Cohen (Eds.), Developmental Psychopathology, Vol. 1: Theory and Methods. Oxford: John Wiley & Sons.Google Scholar
  46. Sanetti, L. M. H., & Kratochwill, T. R. (2009). Toward developing a science of treatment integrity: Introduction to the special series. School Psychology Review, 38, 445–459.Google Scholar
  47. Schoenwald, S. K., Garland, A. F., Chapman, J. E., Frazier, S. L., Sheidow, A. J., & Southam-Gerow, M. A. (2011). Toward the effective and efficient measurement of implementation fidelity. Administration and Policy in Mental Health and Mental Health Services Research, 38, 32–43. doi: 10.1007/s10488-010-0321-0.CrossRefPubMedPubMedCentralGoogle Scholar
  48. Sutherland, K. S., Conroy, M. A., Algina, J., Ladwig, C., Jessee, G., & Gyure, M. (2017). Reducing child problem behaviors and improving teacher-child interactions and relationships: A randomized controlled trial of BEST in CLASS. Early Childhood Research Quarterly. in press.Google Scholar
  49. Sutherland, K. S., Conroy, M. A., Vo, A., & Ladwig, C. (2015). Implementation integrity of practice-based coaching: Preliminary results from the BEST in CLASS efficacy trial. School Mental Health, 7, 21–33. doi: 10.1007/s12310-014-9134-8.CrossRefGoogle Scholar
  50. Sutherland, K. S., McLeod, B. D., Conroy, M. A., Abrams, L., & Smith, M. M. (2014). Preliminary psychometric properties of the BEST in CLASS Adherence and Competence Scale. Journal of Emotional and Behavioral Disorders, 22, 249–259. doi: 10.1177/1063426613493669.CrossRefGoogle Scholar
  51. Sutherland, K. S., McLeod, B. D., Conroy, M. A., & Cox, J. R. (2013). Measuring implementation of evidence-based programs targeting young children at risk for emotional/behavioral disorders conceptual issues and recommendations. Journal of Early Intervention, 35, 129–149. doi: 10.1177/1053815113515025.CrossRefGoogle Scholar
  52. Tschannen-Moran, M., & Hoy, A. W. (2001). Teacher efficacy: Capturing an elusive construct. Teaching and Teacher Education, 17, 783–805. doi: 10.1016/S0742-051X(01)00036-1.CrossRefGoogle Scholar
  53. Wanless, S. B., Rimm-Kaufman, S. E., Abry, T., Larsen, R. A., & Patton, C. L. (2015). Engagement in training as a mechanism to understanding fidelity of implementation of the Responsive Classroom Approach. Prevention Science, 16(8), 1107–1116.CrossRefPubMedGoogle Scholar
  54. Webster-Stratton, C., Reid, M. J., & Hammond, M. (2004). Treating children with early-conduct problems: Intervention outcomes for parent, child, and teacher training. Journal of Clinical Child and Adolescent Psychology, 33, 105–124. doi: 10.1207/S15374424JCCP3301_11.CrossRefPubMedGoogle Scholar
  55. Williford, A. P., Wolcott, C. S., Whittaker, J. V., & Locasale-Crouch, J. (2015). Program and teacher characteristics predicting the implementation of banking time with preschoolers who display disruptive behaviors. Prevention Science, 16, 1054–1063. doi: 10.1007/s11121-015-0544-0.CrossRefPubMedGoogle Scholar
  56. Wolery, M. (2011). Intervention research: The importance of fidelity measurement. Topics in Early Childhood Special Education, 31, 155–157. doi: 10.1177/0271121411408621.CrossRefGoogle Scholar

Copyright information

© Society for Prevention Research 2017

Authors and Affiliations

  • Kevin S. Sutherland
    • 1
    Email author
  • Maureen A. Conroy
    • 2
  • Bryce D. McLeod
    • 1
  • James Algina
    • 2
  • Rachel L. Kunemund
    • 1
  1. 1.Virginia Commonwealth UniversityRichmondUSA
  2. 2.University of FloridaGainesvilleUSA

Personalised recommendations