Prevention Science

, Volume 17, Issue 8, pp 981–991 | Cite as

What Constitutes High-Quality Implementation of SEL Programs? A Latent Class Analysis of Second Step® Implementation

  • Sabina Low
  • Keith Smolkowski
  • Clay Cook


With the increased number of schools adopting social-emotional learning (SEL) programming, there is increased emphasis on the role of implementation in obtaining desired outcomes. Despite this, the current knowledge of the active ingredients of SEL programming is lacking, and there is a need to move from a focus on “whether” implementation matters to “what” aspects of implementation matter. To address this gap, the current study utilizes a latent class approach with data from year 1 of a randomized controlled trial of Second Step® (61 schools, 321 teachers, over 7300 students). Latent classes of implementation were identified, then used to predict student outcomes. Teachers reported on multiple dimensions of implementation (adherence, dosage, competency), as well as student outcomes. Observational data were also used to assess classroom behavior (academic engagement and disruptive behavior). Results suggest that a three-class model fits the data best, labeled as high-quality, low-engagement, and low-adherence classes. Only the low-engagement class showed significant associations with poorer outcomes, when compared to the high-quality class (not the low-adherence class). Findings are discussed in terms of implications for program development and implementation science more broadly.


Implementation Social-emotional learning Fidelity Early intervention 


Compliance with Ethical Standards

Conflict of Interest

The authors declare that they have no conflict of interest.


All procedures were in accordance with the ethical standards of the institutional review boards and have been performed in accordance with the ethical standards as laid down in the 1964 Declaration of Helsinki and its later amendments or comparable ethical standards.


Informed consent was obtained from all individual participants included in the study.


This study was funded by Committee for Children, a non-profit organization in Seattle, WA.


  1. Aarons, G. A., & Sommerfeld, D. H. (2012). Leadership, Innovation Climate, and Attitudes Toward Evidence-Based Practice During a Statewide Implementation. Journal of American Academy of Child and Adolescent Psychiatry, 51, 423–431.CrossRefGoogle Scholar
  2. Berkel, C., Mauricio, A. M., Schoenfelder, E., & Sandler, I. N. (2011). Putting the pieces together: An integrated model of program implementation. Prevention Science, 12, 23–33.Google Scholar
  3. Bradshaw, C., Zmuda, J., Kellam, S., & Ialongo, N. (2009). Longitudinal impact of two universal preventive interventions in first grade on educational outcomes in high school. Journal of Educational Psychology, 101, 926–937. doi: 10.1037/a0016586.CrossRefPubMedPubMedCentralGoogle Scholar
  4. Breitenstein, S. M., Gross, D., et al. (2010). Implementation fidelity in community-based, Interventions. Research in Nursing & Health, 33, 164–173.Google Scholar
  5. Social and Character Development Research Consortium (2010). Efficacy of Schoolwide Programs to Promote Social and Character Development and Reduce Problem Behavior in Elementary School Children (NCER 2011–2001).Google Scholar
  6. Cook, C. R. (2009). The Proactive Classroom Management-Rating Form. Baton Rouge, LA: Louisiana State University.Google Scholar
  7. Cook, C. R., Frye, M., Slemrod, T., Lyon, A. R., Renshaw, T. L., & Zhang, Y. (2015). An integrated approach to universal prevention: Independent and combined effects of PBIS and SEL on youths’ mental health. School Psychology Quarterly, 30, 166.Google Scholar
  8. Carroll, C., Patterson, M., Wood, S., Booth, A., Rick, J., & Balain, S. (2007). A conceptual framework for implementation fidelity. Implementation Science, 2, 1–9.Google Scholar
  9. Denham, S. A., Bassett, H. H., & Zinsser, K. (2012). Early childhood teachers as socializers of young children’s emotional competence. Early Childhood Education Journal, 40, 137–143.CrossRefGoogle Scholar
  10. Domitrovich, C. E., Gest, S. D., Jones, D., Gill, S., & DeRousie, R. M. S. (2010). Implementation quality: Lessons learned in the context of the Head Start REDI trial. Early Childhood Research Quarterly, 25, 284–298.CrossRefPubMedPubMedCentralGoogle Scholar
  11. Domitrovich, C., Pas, E., Bradshaw, C., Becker, K., Keperling, J., Embry, D., et al. (2015). Individual and school organizational factors that influence implementation of the PAX good behavior game intervention. Prevention Science, 16, 1064–1074.Google Scholar
  12. Durlak, J. A., & DuPre, E. P. (2008). Implementation matters: A review of research on the influence of implementation on program outcomes and the factors affecting implementation. American Journal of Community Psychology, 41, 327–350.CrossRefPubMedGoogle Scholar
  13. Durlak, J. A., Weissberg, R. P., Dymnicki, A. B., Taylor, R. D., & Schellinger, K. B. (2011). The impact of enhancing students’ social and emotional learning: A meta‐analysis of school‐based universal interventions. Child Development, 82, 405–432.CrossRefPubMedGoogle Scholar
  14. Dusenbury, L., Brannigan, R., Falco, M., & Hansen, W. B. (2003). A review of research on fidelity of implementation: Implications for drug abuse prevention in school settings. Health Education Research, 18, 237–256.CrossRefPubMedGoogle Scholar
  15. Elias, M. J., Zins, J. E., Graczyk, P. A., & Weissberg, R. P. (2003). Implementation, sustainability, and scaling up of social-emotional and academic innovations in public schools. School Psychology Review, 32, 303–319.Google Scholar
  16. Fixsen, D. L., & Blase, K. A. (2009). Implementation: The missing link between research and practice. NIRN implementation brief, 1. Chapel Hill: The University of North Carolina, FPG, NIRN.Google Scholar
  17. Gager, P. J., & Elias, M. J. (1997). Implementing prevention programs in high-risk environments: Application of the resiliency paradigm. American Journal of Orthopsychiatry, 67, 363.CrossRefPubMedGoogle Scholar
  18. Gearing, R. E., El-Bassel, N., Ghesquiere, A., Baldwin, S., Gillies, J., & Ngeow, E. (2011). Major ingredients of fidelity: A review and scientific guide to improving quality of intervention research implementation. Clinical Psychology Review, 31, 79–88.CrossRefPubMedGoogle Scholar
  19. Glisson, C., & Schoenwald, S. K. (2005). The ARC organizational and community intervention strategy for implementing evidence-based children’s mental health treatments. Mental Health Services Research, 4, 243–259.CrossRefGoogle Scholar
  20. Goodman, R. (1997). The Strengths and Difficulties Questionnaire: A research note. Journal of Child Psychology and Psychiatry, 38, 581–586.CrossRefPubMedGoogle Scholar
  21. Goodman, R. (2001). Psychometric properties of the strengths and difficulties questionnaire. Journal of the American Academy of Child & Adolescent Psychiatry, 40, 1337–1345.CrossRefGoogle Scholar
  22. Goodman, R., & Scott, S. (1999). Comparing the Strengths and Difficulties Questionnaire and the Child Behavior Checklist: Is small beautiful? Journal of Abnormal Child Psychology, 27, 17–24.CrossRefPubMedGoogle Scholar
  23. Greenberg, M. T., Domitrovich, C. E., Graczyk, P. A., & Zins, J. E. (2005). The study of implementation in school-based preventive interventions: Theory, research, and practice. Promotion of Mental Health and Prevention of Mental and Behavioral Disorders 2005 Series V3Google Scholar
  24. Hamre, B. K., & Pianta, R. C. (2007). Learning opportunities in preschool and early elementary classroomsGoogle Scholar
  25. Hirschstein, M. K., Edstrom, L. V. S., Frey, K. S., Snell, J. L., & MacKenzie, E. P. (2007). Walking the talk in bullying prevention: Teacher implementation variables related to initial impact of the Steps to Respect program. School Psychology Review, 36, 3–21.Google Scholar
  26. Hulleman, C. S., Rimm-Kaufman, S. E., & Abry, T. D. S. (2013).Google Scholar
  27. January, A. M., Casey, R. J., & Paulson, D. (2011). A meta-analysis of classroom-wide interventions to build social skills: Do they work? School Psychology Review, 40, 242.Google Scholar
  28. Jennings, P. A., & Greenberg, M. T. (2009). The prosocial classroom: Teacher social and emotional competence in relation to student and classroom outcomes. Review of Educational Research, 79(1), 491–525.Google Scholar
  29. Kass, R. E., & Raftery, A. E. (1995). Bayes factors. Journal of the American Statistical Association, 90, 773–795.CrossRefGoogle Scholar
  30. Landis, J. R., & Koch, G. G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33(1), 159–174.CrossRefPubMedGoogle Scholar
  31. Lo, Y., Mendell, N., & Rubin, D. (2001). Testing the number of dimensions in a normal mixture. Biometrics, 88, 767–778.CrossRefGoogle Scholar
  32. Low, S., Van Ryzin, M. J., Brown, E. C., Smith, B. H., & Haggerty, K. P. (2014). Engagement matters: Lessons from assessing classroom implementation of steps to respect: A bullying prevention program over a one-year period. Prevention Science, 15, 165–176.CrossRefPubMedGoogle Scholar
  33. Low, S., Cook, C. R., Smolkowski, K., & Buntain-Ricklefs, J. (2015). Promoting social–emotional competence: An evaluation of the elementary version of Second Step®. Journal of School Psychology, 53, 463–477.Google Scholar
  34. Maas, C. J. M., & Hox, J. J. (2004a). The influence of violations of assumptions on multilevel parameter estimates and their standard errors. Computational Statistics and Data Analysis, 46, 427–440.Google Scholar
  35. Maas, C. J. M., & Hox, J. J. (2004b). Robustness issues in multilevel regression analysis. Statistica Neerlandica, 58, 127–137.Google Scholar
  36. McIntosh, K., Horner, R. H., & Sugai, G. (2009). Sustainability of systems-level evidence-based practices in schools: Current knowledge and future directions. In W. Sailor, G. Dunlap, G. Sugai, & R. Horner (Eds.), Handbook of positive behavior support (pp. 327–352). New York: Springer.Google Scholar
  37. Murray, D. M. (1998). Design and analysis of group-randomized trials (Vol. 29). Oxford University Press, USA.Google Scholar
  38. Murray, D. M., Hannan, P. J., Pals, S. P., McCowen, R. G., Baker, W. L., & Blitstein, J. L. (2006). A comparison of permutation and mixed-model regression methods for the analysis of simulated data in the context of a group-randomized trial. Statistics in Medicine, 25, 378–388.CrossRefGoogle Scholar
  39. Muthén, B., & Muthén, L. K. (2000). Integrating person‐centered and variable‐centered analyses: Growth mixture modeling with latent trajectory classes. Alcoholism: Clinical and Experimental Research, 24(6), 882–891.CrossRefGoogle Scholar
  40. Nickerson, A. B., & Fishman, C. (2009). Convergent and divergent validity of the Devereux Student Strengths Assessment. School Psychology Quarterly, 24, 48.CrossRefGoogle Scholar
  41. Nylund, K. L., Asparouhov, T., & Muthén, B. O. (2007). Deciding on the number of classes in latent class analysis and growth mixture modeling: A Monte Carlo simulation study. Structural Equation Modeling, 14, 535–569.CrossRefGoogle Scholar
  42. Osher, D., Sprague, J., Weissberg, R. P., Axelrod, J., Keenan, S., Kendziora, K., & Zins, J. E. (2008). A comprehensive approach to promoting social, emotional, and academic growth in contemporary schools. Best Practices in School Psychology, 4, 1263–1278.Google Scholar
  43. Proctor, E. K., Landsverk, J., Aarons, G., Chambers, D., Glisson, C., & Mittman, B. (2009). Implementation research in mental health services: An emerging science with conceptual, methodological, and training challenges. Administration and Policy in Mental Health and Mental Health Services Research, 36, 24–34.CrossRefPubMedGoogle Scholar
  44. Reyes, M. R., Brackett, M. A., Rivers, S. E., Elbertson, N. A., & Salovey, P. (2012). The interaction effects of program training, dosage, and implementation quality on targeted student outcomes for the RULER approach to social and emotional learning. School Psychology Review, 41, 82.Google Scholar
  45. Ringwalt, C. L., Ennett, S., Vincus, A., Thorne, J., Rohrbach, L. A., & Simons-Rudolph, A. (2002). The prevalence of effective substance use prevention curricula in U.S. middle schools. Prevention Science, 2, 257–265.CrossRefGoogle Scholar
  46. Ringwalt, C. L., Pankratz, M. M., Gottfredson, N., Jackson-Newsom, J., Dusenbury, L., Giles, S. M., Curry, D., & Hansen, W. (2009). The effects of students’ curriculum engagement, attitudes towards their teachers, and perception of their teachers’ skills on school-based prevention curriculum outcomes. Journal of Drug Education, 39, 223–237.CrossRefPubMedPubMedCentralGoogle Scholar
  47. Salmivalli, C. (2001). Peer-led intervention campaign against school bullying: Who benefited? Educational Research, 43, 263–278.Google Scholar
  48. Sanetti, L. M. H., & Kratochwill, T. R. (2009). Toward developing a science of treatment integrity: Introduction to the special series. School Psychology Review, 38, 445–459.Google Scholar
  49. SAS Institute. (2009). SAS/STAT® 9.2 user’s guide (2nd ed.). Cary, NC: SAS Institute, Inc. Retrieved March 10, 2010, from the SAS Product Documentation web site:
  50. Shapiro, E. S., & Kratochwill, T. R. (Eds.). (2000). Behavioral assessment in schools: Theory, research, and clinical foundations. Guilford PressGoogle Scholar
  51. Sheridan, S. M., Swanger-Gagné, M., Welch, G. W., Kwon, K., & Garbacz, S. A. (2009). Fidelity measurement in consultation: Psychometric issues and preliminary examination. School Psychology Review, 38(4), 476.Google Scholar
  52. Swanger-Gagné, M. S., Garbacz, S. A., & Sheridan, S. M. (2009). Intervention implementation integrity within conjoint behavioral consultation: Strategies for working with families. School Mental Health, 1, 131–142.CrossRefGoogle Scholar
  53. van Belle, G. (2008). Statistical rules of thumb (2nd ed.). New York: John Wiley & Sons.CrossRefGoogle Scholar
  54. Viera, A. J., & Garrett, J. M. (2005). Understanding interobserver agreement: The kappa statistic. Family Medicine, 37, 360–363.PubMedGoogle Scholar
  55. Wanless, S. B., & Domitrovich, C. E. (2015). Readiness to implement school-based social-emotional learning interventions: Using research on factors related to implementation to maximize quality. Prevention Science, 16, 1037–1043.Google Scholar

Copyright information

© Society for Prevention Research 2016

Authors and Affiliations

  1. 1.Arizona State UniversityTempeUSA
  2. 2.Oregon Research InstituteEugeneUSA
  3. 3.University of MinnesotaMinneapolisUSA

Personalised recommendations