Skip to main content
SpringerLink
Log in

The Impact of Implementation Fidelity on Student Outcomes in the Life Skills Training Program

  • Original Paper
  • Published:
School Mental Health Aims and scope Submit manuscript

Abstract

Social and emotional learning (SEL) programs have become increasingly popular during the last 20 years. Although the benefits of SEL programs are significant, the issue of implementation fidelity often arises. The purpose of this study was to assess the impact of implementation fidelity in the Life Skills Training (LST) program implemented with middle school students in a large Florida school district. Three core elements of implementation fidelity were assessed including: (1) adherence; (2) participant responsiveness; and (3) quality of delivery. Student survey data were collected from 4812 students and 104 classrooms in 16 middle schools that participated in the LST program. Multilevel modeling was used to assess the effect of individual-level [gender, race/ethnicity, and socioeconomic status (SES)] and classroom-level characteristics (adherence, participant responsiveness, and quality of delivery) on students’ SEL outcomes measured at posttest. At the individual level, results indicated that students’ race/ethnicity and SES were significantly associated with predicting student SEL outcomes at posttest. At the classroom level, participant responsiveness was significantly associated with predicting student SEL skills at posttest. Findings are discussed in terms of implications for research and practice.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • Aber, J. L., Brown, J. L., & Jones, S. M. (2003). Developmental trajectories toward violence in middle childhood: Course, demographic differences, and response to school-based intervention. Developmental Psychology,39(2), 324–348.

    PubMed  Google Scholar 

  • Allen, J. D., Linnan, L. A., & Emmons, K. M. (2012). Fidelity and its relationship to implementation effectiveness, adaptation, and dissemination. In R. C. Brownson, G. A. Colditz, & E. K. Proctor (Eds.), Dissemination and implementation research in health: Translating science to practice (pp. 281–305). New York: Oxford University Press.

    Chapter  Google Scholar 

  • Allen, J. D., Shelton, R. C., Emmons, K. M., & Linnan, L. A. (2018). Fidelity and its relationship to implementation effectiveness, adaptation, and dissemination. In R. C. Brownson, G. A. Colditz, & E. K. Proctor (Eds.), Dissemination and implementation research in health: Translating science to practice (2nd ed., pp. 267–284). New York: Oxford University Press.

    Google Scholar 

  • Arbuckle, J. (1996). Full information estimation in the presence of incomplete data. In G. Marcoulides & R. Schumacker (Eds.), Advanced structural equation modeling: Issues and techniques (pp. 243–277). New York: Psychology Press.

    Google Scholar 

  • Bandura, A., Adams, N. E., & Beyer, J. (1977). Cognitive processes mediating behavioral change. Journal of Personality and Social Psychology,35(3), 125–139.

    Article  PubMed  Google Scholar 

  • Berkel, C., Mauricio, A. M., Schoenfelder, E., & Sandler, I. N. (2011). Putting the pieces together: An integrated model of program implementation. Prevention Science,12(1), 23–33.

    Article  PubMed  PubMed Central  Google Scholar 

  • Blueprints for Healthy Youth Development [BHYD]. (n.d.). LifeSkills Training (LST). Retrieved November 8, 2018, from http://www.blueprintsprograms.com/factsheet/lifeskills-training-lst.

  • Botvin, G. J., Baker, E., Dusenbury, L., Botvin, E. M., & Diaz, T. (1995). Long-term follow-up results of a randomized drug abuse prevention trial in a white middle-class population. Journal of the American Medical Association,273(14), 1106–1112.

    Article  PubMed  Google Scholar 

  • Botvin, G. J., Baker, E., Dusenbury, L., Tortu, S., & Botvin, E. M. (1990). Preventing adolescent drug abuse through a multimodal cognitive-behavioral approach: Results of a 3-year study. Journal of Consulting and Clinical Psychology,58(4), 437–446.

    Article  PubMed  Google Scholar 

  • Botvin, G. J., Batson, H. W., Witts-Vitale, S., Bess, V., Baker, E., & Dusenbury, D. (1989). A psychological approach to smoking prevention for urban black youth. Public Health Reports,104(6), 573–582.

    PubMed  PubMed Central  Google Scholar 

  • Botvin, G. J., Epstein, J. A., Baker, E., Diaz, T., & Ifill-Williams, M. (1997). School-based drug abuse prevention with inner-city minority youth. Journal of Child and Adolescent Substance Abuse,6(1), 5–19.

    Article  Google Scholar 

  • Botvin, G. J., & Griffin, K. W. (2004). Life skills training: Empirical findings and future directions. Journal of Primary Prevention,25(4), 211–232.

    Article  Google Scholar 

  • Botvin, G. J., & Griffin, K. W. (2015). Life skills training: A competence enhancement approach to tobacco, alcohol, and drug abuse prevention. In L. M. Scheier (Ed.), Handbook of adolescent drug use prevention: Research, intervention strategies, and practice (pp. 177–196). Washington, DC: American Psychological Association.

    Chapter  Google Scholar 

  • Botvin, G. J., Griffin, K. W., Diaz, T., & Ifill-Williams, M. (2001a). Drug abuse prevention among monitory adolescents: Posttest and one-year follow-up of a school based preventive intervention. Prevention Science,2(1), 1–13.

    Article  PubMed  Google Scholar 

  • Botvin, G. J., Griffin, K. W., Diaz, T., & Ifill-Williams, M. (2001b). Preventing binge drinking during early adolescence: One- and two-year follow-up of a school-based preventive intervention. Psychology of Addictive Behaviors,15(4), 360–365.

    Article  PubMed  Google Scholar 

  • Botvin, G. J., Griffin, K. W., & Nichols, T. D. (2006). Preventing youth violence and delinquency through a universal school-based prevention approach. Prevention Science,7(4), 403–408.

    Article  PubMed  Google Scholar 

  • Botvin, G. J., Schinke, S. P., Epstein, J. A., & Diaz, T. (1994). Effectiveness of culturally focused and generic skills training approaches to alcohol and drug abuse prevention among minority youths. Psychology of Addictive Behaviors,8(2), 116–127.

    Article  Google Scholar 

  • Botvin, G. J., Schinke, S. P., Epstein, J. A., Diaz, T., & Botvin, E. M. (1995). Effectiveness of culturally focused and generic skills training approaches to alcohol and drug abuse prevention among minority adolescents: Two-year follow-up results. Psychology of Addictive Behaviors,9(3), 183–194.

    Article  Google Scholar 

  • Botvin LifeSkills® Training. (n.d.). LST fidelity checklists. Retrieved October 11, 2018, from https://www.lifeskillstraining.com/lst-fidelity-checklists/.

  • Brackett, M. A., Elbertson, N. A., & Rivers, S. E. (2015). Applying theory to the development of approaches to SEL. In J. A. Durlak, C. E. Domitrovich, R. P. Weissberg, & T. P. Gullotta (Eds.), Handbook of social and emotional learning: Research and practice (pp. 20–32). New York: The Guilford Press.

    Google Scholar 

  • Carroll, C., Patterson, M., Wood, S., Booth, A., Rick, J., & Balain, S. (2007). A conceptual framework for implementation fidelity. Implementation Science,2(40), 1–9.

    Google Scholar 

  • Castro-Olivo, S. M. (2014). Promoting social-emotional learning in adolescent Latino ELLs: A study of the culturally adapted Strong Teens program. School Psychology Quarterly,29(4), 567–577.

    Article  PubMed  Google Scholar 

  • Collaborative for Academic, Social, and Emotional Learning. (n.d.). What is SEL? Retrieved October 1, 2018, from http://www.casel.org/what-is-sel/.

  • Damschroder, L. J., Aron, D. C., Keith, R. E., Kirsh, S. R., Alexander, J. A., & Lowery, J. C. (2009). Fostering implementation of health services research finding into practice: A consolidated framework for advancing implementation science. Implementation Science,4(50), 1–15.

    Google Scholar 

  • Dane, A. V., & Schneider, B. H. (1998). Program integrity in primary and early secondary prevention: Are implementation effects out of control?. Clinical Psychology Review,18(1), 23–45.

    Article  PubMed  Google Scholar 

  • Dillman Carpentier, F., Mauricio, A. M., Gonzales, N. A., Millsap, R. E., Meza, C. M., Dumka, L. E., et al. (2007). Engaging Mexican origin families in a school-based preventive intervention. Journal of Primary Prevention,28(6), 521–546.

    Article  PubMed  Google Scholar 

  • Domitrovich, C. E., Gest, S. D., Jones, D., Gill, S., & Sanford DeRousie, R. M. (2010). Implementation quality: Lessons learned in the context of the head start REDI trial. Early Childhood Research Quarterly,25(3), 284–298.

    Article  PubMed  PubMed Central  Google Scholar 

  • Domitrovich, C. E., Pas, E. T., Bradshaw, C. P., Becker, K. D., Keperling, J. P., Embry, D. D., et al. (2015). Individual and school organizational factors that influence implementation of the PAX good behavior game intervention. Prevention Science,16(8), 1064–1074.

    Article  PubMed  Google Scholar 

  • Durlak, J. A. (2015). What everyone should know about implementation. In J. A. Durlak, C. E. Domitrovich, R. P. Weissberg, & T. P. Gullotta (Eds.), Handbook of social and emotional learning: Research and practice (pp. 395–405). New York: The Guilford Press.

    Google Scholar 

  • Durlak, J. A. (2016). Programme implementation in social and emotional learning: Basic issues and research findings. Cambridge Journal of Education,46(3), 333–345.

    Article  Google Scholar 

  • Durlak, J. A., & DuPre, E. P. (2008). Implementation matters: A review of research on the influence of implementation on program outcomes and the factors affecting implementation. American Journal of Community Psychology,41(3–4), 327–350.

    Article  PubMed  Google Scholar 

  • Durlak, J. A., Weissberg, R. P., Dymnicki, A. B., Taylor, R. D., & Schellinger, K. B. (2011). The impact of enhancing students’ social and emotional learning: A meta-analysis of school-based universal interventions. Child Development,82(1), 405–432.

    Article  PubMed  Google Scholar 

  • Dusenbury, L., Brannigan, R., Falco, M., & Hansen, W. B. (2003). A review of research on fidelity of implementation: Implications for drug abuse prevention in school settings. Health Education Research,18(2), 237–256.

    Article  PubMed  Google Scholar 

  • Fixsen, D. L., & Blase, K. A. (n.d.) Implementation and system change [PowerPoint Presentation]. Chapel Hill, NC: University of North Carolina, National Implementation Research Network, FPG Child Development Institute.

  • Garner, P. W., Mahatmya, D., Brown, E. L., & Vesley, C. K. (2014). Promoting desirable outcomes among culturally and ethnically diverse children in social emotional learning programs: A multilevel heuristic model. Educational Psychology Review,26(1), 165–189.

    Article  Google Scholar 

  • Goncy, E. A., Sutherland, K. S., Farrell, A. D., Sullivan, T. N., & Doyle, S. T. (2015). Measuring teacher implementation in delivery of a bullying prevention program: The impact of instructional and procedural adherence and competence on student responsiveness. Prevention Science,16(3), 440–450.

    Article  PubMed  Google Scholar 

  • Graham, J. W. (2012). Missing data: Analysis and design. New York: Springer.

    Book  Google Scholar 

  • Greenberg, M. T. (2004). Current and future challenges in school-based prevention: The researcher perspective. Prevention Science,5(1), 5–13.

    Article  PubMed  Google Scholar 

  • Greenberg, M. T. (2010). School-based prevention: Current status and future challenges. Effective Education,2(1), 27–52.

    Article  Google Scholar 

  • Hahn, E. J., Noland, M. P., Rayens, M. K., & Christie, D. M. (2002). Efficacy of training and fidelity of implementation of the life skills training program. Journal of School Health,72(7), 282–287.

    Article  PubMed  Google Scholar 

  • Humphrey, N., Barlow, A., & Lendrum, A. (2018). Quality matters: Implementation moderates student outcomes in the PATHS curriculum. Prevention Science,19(2), 197–208.

    Article  PubMed  Google Scholar 

  • Jagers, R. J., Harris, A., & Skoog, A. (2015). A review of classroom-based SEL programs at the middle school level. In J. A. Durlak, C. E. Domitrovich, R. P. Weissberg, & T. P. Gullotta (Eds.), Handbook of social and emotional learning: Research and practice (pp. 167–180). New York: The Guilford Press.

    Google Scholar 

  • Jia, Y., Konold, T. R., Cornell, D., & Huang, F. (2018). The impact of validity screening on associations between self-reports of bullying victimization and student outcomes. Educational and Psychological Measurement,78(1), 80–102.

    Article  PubMed  Google Scholar 

  • Klapp, A., Belfield, C., Bowden, B., Levin, H., Shand, R., & Zander, S. (2017). A benefit-cost analysis of a long-term intervention on social and emotional learning in compulsory school. The International Journal of Emotional Education,9(1), 3–19.

    Google Scholar 

  • Low, S., Cook, C., Smolkowski, K., & Burtain-Ricklefs, J. (2015). Promoting social-emotional competence: An evaluation of the elementary version of Second Step®. Journal of School Psychology,53(6), 463–477.

    Article  PubMed  Google Scholar 

  • Low, S., Smolkowski, K., & Cook, C. (2016). What constitutes high-quality implementation of SEL programs? A latent class analysis of Second Step® implementation. Prevention Science,17(8), 981–991.

    Article  PubMed  Google Scholar 

  • Meyers, A. B., Tobin, R. M., Huber, B. J., Conway, D. E., & Shelvin, K. H. (2015). Interdisciplinary collaboration supporting social-emotional learning in rural school systems. Journal of Educational and Psychological Consultation,25(2–3), 109–128.

    Article  Google Scholar 

  • Mihalic, S. (2002). The importance of implementation fidelity. Boulder, CO: University of Colorado, Blueprints Violence Prevention Initiative, Center for the Study and Prevention of Violence. Retrieved from http://www.incredibleyears.com/wp-content/uploads/fidelity-importance.pdf.

  • Mihalic, S. F., & Elliot, D. S. (2015). Evidence-based programs registry: Blueprints for healthy youth development. Evaluation and Program Planning,48, 124–131.

    Article  PubMed  Google Scholar 

  • Mihalic, S. F., Fagan, A. A., & Argamaso, S. (2008). Implementing the life skills training drug prevention program: Factors related to implementation fidelity. Implementation Science,3(5), 1–16.

    Google Scholar 

  • Muthén, L. K., & Muthén, B. O. (1998–2017). Mplus User’s Guide. (8th ed.). Los Angeles, CA: Muthén & Muthén.

  • Nicholson, L. M., Slater, S. J., Chriqui, J. F., & Chaloupka, F. (2014). Validating adolescent socioeconomic status: Comparing school free or reduced price lunch with community measures. Spatial Demography,2(1), 55–65.

    Article  Google Scholar 

  • Pettigrew, J., Graham, J. W., Miller-Day, M., Hecht, M. L., Krieger, J. L., & Shin, Y. J. (2015). Adherence and delivery: Implementation quality and program outcomes for the seventh-grade keepin’ it REAL program. Prevention Science,16(1), 90–99.

    Article  PubMed  PubMed Central  Google Scholar 

  • Prado, G., Pantin, H., Schwartz, S. J., Lupei, N. S., & Szapocznik, J. (2006). Predictors of engagement and retention into a parent-centered ecodevelopmental HIV preventive intervention for Hispanic adolescents and their families. Journal of Pediatric Psychology,31(9), 874–890.

    Article  PubMed  Google Scholar 

  • Reyes, M. R., Brackett, M. A., Rivers, S. E., Elbertson, N. A., & Salovey, P. (2012). The interaction effects of program training, dosage, and implementation quality on targeted student outcomes for the RULER approach to social and emotional learning. School Psychology Review,41(1), 82–99.

    Google Scholar 

  • Schwartz, J. A., & Beaver, K. M. (2015). Making (up) the grade? Estimating the genetic and environmental influences on discrepancies between self-reported grades and official GPA scores. Journal of Youth and Adolescence,44(5), 1125–1138.

    Article  PubMed  Google Scholar 

  • Sutherland, K. S., Conroy, M. A., McLeod, B. D., Algina, J., & Kunemund, R. L. (2018). Factors associated with teacher delivery of a classroom-based tier 2 prevention program. Prevention Science,19(2), 186–196.

    Article  PubMed  PubMed Central  Google Scholar 

  • Taylor, T. H. (2012). Ceiling effect. In N. J. Salkind (Ed.), Encyclopedia of research design (pp. 133–134). Thousand Oaks, CA: Sage Publications.

    Google Scholar 

  • Trevethan, R. (2017). Intraclass correlation coefficients: Clearing the air, extending some cautions, and making some requests. Health Services and Outcomes Research Methodology,17(2), 127–143.

    Article  Google Scholar 

  • Velasco, V., Griffin, K. W., Botvin, G. J., Celata, C., & Lombardia, Gruppo L. S. T. (2017). Preventing adolescent substance use through an evidence-based program: Effects of the Italian adaptation of life skills training. Prevention Science,18(4), 394–405.

    Article  PubMed  Google Scholar 

  • Weissberg, R. P., Durlak, J. A., Domitrovich, C. E., & Gullotta, T. P. (2015). Social and emotional learning: Past, present, and future. In J. A. Durlak, C. E. Domitrovich, R. P. Weissberg, & T. P. Gullotta (Eds.), Handbook of social and emotional learning: Research and practice (pp. 3–19). New York: The Guilford Press.

    Google Scholar 

  • Williford, A. P., Wolcott, C. S., Whittaker, J. V., & Locasale-Crouch, J. (2015). Program and teacher characteristics predicting the implementation of Banking Time with preschoolers who display disruptive behaviors. Prevention Science,16(8), 1054–1063.

    Article  PubMed  Google Scholar 

  • Wright, A., Lamont, A., Wandersman, A., Osher, D., & Gordon, E. S. (2015). Accountability and SEL programs: The Getting To Outcomes® approach. In J. A. Durlak, C. E. Domitrovich, R. P. Weissberg, & T. P. Gullotta (Eds.), Handbook of social and emotional learning: Research and practice (pp. 500–515). New York: The Guilford Press.

    Google Scholar 

  • Zinsser, K. M., & Dusenbury, L. (2015). Recommendations for implementing the new Illinois early learning and development standards to affect classroom practices for social and emotional learning. Early Childhood Research and Practice,17(1), 1–8.

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Child and Family Studies, College of Behavioral and Community Sciences, University of South Florida, 13301 Bruce B. Downs Blvd., Tampa, FL, 33612, USA

    Enya B. Vroom, Oliver T. Massey, Svetlana Yampolskaya & Bruce L. Levin

  2. College of Public Health, University of South Florida, 13201 Bruce B. Downs Blvd., Tampa, FL, 33612, USA

    Bruce L. Levin

Authors
  1. Enya B. Vroom
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Oliver T. Massey
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Svetlana Yampolskaya
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Bruce L. Levin
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Enya B. Vroom.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Ethical Approval

All procedures were in accordance with the ethical standards of the institutional review boards and with the 1964 Declaration of Helsinki and its later amendments or comparable ethical standards. For this type of study, formal consent was not required.

Informed Consent

Informed consent was obtained from all individual participants included in the study.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Vroom, E.B., Massey, O.T., Yampolskaya, S. et al. The Impact of Implementation Fidelity on Student Outcomes in the Life Skills Training Program. School Mental Health 12, 113–123 (2020). https://doi.org/10.1007/s12310-019-09333-1

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12310-019-09333-1

Keywords

Search

Navigation