Effects of After-School Programs with At-Risk Youth on Attendance and Externalizing Behaviors: A Systematic Review and Meta-Analysis
The popularity, demand, and increased federal and private funding for after-school programs have resulted in a marked increase in after-school programs over the past two decades. After-school programs are used to prevent adverse outcomes, decrease risks, or improve functioning with at-risk youth in several areas, including academic achievement, crime and behavioral problems, socio-emotional functioning, and school engagement and attendance; however, the evidence of effects of after-school programs remains equivocal. This systematic review and meta-analysis, following Campbell Collaboration guidelines, examined the effects of after-school programs on externalizing behaviors and school attendance with at-risk students. A systematic search for published and unpublished literature resulted in the inclusion of 24 studies. A total of 64 effect sizes (16 for attendance outcomes; 49 for externalizing behavior outcomes) extracted from 31 reports were included in the meta-analysis using robust variance estimation to handle dependencies among effect sizes. Mean effects were small and non-significant for attendance and externalizing behaviors. A moderate to large amount of heterogeneity was present; however, no moderator variable tested explained the variance between studies. Significant methodological shortcomings were identified across the corpus of studies included in this review. Implications for practice, policy and research are discussed.
KeywordsAfter-school programs Attendance Externalizing behaviors Systematic review Meta-analysis
The authors are grateful for support from the Meadows Center for Preventing Educational Risk, the Greater Texas Foundation, the Institute of Education Sciences (Grants R324A100022, R324B080008, and R305B100016) and from the Eunice Kennedy Shriver National Institute of Child Health and Human Development (P50 HD052117). The content is solely the responsibility of the authors and does not necessarily represent the official views of the supporting entities.
KK participated in the conception and design of the study, acquisition of data, and drafting of the manuscript; BM participated in the conception and design of the study, acquisition and analysis of data, and drafting of the manuscript; JP participated in the acquisition and analysis of data and revision of the manuscript; MV participated in the conception and design of the study and revision of the manuscript; CS participated in the acquisition of data and revision of the manuscript. All authors read and approved the final manuscript.
* References marked with an asterisk indicate studies included in the meta-analysis
- Afterschool Alliance. (2014). America after 3PM: Afterschool programs in demand. http://afterschoolalliance.org/documents/AA3PM-2014/AA3PM_National_Report.pdf.
- Arcaira, E., Vile, J. D., & Reisner, E. R. (2010). Achieving high school graduation: Citizen Schools’ youth outcomes in Boston. Washington, DC: Policy Studies Associates Inc.Google Scholar
- Bushman, B. J., & Wells, G. L. (2001). Narrative impressions of literature: The availability bias and the corrective properties of meta-analytic approaches. Personality and Social Psychology Bulletin, 27(9), 1123–1130.Google Scholar
- Campbell Collaboration. (2014). Campbell Collaboration systematic reviews: Policies and guidelines. http://www.campbellcollaboration.org/lib/download/3308/C2_Policies_Guidelines_Version_1_0.pdf.
- Cooper, H., & Hedges, L. V. (1994). The handbook of research synthesis. New York: Russell Sage Foundation.Google Scholar
- Dynarski, M., James-Burdumy, S., Moore, M., Rosenberg, L., Deke, J., Mansfield, W., et al. (2004). When schools stay open late: The national evaluation of the 21st Century Community Learning Centers program: New findings. U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance.Google Scholar
- Fashola, O. S. (1998). Review of extended-day and after-school programs and their effectiveness. Washington, D.C.: Center for Research on the Education of Students Placed at Risk.Google Scholar
- Fisher, Z., & Tipton, E. (2014). Robumeta (Version 1.1) (Software). http://cran.r-project.org/web/packages/robumeta/robumeta.pdf.
- *Foley, E. M., & Eddins, G. (2001). Preliminary analysis of Virtual Y after-school program participants’ patterns of school attendance and academic performance. New York, NY: Fordham University.Google Scholar
- Fox, J.A., & Newman, S.A. (1997). After-school crime or after-school programs: Tuning into the prime time for violent juvenile crime and implications for national policy. A report to the United States Attorney General. Washington, D.C.: Fight Crime: Invest in Kids.Google Scholar
- *Frazier, S., Mehta, T., Atkins, M., Hur, K., & Rusch, D. (2013). Not just a walk in the park: Efficacy to effectiveness for after school programs in communities of concentrated urban poverty. Administration and Policy in Mental Health and Mental Health Services Research, 40(5), 406–418.PubMedCrossRefGoogle Scholar
- Glass, G. V., McGaw, B., & Smith, M. L. (1981). Meta-analysis in social research. Thousand Oaks, CA: Sage Publications.Google Scholar
- Gottfredson, D., Cross, A., Wilson, D., Connell, N., & Rorie, M. (2010a). A randomized trial of the effects of an enhanced after-school program for middle-school students. Final report submitted to the U.S. Department of Education Institute for Educational Sciences.Google Scholar
- Gozsche, P. C., Hrogjartsson, A., Maric, K., & Tendal, B. (2007). Data extraction errors in meta-analyses that use standardized mean differences. Journal of the American Medical Association, 298, 430–437.Google Scholar
- *Hirsch, B., Hedges, L.V., Stawicki, J., & Mekinda, M.A. (2011). After-school programs for high school students: An evaluation of After School Matters. Chicago, IL: Northwestern University.Google Scholar
- Hollister, R. (2003). The growth in after-school programs and their impact. Washington, DC: Brookings Institution.Google Scholar
- Hopewell, S., Loudon, K., Clarke, M. J., Oxman, A. D., & Dickersin, K. (2009). Publication bias in clinical trials due to statistical significance or direction of trial results. The Cochrane Database of Systematic Reviews, 1(1), 1–26.Google Scholar
- Hopewell, S., McDonald, S., Clarke, M., & Egger, M. (2007). Grey literature in meta-analyses of randomized trials of health care interventions. The Cochrane Database of Systematic Reviews, 2(2).Google Scholar
- Huang, D., Silver, D., Cheung, M., Duong, N., Gualpa, A., Hodson, C., Rivera, G. (2011). Independent statewide evaluation of after school programs: ASES and 21st CCLC year 2 annual report. CRESST Report 789. Los Angeles: National Center for Research on Evaluation, Standards, and Student Testing.Google Scholar
- James-Burdumy, S., Dynarski, M., Moore, M., Deke. J., Mansfield, W., Pistorino, C., et al. (2005). When schools stay open late: The national evaluation of the 21st Century Community Learning Centers program: Final report. U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance.Google Scholar
- Kane, T.J. (2004). The impact of after-school programs: Interpreting the results of four recent evaluations. Working paper. New York, NY: W.T. Grant Foundation.Google Scholar
- *LaFrance, S., Twersky, F., Latham, N., Foley, E., Bott, C., & Lee, L. (2001). A safe place for healthy youth development: A comprehensive evaluation of the Bayview Safe Haven. San Francisco, CA: BTW Consultants & LaFrance Associates.Google Scholar
- Lipsey, M. W., & Wilson, D. B. (2001). Practical meta-analysis. Thousand Oaks, CA: Sage.Google Scholar
- Mahoney, J. L., Parente, M. E., & Zigler, E. F. (2009). Afterschool programs in America: Origins, growth, popularity, and politics. Journal of Youth Development, 4(3), 25–44.Google Scholar
- McKinsey & Company. (2009). The economic impact of the achievement gap in America’s schools. Washington, DC: Social Sector Office.Google Scholar
- *Molina, B. S., Flory, K., Bukstein, O. G., Greiner, A. R., Baker, J. L., Krug, V., et al. (2008). Feasibility and preliminary efficacy of an after-school program for middle schoolers with ADHD: A randomized trial in a large public middle school. Journal of Attention Disorders, 12(3), 207–217.PubMedCrossRefGoogle Scholar
- Newman, S. A., Fox, J. A., Flynn, E. A., & Christeson, W. (2000). America’s after-school choice: The prime time for juvenile crime or youth enrichment and achievement. Washington, DC: Fight Crime: Invest in Kids.Google Scholar
- *Nguyen, D. (2007). A statewide impact study of 21st Century Community Learning Center programs in Florida. Tallahassee, FL: Department of Educational Leadership and Policy Studies, Florida State University. Doctor of Philosophy.Google Scholar
- Parsad, B., & Lewis, L. (2009). After-school programs in public elementary schools (NCES 2009-043). Washington, DC: National Center for Education Statistics.Google Scholar
- *Pastchal-Temple, A. (2012). The effect of regular participation in an after-school program on student achievement, attendance, and behavior. Educational Administration, Mississippi State University. Doctor of Philosophy.Google Scholar
- Pigott, T. D. (1994). Methods for handling missing data in research synthesis. In H. Cooper & L. V. Hedges (Eds.), The handbook of research synthesis (pp. 163–175). New York: Russell Sage Foundation.Google Scholar
- Polanin, J.R., & Pigott, T.D. (2014). The use of meta-analytic statistical significance testing. Research Synthesis Methods. Advanced online publication. doi: 10.1002/jrsm.1124.
- *Prenovost, J. K. E. (2001). A first-year evaluation of after-school learning programs in four urban middle schools in the Santa Ana Unified School District. Irvine, CA: Educational Administration, University of California—Irvine. Doctor of Education.Google Scholar
- Redd, Z., Cochran, S., Hair, E., & Moore, K. (2002). Academic achievement programs and youth development: A synthesis. www.childtrends.org/what_works/clarkwww/acadach/acadachrpt.pdf.
- Shadish, W. R., Cook, T. D., & Campbell, T. D. (2002). Experimental and quasi-experimental design for generalized causal inference. Belmont, CA: Wadsworth Cengage Learning.Google Scholar
- *Sibley-Butler, O. (2004). The impact of a 21st Century Community Learning Center after-school program on school-age children in grades 4 & 5. Hattiesburg, MS: University of Southern Mississippi. Doctor of Philosophy.Google Scholar
- Simpkins, S.C., Little, P.M.D., & Weiss, H.B. (2004). Understanding and measuring attendance in out-of-school time programs (Number 7). http://www.gse.harvard.edu/hfrp/content/projects/afterschool/resources/issuebrief7.pdf.
- *Smeallie, J. E. (1997). An evaluation of an after-school tutorial and study skills program for middle school students at risk of academic failure. Maryland: Department of Curriculum and Instruction, University of Maryland at College Park. Doctor of Education.Google Scholar
- *Tebes, J. K., Feinn, R., Vanderploeg, J. J., Chinman, M. J., Shepard, J., Brabham, T., et al. (2007). Impact of a positive youth development program in urban after-school settings on the prevention of adolescent substance use. Journal of Adolescent Health, 41(3), 239–247.PubMedCrossRefGoogle Scholar
- Tolan, P., Henry, D., Schoeny, M., Bass, A., Lovegrove, P., & Nichols, E. (2013). Mentoring interventions to affect juvenile delinquency and associated problems: A systematic review. Campbell Collaboration Reviews, 2013, 10.Google Scholar
- U.S. Department of Education. (2011). Elementary and secondary education: Part B—21st Century Community Learning Centers. http://www2.ed.gov/policy/elsec/leg/esea02/pg55.html.
- U.S. Department of Education. (2014). Programs: 21st Century Community Learning Centers. http://www2.ed.gov/programs/21stcclc/index.html.
- *Weisman, S. A., Womer, S. C., Kellstrong, M., Bryner, S., Kahler, A., Slocum, L. A., et al. (2003). Maryland After-School Community Grant Program Part 1: Report on the 2002–2003 school year evaluation of the phase 3 after-school programs. College Park, MD: University of Maryland.Google Scholar
- *Welsh, M. E., Russell, C. A., Williams, I., Reisner, E. R., & White, R. N. (2002). Promoting learning and school attendance through after school programs: Student level changes in education across TASC’s first three years. Washington, D. C.: Policy Studies Associates Inc.Google Scholar
- Zief, S.G., Lauver, S., & Maynard, R.A. (2006). Impacts of after-school programs on student outcomes. Campbell Systematic Reviews.Google Scholar