Advertisement

Journal of Youth and Adolescence

, Volume 44, Issue 3, pp 616–636 | Cite as

Effects of After-School Programs with At-Risk Youth on Attendance and Externalizing Behaviors: A Systematic Review and Meta-Analysis

  • Kristen P. Kremer
  • Brandy R. Maynard
  • Joshua R. Polanin
  • Michael G. Vaughn
  • Christine M. Sarteschi
Empirical Research

Abstract

The popularity, demand, and increased federal and private funding for after-school programs have resulted in a marked increase in after-school programs over the past two decades. After-school programs are used to prevent adverse outcomes, decrease risks, or improve functioning with at-risk youth in several areas, including academic achievement, crime and behavioral problems, socio-emotional functioning, and school engagement and attendance; however, the evidence of effects of after-school programs remains equivocal. This systematic review and meta-analysis, following Campbell Collaboration guidelines, examined the effects of after-school programs on externalizing behaviors and school attendance with at-risk students. A systematic search for published and unpublished literature resulted in the inclusion of 24 studies. A total of 64 effect sizes (16 for attendance outcomes; 49 for externalizing behavior outcomes) extracted from 31 reports were included in the meta-analysis using robust variance estimation to handle dependencies among effect sizes. Mean effects were small and non-significant for attendance and externalizing behaviors. A moderate to large amount of heterogeneity was present; however, no moderator variable tested explained the variance between studies. Significant methodological shortcomings were identified across the corpus of studies included in this review. Implications for practice, policy and research are discussed.

Keywords

After-school programs Attendance Externalizing behaviors Systematic review Meta-analysis 

Notes

Acknowledgments

The authors are grateful for support from the Meadows Center for Preventing Educational Risk, the Greater Texas Foundation, the Institute of Education Sciences (Grants R324A100022, R324B080008, and R305B100016) and from the Eunice Kennedy Shriver National Institute of Child Health and Human Development (P50 HD052117). The content is solely the responsibility of the authors and does not necessarily represent the official views of the supporting entities.

Author contributions

KK participated in the conception and design of the study, acquisition of data, and drafting of the manuscript; BM participated in the conception and design of the study, acquisition and analysis of data, and drafting of the manuscript; JP participated in the acquisition and analysis of data and revision of the manuscript; MV participated in the conception and design of the study and revision of the manuscript; CS participated in the acquisition of data and revision of the manuscript. All authors read and approved the final manuscript.

Supplementary material

10964_2014_226_MOESM1_ESM.pdf (9 kb)
Supplementary material 1 (PDF 8 kb)
10964_2014_226_MOESM2_ESM.pdf (255 kb)
Supplementary material 2 (PDF 255 kb)

References

* References marked with an asterisk indicate studies included in the meta-analysis

  1. Afterschool Alliance. (2014). America after 3PM: Afterschool programs in demand. http://afterschoolalliance.org/documents/AA3PM-2014/AA3PM_National_Report.pdf.
  2. *Apsler, R. (2009). After-school programs for adolescents: A review of evaluation research. Adolescence, 44, 1–19.PubMedCrossRefGoogle Scholar
  3. Arcaira, E., Vile, J. D., & Reisner, E. R. (2010). Achieving high school graduation: Citizen Schools’ youth outcomes in Boston. Washington, DC: Policy Studies Associates Inc.Google Scholar
  4. *Biggart, A., Kerr, K., O’Hare, L., & Connolly, P. (2013). A randomised control trial evaluation of a literacy after-school programme for struggling beginning readers. International Journal of Educational Research, 62, 129–140.CrossRefGoogle Scholar
  5. Biglan, A., Metzler, C. W., Wirt, R., Ary, D., Noell, J., Ochs, L., et al. (1990). Social and behavioral factors associated with high-risk sexual behaviors among adolescents. Journal of Behavioral Medicine, 13, 245–261.PubMedCrossRefGoogle Scholar
  6. *Blumer, M. L. C., & Werner-Wilson, R. J. (2010). Leaving no girl behind: Clinical intervention effects on adolescent female academic “high-risk” behaviors. Journal of Feminist Family Therapy, 22(1), 22–42.CrossRefGoogle Scholar
  7. Bohnert, A., Fredricks, J., & Randall, E. (2010). Capturing unique dimensions of youth organized activity involvement theoretical and methodological considerations. Review of Educational Research, 80(4), 576–610.CrossRefGoogle Scholar
  8. Bohnert, A. M., Richards, M., Kohl, K., & Randall, E. (2009). Relationships between discretionary time activities, emotional experiences, delinquency and depressive symptoms among urban African American adolescents. Journal of Youth and Adolescence, 38, 587–601.PubMedCrossRefGoogle Scholar
  9. Bradshaw, C. P., Waasdorp, T. E., Goldweber, A., & Johnson, S. L. (2013). Bullies, gangs, drugs, and school: Understanding the overlap and the role of ethnicity and urbanicity. Journal of Youth and Adolescence, 42(2), 220–234.PubMedCrossRefGoogle Scholar
  10. Buscemi, N., Hartling, L., Vandemeer, B., Tjosvold, L., & Klassen, T. P. (2006). Single data extraction generated more errors than double data extraction in systematic reviews. Journal of Clinical Epidemiology, 59, 697–703.PubMedCrossRefGoogle Scholar
  11. Bushman, B. J., & Wells, G. L. (2001). Narrative impressions of literature: The availability bias and the corrective properties of meta-analytic approaches. Personality and Social Psychology Bulletin, 27(9), 1123–1130.Google Scholar
  12. Campbell Collaboration. (2014). Campbell Collaboration systematic reviews: Policies and guidelines. http://www.campbellcollaboration.org/lib/download/3308/C2_Policies_Guidelines_Version_1_0.pdf.
  13. Cooper, H., DeNeve, K., & Charlton, K. (1997). Finding the missing science: The fate of studies submitted for review by a human subjects committee. Psychological Methods, 2(4), 447.CrossRefGoogle Scholar
  14. Cooper, H., & Hedges, L. V. (1994). The handbook of research synthesis. New York: Russell Sage Foundation.Google Scholar
  15. Cross, A. B., Gottfredson, D. C., Wilson, D. M., Rorie, M., & Connell, N. (2009). The impact of after-school programs on the routine activities of middle-school students: Results from a randomized, controlled trial. Criminology and Public Policy, 8(2), 391–412.CrossRefGoogle Scholar
  16. DuBois, D. L., Portillo, N., Rhodes, J. E., Silverthorn, N., & Valentine, J. C. (2011). How effective are mentoring programs for youth? A systematic assessment of the evidence. Psychological Science in the Public Interest, 12, 57–91.CrossRefGoogle Scholar
  17. Durlak, J. A., Weissberg, R. P., & Pachan, M. (2010). A meta-analysis of after-school programs that seek to promote personal social skills in children and adolescents. American Journal of Community Psychology, 45, 294–309.PubMedCrossRefGoogle Scholar
  18. Dynarski, M., James-Burdumy, S., Moore, M., Rosenberg, L., Deke, J., Mansfield, W., et al. (2004). When schools stay open late: The national evaluation of the 21st Century Community Learning Centers program: New findings. U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance.Google Scholar
  19. Fashola, O. S. (1998). Review of extended-day and after-school programs and their effectiveness. Washington, D.C.: Center for Research on the Education of Students Placed at Risk.Google Scholar
  20. Fisher, Z., & Tipton, E. (2014). Robumeta (Version 1.1) (Software). http://cran.r-project.org/web/packages/robumeta/robumeta.pdf.
  21. *Foley, E. M., & Eddins, G. (2001). Preliminary analysis of Virtual Y after-school program participants’ patterns of school attendance and academic performance. New York, NY: Fordham University.Google Scholar
  22. Fox, J.A., & Newman, S.A. (1997). After-school crime or after-school programs: Tuning into the prime time for violent juvenile crime and implications for national policy. A report to the United States Attorney General. Washington, D.C.: Fight Crime: Invest in Kids.Google Scholar
  23. *Frazier, S., Mehta, T., Atkins, M., Hur, K., & Rusch, D. (2013). Not just a walk in the park: Efficacy to effectiveness for after school programs in communities of concentrated urban poverty. Administration and Policy in Mental Health and Mental Health Services Research, 40(5), 406–418.PubMedCrossRefGoogle Scholar
  24. Glass, G. V., McGaw, B., & Smith, M. L. (1981). Meta-analysis in social research. Thousand Oaks, CA: Sage Publications.Google Scholar
  25. *Gottfredson, D., Cross, A., & Soulé, D. (2007). Distinguishing characteristics of effective and ineffective after-school programs to prevent delinquency and victimization. Criminology and Public Policy, 6(2), 289–318.CrossRefGoogle Scholar
  26. Gottfredson, D., Cross, A., Wilson, D., Connell, N., & Rorie, M. (2010a). A randomized trial of the effects of an enhanced after-school program for middle-school students. Final report submitted to the U.S. Department of Education Institute for Educational Sciences.Google Scholar
  27. Gottfredson, D., Cross, A., Wilson, D., Rorie, M., & Connell, N. (2010b). An experimental evaluation of the All Stars prevention curriculum in a community after school setting. Prevention Science, 11(2), 142–154.PubMedCrossRefGoogle Scholar
  28. Gottfredson, D., Cross, A., Wilson, D., Rorie, M., & Connell, N. (2010c). Effects of participation in after-school programs for middle school students: A randomized trial. Journal of Research on Educational Effectiveness, 3(3), 282–313.CrossRefGoogle Scholar
  29. *Gottfredson, D., Gerstenblith, S. A., Soulé, D., Womer, S. C., & Lu, S. (2004). Do after school programs reduce delinquency? Prevention Science, 5(4), 253–266.PubMedCrossRefGoogle Scholar
  30. Gottfredson, D. C., Gottfredson, G. D., & Weisman, S. A. (2001). The timing of delinquent behavior and its implications for after-school programs. Criminology & Public Policy, 1, 61–86.CrossRefGoogle Scholar
  31. Gozsche, P. C., Hrogjartsson, A., Maric, K., & Tendal, B. (2007). Data extraction errors in meta-analyses that use standardized mean differences. Journal of the American Medical Association, 298, 430–437.Google Scholar
  32. Halpern, R. (1999). After-school programs for low-income children: Promises and challenges. The Future of Children, 9(2), 81–95.PubMedCrossRefGoogle Scholar
  33. Halpern, R. (2002). A different kind of child development institution: The history of after-school programs for low-income children. Teachers College Record, 104, 178–211.CrossRefGoogle Scholar
  34. Hedges, L. V., Tipton, E., & Johnson, M. C. (2010). Robust variance estimation in meta-regression with dependent effect size estimates. Research Synthesis Methods, 1(1), 164–165.CrossRefGoogle Scholar
  35. Higgins, J. P., Altman, D. G., Gotzsche, P. C., Juni, P., Moher, D., Oxman, A. D., et al. (2011). The Cochrane Collaboration’s tool for assessing risk of bias in randomised trials. British Medical Journal, 343, d5928. doi: 10.1136/bmj.d5928.PubMedCentralPubMedCrossRefGoogle Scholar
  36. Higgins, J. P., & Thompson, S. G. (2002). Quantifying heterogeneity in a meta-analysis. Statistics in Medicine, 21(11), 1539–1558.PubMedCrossRefGoogle Scholar
  37. *Hirsch, B., Hedges, L.V., Stawicki, J., & Mekinda, M.A. (2011). After-school programs for high school students: An evaluation of After School Matters. Chicago, IL: Northwestern University.Google Scholar
  38. Hollister, R. (2003). The growth in after-school programs and their impact. Washington, DC: Brookings Institution.Google Scholar
  39. Hopewell, S., Loudon, K., Clarke, M. J., Oxman, A. D., & Dickersin, K. (2009). Publication bias in clinical trials due to statistical significance or direction of trial results. The Cochrane Database of Systematic Reviews, 1(1), 1–26.Google Scholar
  40. Hopewell, S., McDonald, S., Clarke, M., & Egger, M. (2007). Grey literature in meta-analyses of randomized trials of health care interventions. The Cochrane Database of Systematic Reviews, 2(2).Google Scholar
  41. Huang, D., Silver, D., Cheung, M., Duong, N., Gualpa, A., Hodson, C., Rivera, G. (2011). Independent statewide evaluation of after school programs: ASES and 21st CCLC year 2 annual report. CRESST Report 789. Los Angeles: National Center for Research on Evaluation, Standards, and Student Testing.Google Scholar
  42. *James-Burdumy, S., Dynarski, M., & Deke, J. (2007). When elementary schools stay open late: Results from the national evaluation of the 21st Century Community Learning Centers program. Educational Evaluation and Policy Analysis, 29(4), 296–318.CrossRefGoogle Scholar
  43. *James-Burdumy, S., Dynarski, M., & Deke, J. (2008). After-school program effects on behavior: Results from the 21st Century Community Learning Centers program national evaluation. Economic Inquiry, 46(1), 13–18.CrossRefGoogle Scholar
  44. James-Burdumy, S., Dynarski, M., Moore, M., Deke. J., Mansfield, W., Pistorino, C., et al. (2005). When schools stay open late: The national evaluation of the 21st Century Community Learning Centers program: Final report. U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance.Google Scholar
  45. Kane, T.J. (2004). The impact of after-school programs: Interpreting the results of four recent evaluations. Working paper. New York, NY: W.T. Grant Foundation.Google Scholar
  46. *LaFrance, S., Twersky, F., Latham, N., Foley, E., Bott, C., & Lee, L. (2001). A safe place for healthy youth development: A comprehensive evaluation of the Bayview Safe Haven. San Francisco, CA: BTW Consultants & LaFrance Associates.Google Scholar
  47. *Langberg, J., Smith, B. H., Bogle, K. E., Schmidt, J. D., Cole, W. R., & Pender, C. A. S. (2007). A pilot evaluation of small group challenging horizons program (CHP): A randomized trial. Journal of Applied School Psychology, 23(1), 31–58.CrossRefGoogle Scholar
  48. Lauer, P. A., Akiba, M., Wilkerson, S. B., Apthorp, H. S., Snow, D., & Martin-Glenn, M. L. (2006). Out-of-school-time programs: A meta-analysis of effects for at-risk students. Review of Educational Research, 76, 275–313.CrossRefGoogle Scholar
  49. *Le, T., Arifuku, I., Vuong, L., Tran, G., Lustig, D., & Zimring, F. (2011). Community mobilization and community-based participatory research to prevent youth violence among Asian and immigrant populations. American Journal of Community Psychology, 48(1/2), 77–88.PubMedGoogle Scholar
  50. Lipsey, M. W., & Wilson, D. B. (2001). Practical meta-analysis. Thousand Oaks, CA: Sage.Google Scholar
  51. Littell, J. H., Corcoran, J., & Pillai, V. (2008). Systematic reviews and meta-analysis. Oxford: Oxford University Press.CrossRefGoogle Scholar
  52. Mahoney, J. L., Parente, M. E., & Zigler, E. F. (2009). Afterschool programs in America: Origins, growth, popularity, and politics. Journal of Youth Development, 4(3), 25–44.Google Scholar
  53. Maynard, B. R., Peters, K. E., Vaughn, M. G., & Sarteschi, C. M. (2013). Fidelity in after-school program intervention research: A systematic review. Research on Social Work Practice, 23(6), 613–623.CrossRefGoogle Scholar
  54. Maynard, B. R., Salas-Wright, C. P., Vaughn, M. G., & Peters, K. E. (2012). Who are truant youth? Examining distinctive profiles of truant youth using latent profile analysis. Journal of Youth and Adolescence, 41(12), 1671–1684.PubMedCentralPubMedCrossRefGoogle Scholar
  55. McKinsey & Company. (2009). The economic impact of the achievement gap in America’s schools. Washington, DC: Social Sector Office.Google Scholar
  56. McLeod, B. D., & Weisz, J. R. (2004). Using dissertations to examine potential bias in child and adolescent clinical trials. Journal of Consulting and Clinical Psychology, 72(2), 235–251. doi: 10.1037/0022-006x.72.2.235.PubMedCrossRefGoogle Scholar
  57. *Molina, B. S., Flory, K., Bukstein, O. G., Greiner, A. R., Baker, J. L., Krug, V., et al. (2008). Feasibility and preliminary efficacy of an after-school program for middle schoolers with ADHD: A randomized trial in a large public middle school. Journal of Attention Disorders, 12(3), 207–217.PubMedCrossRefGoogle Scholar
  58. Moyer, A., Schneider, S., Knapp-Oliver, S. K., & Sohl, S. J. (2010). Published versus unpublished dissertations in psycho-oncology intervention research. Psycho-Oncology, 19(3), 313–317.PubMedCentralPubMedCrossRefGoogle Scholar
  59. Newman, S. A., Fox, J. A., Flynn, E. A., & Christeson, W. (2000). America’s after-school choice: The prime time for juvenile crime or youth enrichment and achievement. Washington, DC: Fight Crime: Invest in Kids.Google Scholar
  60. *Nguyen, D. (2007). A statewide impact study of 21st Century Community Learning Center programs in Florida. Tallahassee, FL: Department of Educational Leadership and Policy Studies, Florida State University. Doctor of Philosophy.Google Scholar
  61. *Oyserman, D., Terry, K., & Bybee, D. (2002). A possible selves intervention to enhance school involvement. Journal of Adolescence, 25(3), 313.PubMedCrossRefGoogle Scholar
  62. Parsad, B., & Lewis, L. (2009). After-school programs in public elementary schools (NCES 2009-043). Washington, DC: National Center for Education Statistics.Google Scholar
  63. *Pastchal-Temple, A. (2012). The effect of regular participation in an after-school program on student achievement, attendance, and behavior. Educational Administration, Mississippi State University. Doctor of Philosophy.Google Scholar
  64. Pigott, T. D. (1994). Methods for handling missing data in research synthesis. In H. Cooper & L. V. Hedges (Eds.), The handbook of research synthesis (pp. 163–175). New York: Russell Sage Foundation.Google Scholar
  65. Pigott, T. D. (2012). Advances in meta-analysis. New York, NY: Springer.CrossRefGoogle Scholar
  66. Pigott, T. D., Valentine, J. C., Polanin, J. R., Williams, R. T., & Canada, D. D. (2013). Outcome-reporting bias in education research. Educational Researcher, 42(8), 424–434. doi: 10.3102/0013189X13507104.CrossRefGoogle Scholar
  67. Polanin, J.R., & Pigott, T.D. (2014). The use of meta-analytic statistical significance testing. Research Synthesis Methods. Advanced online publication. doi: 10.1002/jrsm.1124.
  68. *Prenovost, J. K. E. (2001). A first-year evaluation of after-school learning programs in four urban middle schools in the Santa Ana Unified School District. Irvine, CA: Educational Administration, University of California—Irvine. Doctor of Education.Google Scholar
  69. Redd, Z., Cochran, S., Hair, E., & Moore, K. (2002). Academic achievement programs and youth development: A synthesis. www.childtrends.org/what_works/clarkwww/acadach/acadachrpt.pdf.
  70. Richards, M. H., Miller, B. V., O’Donnell, P. C., Wasserman, M. S., & Colder, C. (2004). Parental monitoring mediates the effects of age and sex on problem behaviors among African American urban young adolescents. Journal of Youth and Adolescence, 33, 221–233.CrossRefGoogle Scholar
  71. Richardson, J. L., Dwyer, K., McGuigan, K., Hansen, W. B., Dent, C., Johnson, C. A., et al. (1989). Substance use among eighth-grade students who take care of themselves after school. Pediatrics, 84, 556–566.PubMedGoogle Scholar
  72. Ritter, G. W., Barnett, J. H., Denny, G. S., & Albin, G. R. (2009). The effectiveness of volunteer tutoring programs for elementary and middle school students: A meta-analysis. Review of Educational Research, 79, 3–38.CrossRefGoogle Scholar
  73. Roth, J. L., Malone, L. M., & Brooks-Gunn, J. (2010). Does the amount of participation in afterschool programs relate to developmental outcomes? A review of the literature. American Journal of Community Psychology, 45, 310–324.PubMedCrossRefGoogle Scholar
  74. *Schinke, S., Cole, K. C., & Poulin, S. R. (2000). Enhancing the educational achievement of at-risk youth. Prevention Science, 1(1), 51–60.PubMedCrossRefGoogle Scholar
  75. Scott-Little, C., Hamann, M. S., & Jurs, S. G. (2002). Evaluations of after-school programs: A meta-evaluation of methodologies and narrative synthesis of findings. American Journal of Evaluation, 23, 387–419.CrossRefGoogle Scholar
  76. Shadish, W. R., Cook, T. D., & Campbell, T. D. (2002). Experimental and quasi-experimental design for generalized causal inference. Belmont, CA: Wadsworth Cengage Learning.Google Scholar
  77. Shernoff, D. J., & Vandell, D. L. (2007). Engagement in after-school program activities: Quality of experience from the perspective of participants. Journal of Youth and Adolescence, 36, 891–903.CrossRefGoogle Scholar
  78. *Sibley-Butler, O. (2004). The impact of a 21st Century Community Learning Center after-school program on school-age children in grades 4 & 5. Hattiesburg, MS: University of Southern Mississippi. Doctor of Philosophy.Google Scholar
  79. Simpkins, S.C., Little, P.M.D., & Weiss, H.B. (2004). Understanding and measuring attendance in out-of-school time programs (Number 7). http://www.gse.harvard.edu/hfrp/content/projects/afterschool/resources/issuebrief7.pdf.
  80. *Smeallie, J. E. (1997). An evaluation of an after-school tutorial and study skills program for middle school students at risk of academic failure. Maryland: Department of Curriculum and Instruction, University of Maryland at College Park. Doctor of Education.Google Scholar
  81. Tanner-Smith, E. E., & Tipton, E. (2013). Robust variance estimation with dependent effect sizes: Practical considerations including a software tutorial in Stata and SPSS. Published advance online. Research Synthesis Methods,. doi: 10.1002/jrsm.1091.Google Scholar
  82. *Tebes, J. K., Feinn, R., Vanderploeg, J. J., Chinman, M. J., Shepard, J., Brabham, T., et al. (2007). Impact of a positive youth development program in urban after-school settings on the prevention of adolescent substance use. Journal of Adolescent Health, 41(3), 239–247.PubMedCrossRefGoogle Scholar
  83. Tolan, P., Henry, D., Schoeny, M., Bass, A., Lovegrove, P., & Nichols, E. (2013). Mentoring interventions to affect juvenile delinquency and associated problems: A systematic review. Campbell Collaboration Reviews, 2013, 10.Google Scholar
  84. U.S. Department of Education. (2011). Elementary and secondary education: Part B—21st Century Community Learning Centers. http://www2.ed.gov/policy/elsec/leg/esea02/pg55.html.
  85. U.S. Department of Education. (2014). Programs: 21st Century Community Learning Centers. http://www2.ed.gov/programs/21stcclc/index.html.
  86. Vandell, D. L., Shernoff, D. J., Pierce, K. M., Bolt, D. M., Dadisman, K., & Brown, B. B. (2005). Activities, engagement, and emotion in after-school programs (and elsewhere). New Directions for Youth Development, 105, 121–129.PubMedCrossRefGoogle Scholar
  87. *Weisman, S. A., Womer, S. C., Kellstrong, M., Bryner, S., Kahler, A., Slocum, L. A., et al. (2003). Maryland After-School Community Grant Program Part 1: Report on the 2002–2003 school year evaluation of the phase 3 after-school programs. College Park, MD: University of Maryland.Google Scholar
  88. *Welsh, M. E., Russell, C. A., Williams, I., Reisner, E. R., & White, R. N. (2002). Promoting learning and school attendance through after school programs: Student level changes in education across TASC’s first three years. Washington, D. C.: Policy Studies Associates Inc.Google Scholar
  89. Zief, S.G., Lauver, S., & Maynard, R.A. (2006). Impacts of after-school programs on student outcomes. Campbell Systematic Reviews.Google Scholar

Copyright information

© Springer Science+Business Media New York 2014

Authors and Affiliations

  • Kristen P. Kremer
    • 1
  • Brandy R. Maynard
    • 1
  • Joshua R. Polanin
    • 2
  • Michael G. Vaughn
    • 1
  • Christine M. Sarteschi
    • 3
  1. 1.School of Social Work, College for Public Health and Social JusticeSaint Louis UniversitySt. LouisUSA
  2. 2.Peabody Research InstituteVanderbilt UniversityNashvilleUSA
  3. 3.Department of Social Work and CriminologyChatham UniversityPittsburghUSA

Personalised recommendations