An Initial Investigation of the Efficacy of the Resilience Education Program (REP)

  • Amanda N. Allen
  • Stephen P. Kilgus
  • Katie Eklund
Original Paper


Early intervention and prevention services for students at risk of emotional and behavioral problems are increasingly plentiful in the schools. However, recent reviews have revealed gaps in the literature regarding supports for students with internalizing problems. This gap has set precedent for the development of novel Tier 2 interventions that specifically address internalizing behavior problems. The primary purpose of this study was to investigate the efficacy of a novel, targeted intervention for elementary students who are at risk of internalizing problems: the Resilience Education Program (REP). REP is founded upon a social–ecological framework, integrating small-group cognitive–behavioral instruction with a modified check-in/check-out procedure. The current study employed a multiple-baseline single-case design with three student participants at risk of internalizing concerns. Visual analysis of direct observation data and single-case effect size estimates indicated that the REP intervention resulted in the decrease in internalizing behaviors and an increase in social engagement relative to baseline levels for two of three participants. Findings were further supported in that the changes noted in observational data were also partially confirmed by teacher ratings. Limitations of this study and future directions for researchers and practitioners are discussed.


Internalizing behavior Cognitive–behavioral intervention School mental health Check-in/check-out 



This research was funded through a University of Missouri Research Council Grant (Award Number: URC-16-109).

Compliance with Ethical Standards

Conflict of interest

The authors declare that they have no conflict of interest and confirm that the contents of the manuscript and the associated investigation are and were consistent with the APA Ethical Principles of Psychologists and Code of Conduct.

Ethical Approval

All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional research committee and with the 1964 Helsinki Declaration and its later amendments or comparable ethical standards.

Informed Consent

Informed consent was obtained from all individual participants included in the study.


  1. Balázs, J., Miklósi, M., Keresztény, Á., Hoven, C. W., Carli, V., Wasserman, C., et al. (2013). Adolescent subthreshold-depression and anxiety: Psychopathology, functional impairment and increased suicide risk. Journal of Child Psychology and Psychiatry, 54, 670–677.CrossRefPubMedGoogle Scholar
  2. Benson, P. L., Leffert, N., Scales, P. C., & Blyth, D. A. (1998). Beyond the “village” rhetoric: Creating healthy communities for children and adolescents. Applied Developmental Science, 2, 138–159.CrossRefGoogle Scholar
  3. Briesch, A. M., Chafouleas, S. M., Neugebauer, S. R., & Riley-Tillman, T. C. (2013). Assessing influences on intervention implementation: Revision of the usage rating profile-intervention. Journal of School Psychology, 51, 81–96.CrossRefPubMedGoogle Scholar
  4. Busk, P. L., & Serlin, R. C. (1992). Meta-analysis for single- case research. In T. R. Kratochwill & J. R. Levin (Eds.), Single-case research design and analysis: New directions for psychology and education (pp. 187–212). Hillsdale, NJ: Lawrence Erlbaum.Google Scholar
  5. Carrellas, N. W., Biederman, J., & Uchida, M. (2016). How prevalent and morbid are subthreshold manifestations of major depression in adolescents? A literature review. Journal of Affective Disorders, 210, 166–173.CrossRefPubMedGoogle Scholar
  6. Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20, 37–46. CrossRefGoogle Scholar
  7. Cook, C. R., Rasetshwane, K. B., Truelson, E., Grant, S., Dart, E. H., Collins, T. A., & Sprague, J. (2011). Development and validation of the Student Internalizing Behavior Screener: Examination of reliability, validity, and classification accuracy. Assessment for Effective Intervention, 36, 71–79.CrossRefGoogle Scholar
  8. Cook, C. R., Volpe, R. J., & Livanis, A. (2010). Constructing a roadmap for future universal screening research beyond academics. Assessment for Effective Intervention, 35, 197–205.CrossRefGoogle Scholar
  9. Cook, C. R., Xie, S. R., Earl, R. K., Lyon, A. R., Dart, E., & Zhang, Y. (2015). Evaluation of the courage and confidence mentor program as a Tier 2 intervention for middle school students with identified internalizing problems. School Mental Health, 7, 132–146.CrossRefGoogle Scholar
  10. Crone, D. A., Hawken, L. S., & Horner, R. H. (2010). Responding to problem behavior in schools: The behavior education program. New York: Guilford Press.Google Scholar
  11. Dart, E. H., Furlow, C. M., Collins, T. A., Brewer, E., Gresham, F. M., & Chenier, K. H. (2015). Peer-mediated check-in/check-out for students at-risk for internalizing disorders. School Psychology Quarterly, 30, 229.CrossRefPubMedGoogle Scholar
  12. Dill, E. J., Vernberg, E. M., Fonagy, P., Twemlow, S. W., & Gamm, B. K. (2004). Negative affect in victimized children: The roles of social withdrawal, peer rejection, and attitudes toward bullying. Journal of Abnormal Child Psychology, 32, 159–173.CrossRefPubMedGoogle Scholar
  13. DiPerna, J. C. (2006). Academic enablers and student achievement: Implications for assessment and intervention services in the schools. Psychology in the Schools, 43, 7–17.CrossRefGoogle Scholar
  14. Farmer, E. M., Burns, B. J., Phillips, S. D., Angold, A., & Costello, E. J. (2003). Pathways into and through mental health services for children and adolescents. Psychiatric Services, 54, 60–66.CrossRefPubMedGoogle Scholar
  15. Fergusson, D. M., Horwood, L. J., Ridder, E. M., & Beautrais, A. L. (2005). Subthreshold depression in adolescence and mental health outcomes in adulthood. Archives of General Psychiatry, 62, 66–72.CrossRefPubMedGoogle Scholar
  16. Gambone, M. A., Klem, A. M., Summers, J. A., Akey, T. A., & Sipe, C. L. (2004). Turning the tide: The achievements of the First Things First education reform in the Kansas City Kansas Public School District. Philadelphia: Youth Development Strategies Inc.Google Scholar
  17. Goodman, M. R., Stormshak, E. A., & Dishion, T. J. (2001). The significance of peer victimization at two points in development. Journal of Applied Developmental Psychology, 22, 507–526.CrossRefGoogle Scholar
  18. Gosch, E. A., Flannery-Schroeder, E., Mauro, C. F., & Compton, S. N. (2006). Principles of cognitive-behavioral therapy for anxiety disorders in children. Journal of Cognitive Psychotherapy, 20, 247–262.CrossRefGoogle Scholar
  19. Greenberg, M. T., Weissberg, R. P., O’Brien, M. U., Zins, J. E., Fredericks, L., Resnick, H., et al. (2003). Enhancing school-based prevention and youth development through coordinated social, emotional, and academic learning. American Psychologist, 58, 466–474.CrossRefPubMedGoogle Scholar
  20. Gresham, F. M., & Kern, L. (2004). Internalizing behavior problems in children and adolescents. In R. B. Rutherford, M. M. Quinn, & S. R. Mathur (Eds.), Handbook of research in emotional and behavioral disorders (pp. 262–281). New York, NY: Guilford Press.Google Scholar
  21. Herman, K. C., Merrell, K. W., Reinke, W. M., & Tucker, C. M. (2004). The role of school psychology in preventing depression. Psychology in the Schools, 41, 763–775.CrossRefGoogle Scholar
  22. Herman, K. C., Reinke, W. M., Parkin, J., Traylor, K. B., & Agarwal, G. (2009). Childhood depression: Rethinking the role of the school. Psychology in the Schools, 46, 433–446.CrossRefGoogle Scholar
  23. Hill, R. M., Pettit, J. W., Lewinsohn, P. M., Seeley, J. R., & Klein, D. N. (2014). Escalation to major depressive disorder among adolescents with subthreshold depressive symptoms: Evidence of distinct subgroups at risk. Journal of Affective Disorders, 158, 133–138.CrossRefPubMedPubMedCentralGoogle Scholar
  24. Hodges, E. V. E., & Perry, D. G. (1999). Personal and interpersonal antecedents and consequences of victimization by peers. Journal of Personality and Social Psychology, 76, 677–685.CrossRefPubMedGoogle Scholar
  25. Hunter, K. K., Chenier, J. S., & Gresham, F. M. (2014). Evaluation of Check In/Check Out for students with internalizing behavior problems. Journal of Emotional and Behavioral Disorders, 22, 135–148.CrossRefGoogle Scholar
  26. Hurlburt, M. S., Garland, A. F., Nguyen, K., & Brookman-Frazee, L. (2010). Child and family therapy process: Concordance of therapist and observational perspectives. Administration and Policy In Mental Health And Mental Health Services Research, 37, 230–244.CrossRefPubMedGoogle Scholar
  27. Kessler, R. C., Heeringa, S., Lakoma, M. D., Petukhova, M., Rupp, A. E., Schoenbaum, M., et al. (2008). Individual and societal effects of mental disorders on earnings in the United States: Results from the national comorbidity survey replication. American Journal of Psychiatry, 165, 703–711.CrossRefPubMedPubMedCentralGoogle Scholar
  28. Kilgus, S. P., Chafouleas, S. M., & Riley-Tillman, T. C. (2013). Development and initial validation of the Social and Academic Behavior Risk Screener for elementary grades. School Psychology Quarterly, 28, 210–226.CrossRefPubMedGoogle Scholar
  29. Kilgus, S. P., Dart, E., von der Embse, N. P., & Collins, T. (2017). Development and validation of the Internalizing Behavior Observation Protocol (IBOP). (Manuscript in preparation).Google Scholar
  30. Kilgus, S. P., Eklund, K., von der Embse, N. P., Taylor, C. N., & Sims, W. A. (2016). Psychometric defensibility of the Social, Academic, and Emotional Behavior Risk Screener (SAEBRS) Teacher Rating Scale and multiple gating procedure within elementary and middle school samples. Journal of School Psychology, 58, 21–39.CrossRefPubMedGoogle Scholar
  31. Kilgus, S. P., Reinke, W. M., & Jimerson, S. R. (2015). Understanding mental health intervention and assessment within a multi-tiered framework: Contemporary science, practice, and policy. School Psychology Quarterly, 30, 159.CrossRefPubMedGoogle Scholar
  32. Kilgus, S. P., Sims, W. A., von der Embse, N. P., & Riley-Tillman, T. C. (2015). Confirmation of models for interpretation and use of the Social and Academic Behavior Risk Screener (SABRS). School Psychology Quarterly, 30, 335–352.CrossRefPubMedGoogle Scholar
  33. Kilgus, S. P. & von der Embse, N. P. (2014). Unpublished technical manual of the social, academic, and emotional behavior risk screener.Google Scholar
  34. Klein, D. N., Shankman, S. A., Lewinsohn, P. M., & Seeley, J. R. (2009). Subthreshold depressive disorder in adolescents: Predictors of escalation to full-syndrome depressive disorders. Journal of the American Academy of Child and Adolescent Psychiatry, 48, 703–710.CrossRefPubMedPubMedCentralGoogle Scholar
  35. Lonigan, C. J., & Vasey, M. W. (2009). Negative affectivity, effortful control, and attention to threat-relevant stimuli. Journal of Abnormal Child Psychology, 37, 387–399.CrossRefPubMedGoogle Scholar
  36. Maggin, D. M., Zurheide, J., Pickett, K. C., & Baillie, S. J. (2015). A systematic evidence review of the check-in/check-out program for reducing student challenging behaviors. Journal of Positive Behavior Interventions, 17, 197–208.CrossRefGoogle Scholar
  37. Masten, A. S., Roisman, G. I., Long, J. D., Burt, K. B., Obradović, J., Riley, J. R., et al. (2005). Developmental cascades: Linking academic achievement and externalizing and internalizing symptoms over 20 years. Developmental Psychology, 41, 733.CrossRefPubMedGoogle Scholar
  38. Merikangas, K. R., He, J. P., Burstein, M., Swanson, S. A., Avenevoli, S., Cui, L., et al. (2010). Lifetime prevalence of mental disorders in US adolescents: Results from the National Comorbidity Survey Replication-Adolescent Supplement (NCS-A). Journal of the American Academy of Child and Adolescent Psychiatry, 49, 980–989.CrossRefPubMedPubMedCentralGoogle Scholar
  39. Merikangas, K. R., Zhang, H., Avenevoli, S., Acharyya, S., Neuenschwander, M., & Angst, J. (2003). Longitudinal trajectories of depression and anxiety in a prospective community study: The Zurich cohort study. Archives of General Psychiatry, 60, 993–1000.PubMedGoogle Scholar
  40. Merrell, K. W. (1995). An investigation of the relationship between social skills and internalizing problems in early childhood: Construct validity of the Preschool and Kindergarten Behavior Scales. Journal of Psychoeducational Assessment, 13, 230–240.CrossRefGoogle Scholar
  41. Mychailyszyn, M. P., Brodman, D. M., Read, K. L., & Kendall, P. C. (2012). Cognitive-behavioral school-based interventions for anxious and depressed youth: A meta-analysis of outcomes. Clinical Psychology: Science and Practice, 19, 129–153.Google Scholar
  42. National Research Council and Institute of Medicine. (2009). In M.E. O’Connell, T. Boat, & K. E. Warner (Eds.) Preventing mental, emotional, and behavioral disorders among young people: Progress and possibilities. Washington, DC: The National Academies Press.Google Scholar
  43. Parker, R. I., Hagan-Burke, S., & Vannest, K. (2007). Percentage of all non-overlapping data (PAND) an alternative to PND. Journal of Special Education, 40, 194–204.CrossRefGoogle Scholar
  44. Reynolds, C. R., & Kamphaus, R. W. (2015). Behavior assessment system for children—Third edition (BASC-3). Bloomington, MN: Pearson.Google Scholar
  45. Rudolph, K. D., & Clark, A. G. (2001). Conceptions of relationships in children with depressive and aggressive symptoms: Social-cognitive distortion or reality? Journal of Abnormal Child Psychology, 29, 41–56.CrossRefPubMedGoogle Scholar
  46. Ryan, R. M., & Deci, E. L. (2009). Promoting self determined school engagement: Motivation, learning, and well-being. In K. R. Wentzel & D. B. Miele (Eds.), Handbook of motivation at school (pp. 171–195). New York: Routledge.Google Scholar
  47. Scruggs, T. E., & Mastropieri, M. A. (1998). Summarizing single-subject research: Issues and applications. Behavior Modification, 22, 221–242.CrossRefPubMedGoogle Scholar
  48. Simonsen, B., Myers, D., & Briere, D. E., III. (2011). Comparing a behavioral check-in/check-out (CICO) intervention to standard practice in an urban middle school setting using an experimental group design. Journal of Positive Behavior Interventions, 13, 31–48.CrossRefGoogle Scholar
  49. Spence, S. H., & Rapee, R. M. (2016). The etiology of social anxiety disorder: An evidence-based model. Behaviour Research and Therapy, 86, 50–67.CrossRefPubMedGoogle Scholar
  50. U. S. Department of Health and Human Services, National Institutes of Health, National Institute of Mental Health. (2015). NIMH Strategic Plan for Research (NIH Publication No. 02-2650).
  51. Walker, H. M., Irvin, L. K., Noell, J., & Singer, G. H. S. (1992). A construct score approach to the assessment of social competence: Rationale, technological considerations, and anticipated outcomes. Behavior Modification, 16, 448–474.CrossRefGoogle Scholar
  52. Walker, H. M., & Severson, H. H. (1990). Systematic screening for behavior disorders. Longmont, CO: Sopris West.Google Scholar
  53. Youngblade, L. M., Theokas, C., Schulenberg, J., Curry, L., Huang, I. C., & Novak, M. (2007). Risk and promotive factors in families, schools, and communities: A contextual model of positive youth development in adolescence. Pediatrics, 119, S47–S53.CrossRefPubMedGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2018

Authors and Affiliations

  • Amanda N. Allen
    • 1
  • Stephen P. Kilgus
    • 1
  • Katie Eklund
    • 1
  1. 1.Department of Educational, School, and Counseling Psychology, 16 Hill HallUniversity of MissouriColumbiaUSA

Personalised recommendations