Numerous evidence-based programs (EBPs) exist for delivery in schools to promote youth mental health outcomes. However, school systems often lack the internal infrastructure to support the effective implementation and sustainment of EBPs when external supports are withdrawn, resulting in notable attenuation in the benefits in youth clinical outcomes that are associated with EBPs. This paper illustrates how to leverage concepts from improvement science and implementation science to develop learning school systems dedicated to enhancing the infrastructure capacity of a school to advance the implementation and sustainment of EBPs. In particular, we discuss how treatment integrity (extent to which an EBP is delivered as designed) and youth mental health outcome data are (a) collected, (b) analyzed and interpreted and (c) fed back into the school system to increase organizational supports and promote school practitioners’ behavior change to produce improvements in youth mental health outcomes. We also discuss psychological safety among the people within a school system as a key characteristic of a learning school system. We then present a theory-informed approach to learning school systems to demonstrate how data generated by a learning school system can lead to precise and effective plans that continuously improve implementation and result in the eventual sustainment of EBPs. We conclude with a brief research agenda with concrete steps toward realizing the potential of learning school systems to support the implementation and sustainment of EBPs for mental health problems.
This is a preview of subscription content, access via your institution.
Buy single article
Instant access to the full article PDF.
Tax calculation will be finalised during checkout.
Subscribe to journal
Immediate online access to all issues from 2019. Subscription will auto renew annually.
Tax calculation will be finalised during checkout.
Beidas, R. S., & Stirman, S. W. (2020). Realizing the promise of learning organizations to transform mental health care: Telepsychiatry care as an exemplar. Psychiatric Services. Advance online publication.
Bickman, L., Lyon, A. R., & Wolpert, M. (2016). Achieving precision mental health through effective assessment, monitoring and feedback processes. Administration and Policy in Mental Health and Mental Health Services Research, 43, 271–276.
Boyd, B. A., Odom, S. L., Humphreys, B. P., & Sam, A. M. (2010). Infants and toddlers with autism spectrum disorder: Early identification and early intervention. Journal of Early Intervention, 32, 75–98.
Bradley, R., Doolittle, J., & Bartolotta, R. (2008). Building on the data and adding to the discussion: The experiences and outcomes of students with emotional disturbance. Journal of Behavioral Education, 17(1), 4–23.
Brown, B., Gude, W. T., Blakeman, T., Van Der Veer, S. N., Ivers, N., Francis, J. J., et al. (2019). Clinical Performance Feedback Intervention Theory (CP-FIT): A new theory for designing, implementing and evaluating feedback in health care based on a systematic review and meta-synthesis of qualitative research. Implementation Science, 14(1), 1–25.
Bruns, E. J., Duong, M. T., Lyon, A. R., Pullmann, M. D., Cook, C. R., Cheney, D., & McCauley, E. (2016). Fostering SMART partnerships to develop an effective continuum of behavioral health services and supports in schools. American Journal of Orthopsychiatry, 86(2), 156–170.
Bunderson, J. S., & Boumgarden, P. (2010). Structure and learning in self-managed teams: Why “bureaucratic” teams can be better learners. Organizational Science, 21, 609–624.
Burnham, J. J., Mantero, M., & Hooper, L. M. (2009). Experiential counseling training: Connecting school counselors-in-training, ESL teachers and ESL. Journal of Multicultural Counseling & Human Development, 37, 2–14.
Carmeli, A., Brueller, D., & Dutton, J. E. (2009). Learning behaviours in the workplace: The role of high-quality interpersonal relationships and psychological safety. Systematic Review of Research in Behavioral Sciences, 26, 81–98.
Carmeli, A., Tishler, A., & Edmondson, A. C. (2012). CEO relational leadership and strategic decision quality in top management teams: The role of team trust and learning from failure. Strategic Organization, 10(1), 31–54.
Caron, E., & Dozier, M. (2019). Effects of fidelity-focused consultation on clinicians’ implementation: An exploratory multiple baseline design. Administration and Policy in Mental Health and Mental Health Services Research, 46(4), 445–457.
Chafouleas, S. M., Riley-Tillman, T. C., & Christ, T. J. (2009). Direct Behavior Rating (DBR): An emerging method for assessing behavior within a tiered intervention system. Assessment for Effective Intervention, 34, 157–173.
Chambers, D. A., Feero, W. G., & Khoury, M. J. (2016). Convergence of implementation science, precision medicine and the learning health care system: A new model for biomedical research. JAMA, 315(18), 1941–1942.
Chambers, D. A., & Norton, W. E. (2016). The Adaptome: Advancing the science of intervention adaptation. American Journal of Preventive Medicine, 51(4), S124–S131.
Chaney, E. F., Rubenstein, L. V., Liu, C. F., Yano, E. M., Bolkan, C., & Uman, J. (2011). Implementing collaborative care for depression treatment in primary care: A cluster randomized evaluation of a quality improvement practice redesign. Implementation Science, 6, 121.
Chapman, J. E., McCart, M. R., Letourneau, E. J., & Sheidow, A. J. (2013). Comparison of youth, caregiver, therapist, trained and treatment expert raters of therapist adherence to a substance abuse treatment protocol. Journal of Consulting & Clinical Psychology, 81, 674–680.
Check, D. K., Zullig, L. L., Davis, M. M., Davies, L., Chambers, D., Fleisher, L., Kaplan, S. J., Proctor, E., Ramanadhan, S., Schroeck, F. R., Stover, A. M., & Koczwara, B. (2020). Improvement science and implementation science in cancer care: Identifying areas of synergy and opportunities for further integration. Journal of General Internal Medicine. Advance online publication.
Connors, E., Lawson, G., Wheatley-Rowe, D., & Hoover, S. (2020). Exploration, preparation and implementation of standardized assessment in a multi-agency school behavioral health network. Administration and Policy in Mental Health and Mental Health Services Research. Advance online publication.
Cook, B. G., & Odom, S. (2013). Evidence-based practices and implementation science in special education. Exceptional Children, 79(2), 135–144.
Cook, C. R., Lyon, A. R., Locke, J., Waltz, T., & Powell, B. J. (2019). Adapting a compilation of implementation strategies to advance school-based implementation research and practice. Prevention Science, 20(6), 914–935.
Cook, C. R., Williams, K. R., Guerra, N. G., Kim, T. E., & Sadek, S. (2010). Predictors of bullying and victimization in childhood and adolescence: A meta-analytic investigation. School Psychology Quarterly, 25(2), 65–83.
Damschroder, L. J., Aron, D. C., Keith, R. E., Kirsh, S. R., Alexander, J. A., & Lowery, J. C. (2009). Fostering implementation of health services research findings into practice: A consolidated framework for advancing implementation science. Implementation Science, 4(1), 1–15.
Desocio, J., & Hootman, J. (2015). Feature article children’s mental health and school. The Journal of School Nursing, 20(4), 189–196.
Durlak, J. A., & Weissberg, R. P. (2011). Promoting social and emotional development is an essential part of students’ education. Human Development, 54(1), 1–3.
Edmondson, A. C. (2018). The fearless organization: Creating psychological safety in the workplace for learning, innovation and growth. Hoboken: Wiley.
Edmondson, A. C., Bohmer, R. M., & Pisano, G. P. (2001). Disrupted routines: Team learning and new technology implementation in hospitals. Administrative Science Quarterly, 46(4), 685–716.
Edmondson, A. C., Higgins, M., Singer, S., & Weiner, J. (2016). Understanding psychological safety in health care and education organizations: A comparative perspective. Research in Human Development, 13, 65–83.
Edmondson, A. C., & Lei, Z. (2014). Psychological safety: The history, renaissance and future of an interpersonal construct. Annual Review of Organizational Psychology and Organizational Behavior, 1, 23–43.
Elementary and Secondary Education Act as the Every Student Succeeds Act, (2015).
Foley, T., & Fairmichael, F. (2015). The potential of learning health systems. The Learning Healthcare Project.
French, S. D., Green, S. E., O’Connor, D. A., McKenzie, J. E., Francis, J. J., & Grimshaw, J. M. (2012). Developing theory-informed behaviour change interventions to implement evidence into practice: a systematic approach using the Theoretical Domains Framework. Implementation Science, 7(1), 1–8.
Garland, A., & Schoenwald, S. K. (2013). Use of effective and efficient quality control methods to implement psychosocial interventions. Clinical Psychology: Science and Practice, 20, 33–43.
Ghandour, R. M., Sherman, L. J., Vladutiu, C. J., Ali, M. M., Lynch, S. E., Bitsko, R. H., & Blumberg, S. J. (2019). Prevalence and treatment of depression, anxiety and conduct problems in U.S. children. Journal of Pediatrics, 206, 256–267.
Glasgow, R. E., & Riley, W. T. (2013). Pragmatic measures: What they are and why we need them. American Journal of Preventative Medicine, 45(2), 237–243.
Goodman, R. (2001). Psychometric properties of the strengths and difficulties questionnaire. Journal of the American Academy of Child and Adolescent Psychiatry, 40, 1337–1345.
Greenberg, M. T., & Abenavoli, R. (2017). Universal interventions: Fully exploring their impacts and potential to produce population-level impacts. Journal of Research on Educational Effectiveness, 10(1), 40–67.
Hargreaves, D. H. (1995). School culture, school effectiveness and school improvement. School Effectiveness and School Improvement, 6, 23–46.
Haynes, S. N., Richard, D. C., & Kubany, E. S. (1995). Content validity in psychological assessment: A functional approach to concepts and methods. Psychological Assessment, 7, 238–247.
Higgins, M., Ishimaru, A., Holcombe, R., & Fowler, A. (2012). Examining organizational learning in schools: The role of psychological safety, experimentation and leadership that reinforces learning. Journal of Educational Change, 13(1), 67–94.
Hogue, A., Bobek, M., Dauber, S., Henderson, C. E., McLeod, B. D., & Southam-Gerow, M. A. (2017). Distilling the core elements of family therapy for adolescent substance use : Conceptual and empirical solutions. Journal of Child & Adolescent Substance Abuse, 26(6), 437–453.
Hogue, A., Dauber, S., Lichvar, E., Bobek, M., & Henderson, C. E. (2015). Validity of therapist self-report ratings of fidelity to evidence-based practices for adolescent behavior problems: Correspondence between therapists and observers. Administration and Policy in Mental Health and Mental Health Services Research, 42(2), 229–243.
Hogue, A., Henderson, C. E., Dauber, S., Barajas, P. C., Fried, A., & Liddle, H. A. (2008). Treatment adherence, competence and outcome in individual and family therapy for adolescent behavior problems. Journal of Consulting and Clinical Psychology, 76(4), 544–555.
Hogue, A., Ozechowski, T. J., Robbins, M. S., & Waldron, H. B. (2013). Making fidelity an intramural game: Localizing quality assurance procedures to promote sustainability of evidence-based practices in usual care. Clinical Psychology: Science and Practice, 20, 60–77.
Horner, R. H., Newton, J. S., Todd, A. W., Algozzine, B., Algozzine, K., Cusumano, D., & Preston, A. J. (2018). A randomized waitlist controlled analysis of team-initiated problem solving professional development and use. Behavior Disorders, 43, 444–456.
Hsu, E. R., Klemm, J. D., Kerlavage, A. R., Kusnezov, D., & Kibbe, W. A. (2017). Cancer Moonshot data and technology team: Enabling a national learning healthcare system for cancer to unleash the power of data. Clinical Pharmacology and Therapeutics, 101(5), 613–615.
Institute of Medicine. (2015). The learning health system series.
Jensen Doss, A. (2005). Evidence-based diagnosis: Incorporating diagnostic instruments into clinical practice. Journal of the American Academy of Child & Adolescent Psychiatry, 44, 947–952.
Krause, J., Lieshout, J. V., Klomp, R., Huntink, E., Aakhus, E., & Baker, R. (2014). Identifying determinants of care for tailoring implementation in chronic diseases : an evaluation of different methods. Implementation Science, 9, 102.
Kroenke, K., & Spitzer, R. L. (2002). The PHQ-9: a new depression diagnostic and severity measure. Psychiatric Annuals, 32(9), 509–515.
Lane, K. L., & Menzies, H. M. (2009). Student Risk Screening Scale for Internalizing and Externalizing (SRSS-IE). Retrieved from http://www.ci3t.org/screening
Lewis, C. C., Klasnja, P., Powell, B. J., Lyon, A. R., Tuzzio, L., Jones, S., et al. (2018). From classification to causality: Advancing understanding of mechanisms of change in implementation science. Frontiers in Public Health, 6, 136.
Locke, J., Lawson, G. M., Beidas, R. S., Xie, M., Aarons, G. A., Spaulding, C., et al. (2019). Individual and organizational factors that affect implementation of evidence-based practices for children with autism in public schools. Implementation Science, 14, 39.
Locke, J., Olsen, A., Wideman, R., Downey, M. M., Kretzmann, M., Kasari, C., & Mandell, D. S. (2015). A tangled web: The challenges of implementing an evidence-based social engagement intervention for children with autism in urban public school settings. Behavior Therapy, 46(1), 54–67.
Lyon, A. R., & Bruns, E. J. (2019). From evidence to impact: Joining our best school mental health practices with our best implementation strategies. School Mental Health, 11(1), 106–114.
Lyon, A. R., Whitaker, K., Locke, J., Cook, C. R., King, K. M., Duong, M., & Aarons, G. A. (2018). The impact of inter-organizational alignment (IOA) on implementation outcomes: evaluating unique and shared organizational influences in education sector mental health. Implementation Science, 13, 123.
McHugh, R. K., & Barlow, D. H. (2010). The dissemination and implementation of evidence-based psychological treatments: A review of current efforts. American Psychologist, 65(2), 73–84.
McIntosh, K., Filter, K. J., Bennett, J. L., Ryan, C., & Sugai, G. (2010). Principles of sustainable prevention: Designing scale-up of school-wide positive behavior support to promote durable systems. Psychology in the Schools, 47, 55–21.
McLeod, B. D., Kunemund, R., Nemer, S., & Lyon, A. R. (2020). Leveraging implementation science and practice to support the delivery of evidence-based practices in services for youth with emotional and behavioral disorders. In T. W. Farmer, M. A. Conroy, E. M. Z., Farmer, & K. S. Sutherland (Eds), Handbook of research on emotional and behavioral disorders: Interdisciplinary developmental perspectives on children and youth (pp. 417-432). New York: Routledge/Taylor & Francis.
McLeod, B. D., Southam-Gerow, M. A., Tully, C. B., Rodríguez, A., Smith, M. M., Rodriguez, A., & Smith, M. M. (2013). Making a case for treatment integrity as a psychosocial treatment quality indicator for youth mental health care. Clinical Psychology: Science & Practice, 20(1), 14–32.
McLeod, B. D., Southam-Gerow, M. A., & Weisz, J. R. (2009). Conceptual and methodological issues in treatment integrity measurement. School Psychology Review, 38(4), 541–546.
Michie, S., Atkins, L., & West, R. (2014). The behaviour change wheel: A guide to designing interventions. Sutton: Silverback Publishing.
Moullin, J. C., Dickson, K. S., Stadnick, N. A., Rabin, B., & Aarons, G. A. (2019). Systematic review of the exploration, preparation, implementation, sustainment (EPIS) framework. Implementation Science, 14(1), 1.
Newman, A., Donohue, R., & Eva, N. (2017). Psychological safety: A systematic review of the literature. Human Resource Management Review, 27(3), 521–535.
Nilsen, P. (2015). Making sense of implementation theories, models and frameworks. Implementation Science, 10, 53.
Polanczyk, G. V., Salum, G. A., Sugaya, L. S., Caye, A., & Rohde, L. A. (2015). Annual Research Review: A meta-analysis of the worldwide prevalence of mental disorders in children and adolescents. Journal of Child Psychology & Psychiatry, 56(3), 345–365.
Powell, B. J., Waltz, T. J., Chinman, M. J., Damschroder, L. J., Smith, J. L., Matthieu, M. M., et al. (2015). A refined compilation of implementation strategies: results from the Expert Recommendations for Implementing Change (ERIC) project. Implementation Science, 10(1), 21.
Renshaw, T. L., & Cook, C. R. (2016). Initial development and validation of youth externalizing problems screener. Retrieved from http: //tylerrenshaw.com/measures.
Renshaw, T. L., & Cook, C. R. (2018). Initial development and validation of youth internalizing problems screener. Journal of Psychoeducational Assessment, 36(4), 366–378.
Sanetti, L. M., Charbonneau, S., Knight, A., Cochrane, W. S., Kulcyk, M. C. M., & Kraus, K. (2020). Treatment fidelity reporting in intervention outcome studies in the school psychology literature from 2009 to 2016. Psychology in the Schools, 57(6), 901–922.
Sanetti, L. M., & Collier-Meek, M. A. (2019). Increasing implementation science literacy to address the research-to-practice gap in school psychology. Journal of School Psychology, 76, 33.
Sanetti, L. M., Collier-Meek, M. A., Long, A. C., Byron, J., & Kratochwill, T. R. (2015). Increasing teacher treatment integrity of behavior support plans through consultation and implementation planning. Journal of School Psychology, 53, 209–229.
Sanetti, L. M. H., Gritter, K. L., & Dobey, L. M. (2011). Treatment integrity of interventions with children in the school psychology literature from 1995 to 2008. Journal of Positive Behavior Interventions, 40(1), 72–84.
Sanetti, L. M. H., & Kratochwill, T. R. (2009). Toward developing a science of treatment integrity: Introduction to the special series. School Psychology Review, 38(4), 445–459.
Scaccia, J. P., Cook, B. S., Lamont, A., Wandersman, A., Cateloow, J., Katz, J., & Beidas, R. S. (2015). A practical implementaiton science heuristic for organizational readiness: R = MC2. Journal of Community Psychology, 43(4), 484–501.
Schoenwald, S. K., Garland, A. F., Chapman, J. E., Frazier, S. L., Sheidow, A. J., & Southam-Gerow, M. A. (2011). Toward the effective and efficient measurement of implementation fidelity. Administration and Policy in Mental Health and Mental Health Services Research, 38(1), 32–43.
Scott, K., & Lewis, C. C. (2015). Using measurement-based care to enhance any treatment. Cognitive Behavioral and Practice, 22(1), 49–59.
Southam-Gerow, M. A., & McLeod, B. D. (2013). Advances in applying treatment integrity research for dissemination and implementation science: Introduction to special issue. Clinical Psychology: Science and Practice, 20(1), 1–13.
Stahmer, A. C., Rieth, S., Lee, E., Reisinger, E. M., Mandell, D. S., & Connell, J. E. (2015). Training teachers to use evidence-based practices for autism: Examining procedural implementation fidelity. Psychology in the Schools, 52, 181–195.
Stanick, C. F., Halko, H. M., Nolen, E. A., Powell, B. J., Dorsey, C. N., Mettert, K. D., & Lewis, C. C. (2019). Pragmatic measures for implementation research: development of the Psychometric and Pragmatic Evidence Rating Scale (PAPERS). Translational Behavioral Medicine, ibz164.
Suhrheinrich, J., Stahmer, A. C., Reed, S., Schreibman, L., Reisinger, E., & Mandell, D. S. (2013). Implementation challenges in translating pivotal response training into community settings. Journal of Autism and Developmental Disorders, 43(12), 2970–2976.
Sutherland, K. S., McLeod, B. D., Conroy, M. A., & Cox, J. R. (2013). Measuring implementation of evidence-based programs targeting young children at risk for emotional/behavioral disorders: Conceptual issues and recommendations. Journal of Early Intervention, 35, 129–149.
Tabak, R. G., Khoong, E. C., Chambers, D., & Brownson, R. C. (2012). Bridging research and practice: models for dissemination and implementation research. American Journal of Preventive Medicine, 43(3), 337–350.
Taylor, M. J., McNicholas, C., Nicolay, C., Darzi, A., Bell, D., & Reed, J. E. (2014). Systematic review of the application of the plan-do-study-act method to improve quality in healthcare. BMJ Quality and Safety, 23, 290–298.
Todd, A. W., Horner, R. H., Newton, J. S., Algozzine, R. E., Algozzine, K. M., & Frank, J. L. (2011). Effects of Team Initiated Problem Solving on meeting practices of schoolwide behavior support teams. Journal of Applied School Psychology, 27, 42–51.
Wang, M. T., & Degol, J. L. (2016). School climate: A review of the construct, measurement and impact on student outcomes. Educational Psychology Review, 28, 315–352.
Ware, N. C., Tugenberg, T., & Dickey, B. (2003). Ethnography and measurement in mental health: Qualitative validation of a measure of continuity of care (CONNECT). Qualitative Health Research, 13, 1393–1406.
Webb, C. A., DeRubeis, R. J., & Barber, J. P. (2010). Therapist adherence/competence and treatment outcome: A meta-analytic review. Journal of Consulting and Clinical Psychology, 78(2), 200–211.
Williams, N. J., & Beidas, R. S. (2019). Annual research review: The state of implementation science in child psychology and psychiatry: a review and suggestions to advance the field. Journal of Child Psychology and Psychiatry and Allied Disciplines, 60(4), 430–450.
Wilson, S. J., Tanner-Smith, E. E., Lipsey, M. W., Steinka-Fry, K., & Morrison, J. (2011). Dropout prevention and intervention programs: effects on school completion and dropout among school-aged children and youth. Campbell Systematic Reviews, 7(1), 1–61.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Preparation of this article was supported in part by grants from the Institute of Education Science (R305A170292, Cook & Lyon; R305A180182, Sutherland & Conroy).
About this article
Cite this article
McLeod, B.D., Cook, C.R., Sutherland, K.S. et al. A Theory-Informed Approach to Locally Managed Learning School Systems: Integrating Treatment Integrity and Youth Mental Health Outcome Data to Promote Youth Mental Health. School Mental Health (2021). https://doi.org/10.1007/s12310-021-09413-1
- Learning school system
- Treatment integrity
- Clinical outcomes