Skip to main content
SpringerLink
Log in

Tracking Time and Resources Associated with Systems Change and the Adoption of Evidence-Based Programs: The “Hidden Costs” of School-Based Coaching

  • Original Paper
  • Published:
Administration and Policy in Mental Health and Mental Health Services Research Aims and scope Submit manuscript

Abstract

This study leveraged data from a 40-school randomized controlled trial to understand the cost of coaching to support implementation of evidence-based programs (EBPs) through a multi-tiered system of supports for behavior (MTSS-B) model. Coach activity log data were utilized to generate the annual average, per school, costs of coaching of $8198. The cost of school personnel time for coaching was estimated to be $3028. Data on coach-rated administrator buy-in, school MTSS-B engagement, and implementation infrastructure and capacity were also collected and found to be associated with coaching activities. Notably, coaches did not spend significantly different amounts of time in schools using few EBPs relative to more EBPs, indicating some inefficiency in the use of coaches’ time. These findings highlight the often-overlooked resources needed to support EBP implementation in schools.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1

Similar content being viewed by others

References

  • Aarons, G. A., Green, A. E., Trott, E., Willging, C. E., Torres, E. M., Ehrhart, M. G., et al. (2016). The roles of system and organizational leadership in system-wide evidence-based intervention sustainment: A mixed-method study. Administration and Policy in Mental Health and Mental Health Services Research,43, 991–1008. https://doi.org/10.1007/s10488-016-0751-4.

    Article  PubMed  Google Scholar 

  • Barrett, S. B., Bradshaw, C. P., & Lewis-Palmer, T. (2008). Maryland statewide PBIS initiative: Systems, evaluation, and next steps. Journal of Positive Behavior Interventions,10(2), 105–114. https://doi.org/10.1177/1098300707312541.

    Article  Google Scholar 

  • Barrett, S. B., Eber, L., & Weist, M. (2012). Advancing education effectiveness: Interconnecting school mental health and school-wide positive behavior support. Retrieved from https://www.pbis.org/resource/advancing-education-effectiveness-interconnecting-school-mental-health-and-school-wide-positive-behavior-support.

  • Belfield, C., Bowden, A. B., Klapp, A., Levin, H., Shand, R., & Zander, S. (2015). The economic value of social and emotional learning. Journal of Benefit to Cost Analysis,6(3), 508–544. https://doi.org/10.1017/bca.2015.55.

    Article  Google Scholar 

  • Botvin, G. J., Griffin, K. W., & Nichols, T. R. (2006). Preventing youth violence and delinquency through a universal school-based prevention approach. Prevention Science,7(4), 403–408. https://doi.org/10.1007/s11121-006-0057-y.

    Article  PubMed  Google Scholar 

  • Bowden, A. B., Shand, R., Belfield, C. R., Wang, A., & Levin, H. M. (2017). Evaluating educational interventions that induce service receipt: A case study application of City Connects. American Journal of Evaluation,38(3), 405–419. https://doi.org/10.1177/1098214016664983.

    Article  Google Scholar 

  • Bradshaw, C. P., Debnam, K. J., Koth, C. W., & Leaf, P. J. (2009a). Preliminary validation of the Implementation Phases Inventory for assessing fidelity of schoolwide positive behavior supports. Journal of Positive Behavior Interventions,11(3), 145–160. https://doi.org/10.1177/1098300708319126.

    Article  Google Scholar 

  • Bradshaw, C. P, Debnam, K. J., Lindstrom Johnson, S., Pas, E. T., Hershfeldt, P., Alexander, A., … & J Leaf, P. (2014). Maryland's evolving system of social, emotional, and behavioral interventions in public schools: The Maryland Safe and Supportive Schools Project. Adolescent Psychiatry, 4(3), 194-206. https://doi.org/10.2174/221067660403140912163120.

  • Bradshaw, C., Debnam, K., Player, D., Bowden, A. B., & Lindstrom Johnson, S. (2020a). A multi-method approach for embedding cost-analysis within fidelity assessment in school-based programs. Manuscript submitted for publication.

  • Bradshaw, C. P., Koth, C. W., Bevans, K. B., Ialongo, N. S., & Leaf, P. J. (2008). The impact of school-wide Positive Behavioral Interventions and Supports (PBIS) on the organizational health of elementary schools. School Psychology Quarterly,23(4), 462–473. https://doi.org/10.1177/1098300709334798.

    Article  Google Scholar 

  • Bradshaw, C. P., Koth, C. W., Thornton, L. A., & Leaf, P. J. (2009b). Altering school climate through school-wide Positive Behavioral Interventions and Supports: Findings from a group-randomized effectiveness trial. Prevention Science,10(2), 100–115. https://doi.org/10.1007/s11121-008-0114-9.

    Article  PubMed  Google Scholar 

  • Bradshaw, C. P., Pas, E. T., Bottiani, J., Debnam, K. J., Reinke, W., Herman, K., et al. (2018). Promoting cultural responsivity and student engagement through Double Check coaching of classroom teachers: An efficacy study. School Psychology Review,47(2), 118–134. https://doi.org/10.17105/SPR-2017-0119.V47-2.

    Article  Google Scholar 

  • Bradshaw, C. P, Pas, E. T., Debnam, K. J., & Lindstrom Johnson, S. (2020b). Preventing behavioral and mental health problems among high schoolers through improved school climate and social-emotional learning: The Maryland Safe and Supportive Schools trial. Manuscript submitted for publication.

  • Bradshaw, C. P., Pas, E. T., Goldweber, A., Rosenberg, M., & Leaf, P. J. (2012). Integrating School-wide Positive Behavioral Interventions and Supports with tier 2 coaching to student support teams: The PBISplus model. Advances in School Mental Health Promotion,5(3), 177–193. https://doi.org/10.1080/1754730X.2012.707429.

    Article  Google Scholar 

  • Bradshaw, C. P., Waasdorp, T. E., & Leaf, P. J. (2012). Effects of school-wide positive behavioral interventions and supports on child behavior problems. Pediatrics,130(5), e1136–e1145. https://doi.org/10.1542/peds.2012-0243.

    Article  PubMed  PubMed Central  Google Scholar 

  • Chorpita, B. F., & Daleiden, E. L. (2010). Building evidence-based systems in children's mental health. In J. R. Weisz & A. E. Kazdin (Eds.), Evidence-based psychotherapies for children and adolescents (pp. 482–499). New York: The Guilford Press.

    Google Scholar 

  • Cook, C. R., Lyon, A. R., Kubergovic, D., Browning Wright, D., & Zhang, Y. (2015). A supportive beliefs intervention to facilitate the implementation of evidence-based practices within a multi-tiered system of supports. School Mental Health,7(1), 49–60. https://doi.org/10.1007/s12310-014-9139-3.

    Article  Google Scholar 

  • Crowley, D. M., Dodge, K. A., Barnett, W. S., Corso, P., Duffy, S., Graham, P., … Plotnick, R. (2018). Standards of evidence for conducting and reporting economic evaluations in prevention science. Prevention Science, 19(3), 366-390. https://doi.org/10.1007/s11121-017-0858-1.

  • Dane, A. V., & Schneider, B. H. (1998). Program integrity in primary and early secondary prevention: Are implementation effects out of control? Clinical Psychology Review,18(1), 23–45. https://doi.org/10.1016/s0272-7358(97)00043-3.

    Article  CAS  PubMed  Google Scholar 

  • Domitrovich, C. E., Bradshaw, C. P., Poduska, J. M., Hoagwood, K. E., Buckley, J. A., Olin, S., … Ialongo, N. S. (2008). Maximizing the implementation quality of evidence-based preventive interventions in schools: A conceptual framework. Advances in School Mental Health Promotion, 1(3), 6-28. https://doi.org/10.1080/1754730x.2008.9715730.

  • Eagle, J. W., Dowd-Eagle, S. E., Snyder, A., & Gibbons Holtzman, E. (2015). Implementing a Multi-Tiered System of Support (MTSS): Collaboration between school psychologists and administrators to promote systems-level change. Journal of Educational and Psychological Consultation,25(2–3), 160–177. https://doi.org/10.1080/10474412.2014.929960.

    Article  Google Scholar 

  • Erchul, W. P. (2011). School consultation and response to intervention: A tale of two literatures. Journal of Educational and Psychological Consultation,21(3), 191–208. https://doi.org/10.1080/10474412.2011.595198.

    Article  Google Scholar 

  • Fagan, A. A., Bumbarger, B. K., Barth, R. P., Bradshaw, C. P., Rhoades Cooper, B., Supplee, L. H., et al. (2019). Scaling up evidence-based interventions in US public systems to prevent behavioral health problems: Challenges and opportunities. Prevention Science,20(8), 1147–1168. https://doi.org/10.1007/s11121-019-01048-8.

    Article  PubMed  PubMed Central  Google Scholar 

  • Findorff, M. J., Wyman, J. F., Croghan, C. F., & Nyman, J. A. (2005). Use of time studies for determining intervention costs. Nursing Research,54(4), 280–284. https://doi.org/10.1097/00006199-200507000-00011.

    Article  PubMed  Google Scholar 

  • Fixsen, D. L., & Blasé, K. A. (2008). Drivers framework. Chapel Hill, NC: University of North Carolina, The National Implementation Research Network, Frank Porter Graham Child Development Institute.

  • Fixsen, D. L., Naoom, S. F., Blasé, K. A., Friedman, R. M., & Wallace, F. (2005). Implementation research: A synthesis of the literature. Tampa: University of South Florida, Louis de la Parte Florida Mental Health Institute, The National Implementation Research Network. Retrieved from https://nirn.fpg.unc.edu/sites/nirn.fpg.unc.edu/files/resources/NIRN-MonographFull-01-2005.pdf.

  • Fullan, M., & Knight, J. (2011). Coaches as systems leaders. Educational Leadership,69(2), 50–53.

    Google Scholar 

  • Gregory, A., Huang, F. L., Anyon, Y., Greer, E., & Downing, B. (2018). An examination of restorative interventions and racial equity in out-of-school suspensions. School Psychology Review,47(2), 167–182. https://doi.org/10.17105/SPR-2017-0073.V47-2.

    Article  Google Scholar 

  • Gutkin, T. B., & Curtis, M. J. (2009). School-based consultation: The science and practice of indirect service delivery. In T. B. Gutkin & C. R. Reynolds (Eds.), The handbook of school psychology (4th ed., pp. 591–635). Hoboken, NJ: Wiley.

    Google Scholar 

  • Hicks, T. B., Shahidullah, J. D., Carlson, J. S., & Palejwala, M. H. (2014). Nationally Certified School Psychologists' use and reported barriers to using evidence-based interventions in schools: The influence of graduate program training and education. School Psychology Quarterly,29(4), 469–487. https://doi.org/10.1037/spq0000059.

    Article  PubMed  Google Scholar 

  • Hollands, F., Pan, Y., & Escueta, M. (2019). What is the potential for applying cost-utility analysis to facilitate evidence-based decision making in schools? Educational Researcher,48(5), 287–295. https://doi.org/10.3102/0013189X19852101.

    Article  Google Scholar 

  • Horner, R. H., Kincaid, D., Sugai, G., Lewis, T. J., Eber, L., Barrett, S., … Johnson, N. (2014). Scaling up School-Wide Positive Behavioral Interventions and Supports: Experiences of seven states with documented success. Journal of Positive Behavior Interventions, 16(4), 197-208. https://doi.org/10.1177/1098300713503685.

  • Horner, R. H., Sugai, G., & Anderson, C. M. (2010). Examining the evidence base for schoolwide positive behavior support. Focus on Exceptional Children,42(8), 1–14. https://doi.org/10.17161/fec.v42i18.6906.

    Article  Google Scholar 

  • Horner, R. H., Sugai, G., Smolkowski, K., Eber, L., Nakasato, J., Todd, A. W., et al. (2009). A randomized, wait-list controlled effectiveness trial assessing school-wide Positive Behavior Support in elementary schools. Journal of Positive Behavior Interventions,11(3), 133–144. https://doi.org/10.1177/1098300709332067.

    Article  Google Scholar 

  • Knight, D. S. (2012). Assessing the cost of instructional coaching. Journal of Education Finance,38(1), 52–80. https://doi.org/10.1353/jef.2012.0010.

    Article  Google Scholar 

  • Kraft, M. A., Blazar, D., & Hogan, D. (2018). The effect of teacher coaching on instruction and achievement: A meta-analysis of the causal evidence. Review of Educational Research,88(4), 547–588. https://doi.org/10.3102/0034654318759268.

    Article  Google Scholar 

  • Lee, A., & Gage, N. A. (2020). Updating and expanding systematic reviews and meta-analyses on the effects of school-wide positive behavior interventions and supports. Psychology in the Schools. Advance online publication. https://doi.org/10.1002/pits.22336.

  • Levin, H. M., McEwan, P. J., Belfield, C., Bowden, A. B., & Shand, R. (2017). Economic evaluation in education: Cost-effectiveness and benefit-cost analysis. Thousand Oaks: SAGE Publications.

    Google Scholar 

  • Lewis-Palmer, T., Todd, A. W., Horner, R. H., Sugai, G., & Sampson, N. K. (2005). Individual student systems evaluation tool, version 1.2. Eugene, OR: Educational and Community Supports, University of Oregon.

    Google Scholar 

  • Lochman, J. E., & Wells, K. C. (2002). Contextual social-cognitive mediators and child outcome: A test of the theoretical model in the coping power program. Development and Psychopathology,14(4), 945–967. https://doi.org/10.1017/s0954579402004157.

    Article  PubMed  Google Scholar 

  • Lochman, J. E., & Wells, K. C. (2003). Effectiveness of the coping power program and of classroom intervention with aggressive children: Outcomes of a 1-year follow-up. Behavior Therapy,34(4), 493–515. https://doi.org/10.1016/S0005-7894(03)80032-1.

    Article  Google Scholar 

  • Lyon, A. R., Cook, C. R., Brown, E. C., Locke, J., Davis, C., Ehrhart, M., et al. (2018). Assessing organizational implementation context in the education sector: Confirmatory factor analysis of measures of implementation leadership, climate, and citizenship. Implementation Science,13(1), 5. https://doi.org/10.1186/s13012-017-0705-6.

    Article  PubMed  Google Scholar 

  • March, A. L., Castillo, J. M., Batsche, G. M., & Kincaid, D. (2016). Relationship between systems coaching and problem-solving implementation fidelity in a Response-to-Intervention model. Journal of Applied School Psychology,32(2), 147–177. https://doi.org/10.1080/15377903.2016.1165326.

    Article  Google Scholar 

  • March, A. L., & Gaunt, B. T. (2013). Systems coaching: A model for building capacity. Tampa, FL: University of South Florida, Florida’s Multitiered System of Supports Project. Retrieved from https://www.floridarti.usf.edu/resources/format/pdf/SystemsCoaching.pdf.

  • McCamish, C., Reynolds, H., Algozzine, B., & Cusumano, D. (2015). An investigation of characteristics, practices, and leadership styles of PBIS coaches. Journal of Applied Educational and Policy Research, 1(1), 15–34. Retrieved from https://fpg.unc.edu/sites/fpg.unc.edu/files/resources/journal-articles/McCamish%20Reynolds%20Algozzine%20Cusumano%202015.pdf.

  • Meyers, A. B., Meyers, J., Graybill, E. C., Proctor, S. L., & Huddleston, L. (2012). Ecological approaches to organizational consultation and systems change in educational settings. Journal of Educational and Psychological Consultation,22(1–2), 106–124. https://doi.org/10.1080/10474412.2011.649649.

    Article  Google Scholar 

  • O’Connell, M. E., Boat, T., & Warner, K. E. (2009). Preventing mental, emotional, and behavioral disorders among young people: Progress and possibilities. Washington, DC: The National Academies Press. Committee on the Prevention of Mental Disorders and Substance Abuse Among Children, Youth, and Young Adults: Research Advances and Promising Interventions.

  • Pas, E. T., Larson, K., Reinke, W., Herman, K., & Bradshaw, C. P. (2016). Implementation and acceptability of an adapted Classroom Check-Up coaching model to promote culturally-responsive classroom management. The Education & Treatment of Children,39(4), 467–492. https://doi.org/10.1353/etc.2016.0021.

    Article  Google Scholar 

  • Pas, E. T., Ryoo, J. H., Musci, R., & Bradshaw, C. P. (2019). The effects of a state-wide scale-up of school-wide Positive Behavior Intervention and Supports on behavioral and academic outcomes: A quasi-experimental examination. Journal of School Psychology,73(1), 41–55. https://doi.org/10.1016/j.jsp.2019.03.001.

    Article  PubMed  Google Scholar 

  • Real, K., & Poole, M. S. (2005). Innovation implementation: Conceptualization and measurement in organizational research. In R. Woodman & W. Pasmore (Eds.), Research in organizational change and development (Vol. 15, pp. 63–135). Amsterdam: Elsevier/JAI.

    Chapter  Google Scholar 

  • Reinke, W. M., Herman, K., & Sprick, R. (2011). Motivational interviewing for effective classroom management: The Classroom Check-Up. New York, NY: The Guilford Press.

    Google Scholar 

  • Scott, T. M., Gage, N. A., Hirn, R. G., Shearer Lingo, A., & Burt, J. (2019). An examination of the association between MTSS implementation fidelity measures and student outcomes. Preventing School Failure,63(4), 308–316. https://doi.org/10.1080/1045988X.2019.1605971.

    Article  Google Scholar 

  • Sinclair, M. F., Christenson, S. L., & Thurlow, M. L. (2005). Promoting school completion of urban secondary youth with emotional or behavioral disabilities. Exceptional Children,71(4), 465–482. https://doi.org/10.1177/001440290507100405.

    Article  Google Scholar 

  • Sprick, R., Knight, J., Reinke, W. M., & McKale, T. (2006). Coaching for positive classrooms: Supporting teachers with classroom management. Eugene, OR: Pacific Northwest Publishing.

    Google Scholar 

  • Stormont, M., Reinke, W., & Herman, K. (2011). Teachers’ knowledge of evidence-based interventions and available school resources for children with emotional and behavioral problems. Journal of Behavioral Education,20(2), 138–147. https://doi.org/10.1007/s10864-011-9122-0.

    Article  Google Scholar 

  • Strein, W., Hoagwood, K., & Cohn, A. (2003). School psychology: A public health perspective: Prevention, populations, and system change. Journal of School Psychology,41(1), 23–38. https://doi.org/10.1016/S0022-4405(02)00142-5.

    Article  Google Scholar 

  • Sugai, G., & Horner, R. H. (2002). The evolution of discipline practices: School-wide positive behavior supports. Child & Family Behavior Therapy,24(1–2), 23–50. https://doi.org/10.1300/J019v24n01_03.

    Article  Google Scholar 

  • Sugai, G., & Horner, R. H. (2006). A promising approach for expanding and sustaining School-Wide Positive Behavior Support. School Psychology Review, 35(2), 245–259. Retrieved from https://www.icareby.org/sites/www.icareby.org/files/spr352sugai.pdf.

  • Sugai, G., Lewis-Palmer, T., Todd, A., & Horner, R. H. (2001). School-wide evaluation tool. PBIS Technical Assistance Center. Eugene, OR: University of Oregon. Retrieved from https://www.pbisapps.org/Resources/SWIS%20Publications/School-Wide%20Evaluation%20Tool%20(SET).pdf.

  • U.S. Bureau of Labor Statistics (2018a). Consumer Price Index (CPI) Inflation Calculator. Retrieved from https://data.bls.gov/cgi-bin/cpicalc.pl.

  • U.S. Bureau of Labor Statistics (2018b). Employer Costs for Employee Compensation for the Regions—December 2018, U.S. North East Data. Retrieved from https://www.bls.gov/regions/southwest/news-release/employercostsforemployeecompensation_regions.htm.

  • U.S. Bureau of Labor Statistics (2018c). Occupational Employment Statistics: Occupational Employment and Wages, May 2018. Retrieved from https://www.bls.gov/oes/current/oes252022.htm#(3).

  • Wandersman, A., Duffy, J., Flaspohler, P., Noonan, R., Lubell, K., Stillman, L., … Saul, J. (2008). Bridging the gap between prevention research and practice: The Interactive Systems Framework for Dissemination and Implementation. American Journal of Community Psychology, 41(3–4), 171-181. https://doi.org/10.1007/s10464-008-9174-z.

Download references

Acknowledgements

We would like to thank the six coaches who served on the MDS3 project; Sandra Hardee, Christina Knepper, Kristine Larson, LaKisha Liggins, Stephanie Parkhurst, and Rebecca Piermattei. We also thank Lauren Kaiser for assistance in the development and training for the systems coaching and Wendy Reinke for providing consultation regarding the adaptation of the Classroom Check-up for this project.

Funding

Support for this research study comes from Award No. 2014-CK-BX-005, awarded by the National Institute of Justice, Office of Justice Programs, U.S. Department of Justice and from the Institute of Education Sciences (R305H150027) awarded to Dr. Catherine Bradshaw. The opinions, findings, and conclusions or recommendations expressed in this publication/program/exhibition are those of the author(s) and do not necessarily reflect those of the Department of Justice or the Institute of Education Sciences.

Author information

Authors and Affiliations

  1. Bloomberg School of Public Health, Johns Hopkins University, 415 N. Washington Street, Baltimore, MD, 21231, USA

    Elise T. Pas & Y. Natalia Alfonso

  2. T. Denny Sanford School of Social and Family Dynamics, Arizona State University, Tempe, AZ, USA

    Sarah Lindstrom Johnson

  3. Curry School of Education and Human Development, University of Virginia, Charlottesville, VA, USA

    Catherine P. Bradshaw

Authors
  1. Elise T. Pas
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Sarah Lindstrom Johnson
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Y. Natalia Alfonso
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Catherine P. Bradshaw
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Elise T. Pas.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Ethical Approval

No identifying data were collected as part of this study and thus was not considered human subjects research.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Pas, E.T., Lindstrom Johnson, S., Alfonso, Y.N. et al. Tracking Time and Resources Associated with Systems Change and the Adoption of Evidence-Based Programs: The “Hidden Costs” of School-Based Coaching. Adm Policy Ment Health 47, 720–734 (2020). https://doi.org/10.1007/s10488-020-01039-w

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10488-020-01039-w

Keywords

Search

Navigation