Skip to main content

Measuring Sustainment of Multiple Practices Fiscally Mandated in Children’s Mental Health Services


Examining sustainment of multiple evidence-based practices is crucial to understanding the processes and outcomes of system-driven implementation efforts that are increasingly common. This study used administrative claims data to characterize volume and penetration of six practices over 19 fiscal quarters following initial implementation within the context of a system-driven, fiscally mandated implementation effort in Los Angeles County Department of Mental Health’s Prevention and Early Intervention services. Patterns of volume changes over time revealed an overall ramp up of practice claims over time, but variability in patterns of volume and penetration for each practice. Findings varied by the methods used to index and analyze volume and penetration. Furthermore, a number of client case-mix and therapist characteristics were associated with the volume of therapists’ claims for each practice relative to their claims for the other practices.

This is a preview of subscription content, access via your institution.

Fig. 1
Fig. 2
Fig. 3


  1. 1.

    MAP trainings began in Quarter 2 of FY 2010–2011.

  2. 2.

    Inclusion of the total number of psychotherapy claims across the six practices per fiscal quarter as a repeated-measures covariate was considered but was not feasible due to high collinearity between total claim volume and time.


  1. Aarons, G. A., Hurlburt, M., & Horwitz, S. M. (2011). Advancing a conceptual model of evidence-based practice implementation in public service sectors. Adm Policy Ment Health, 38, 4–23. doi:10.1007/s10488-010-0327-7.

    Article  PubMed  Google Scholar 

  2. Beretvas, S. N. (2011). Cross-classified and multiple-membership models. In J. J. Hox & J. K. Roberts (Eds.), Handbook for advanced multilevel analysis (pp. 313–334). New York: Routledge.

    Google Scholar 

  3. Bond, G. R., Drake, R. E., McHugo, G. J., Peterson, A. E., Jones, A. M., & Williams, J. (2014). Long-term sustainability of evidence-based practices in community mental health agencies. Adm Policy Ment Health, 41, 228–236. doi:10.1007/s10488-012-0461-5.

    Article  PubMed  Google Scholar 

  4. Cafri, G., Hedeker, D., & Aarons, G. A. (2015). An introduction and integration of cross-classified, multiple membership, and dynamic group random-effects models. Psychol Methods,. doi:10.1037/met0000043.

    PubMed  Google Scholar 

  5. Chambers, D. A., & Rupp, A. (2015). Sharing state mental health data for research: building toward ongoing learning in mental health care systems. Adm Policy Ment Health, 42, 586–587. doi:10.1007/s10488-015-0624-2.

    Article  PubMed  Google Scholar 

  6. Chorpita, B. F., Bernstein, A., & Daleiden, E. L. (2011). Empirically guided coordination of multiple evidence-based treatments: an illustration of relevance mapping in children’s mental health services. J Consult Clin Psychol, 79, 470–480. doi:10.1037/a0023982.

    Article  PubMed  Google Scholar 

  7. Chorpita, B. F., & Daleiden, E. L. (2009). Mapping evidence-based treatments for children and adolescents: application of the distillation and matching model to 615 treatments from 322 randomized trials. J Consult Clin Psychol, 77, 566–579. doi:10.1037/a0014565.

    Article  PubMed  Google Scholar 

  8. Chorpita, B. F., Yim, L. M., Donkervoet, J. C., Arensdorf, A., Amundsen, M. J., McGee, C., & Morelli, P. (2002). Toward large-scale implementation of empirically supported treatments for children: a review and observations by the Hawaii empirical basis to services task force. Clin Psychol, 9, 165–190. doi:10.1111/j.1468-2850.2002.tb00504.x.

    Google Scholar 

  9. Cooper, J. L., Aratani, Y., Knitzer, J., Douglas-Hall, A., Masi, R., Banghart, P. L., Dababnah, S. (2008). Unclaimed children revisited: the status of children’s mental health policy in the United States. Retrieved from Columbia University, Academic Commons.

  10. Daleiden, E. L., Chorpita, B. F., Donkervoet, C., Arensdorf, A. M., Brogan, M., & Hamilton, J. D. (2006). Getting better at getting them better: Health outcomes and evidence-based practice within a system of care. J Am Acad Child Adolesc Psychiatry, 45, 749–756. doi:10.1097/01.chi.0000215154.07142.63.

    Article  PubMed  Google Scholar 

  11. Enders, C. K. (2010). Applied missing data analysis. New York: Guilford Press.

    Google Scholar 

  12. Garland, A. F., Brookman-Frazee, L., Hurlburt, M. S., Accurso, E. C., Zoffness, R. J., Haine-Schlagel, R., & Ganger, W. (2010). Mental health care for children with disruptive behavior problems: a view inside therapists’ offices. Psychiatr Serv, 61, 788–795. doi:10.1176/ps.2010.61.8.788.

    Article  PubMed  PubMed Central  Google Scholar 

  13. Greenhalgh, T., Robert, G., Macfarlane, F., Bate, P., & Kyriakidou, O. (2004). Diffusion of innovations in service organizations: systematic review and recommendations. Milbank Q, 82, 581–629. doi:10.1111/j.0887-378X.2004.00325.x.

    Article  PubMed  PubMed Central  Google Scholar 

  14. Hoagwood, K. E., Essock, S., Morrissey, J., Libby, A., Donahue, S., Druss, B., & Zerzan, J. (2015). Use of pooled state administrative data for mental health services research. Adm Policy Ment Health, 43(1), 67–78. doi:10.1007/s10488-014-0620-y.

    Article  PubMed Central  Google Scholar 

  15. Hoagwood, K. E., Olin, S. S., Horwitz, S., McKay, M., Cleek, A., Gleacher, A., & Hogan, M. (2014). Scaling up evidence-based practices for children and families in New York State: toward evidence-based policies on implementation for state mental health systems. J Clin Child Adolesc Psychol, 43, 145–157. doi:10.1080/15374416.2013.869749.

    Article  PubMed  PubMed Central  Google Scholar 

  16. Lau, A. S., & Brookman-Frazee, L. (2016). The 4KEEPS study: Identifying predictors of sustainment of multiple practices fiscally mandated in children’s mental health services. Implementation Science, 11, 31. doi:10.1186/s13012-016-0388-4.

  17. Los Angeles County Department of Mental Health (2011). Status report on the MHSA PEI plan: systems leadership team meeting. Accessed 15 Sept 2015.

  18. Los Angeles County Department of Mental Health (n.d.). About CDMH. Accessed 28 Sept 2015.

  19. Luo, W., & Kwok, O.-M. (2009). The impacts of ignoring a crossed factor in analyzing cross-classified data. Multivariate Behav Res, 44, 182–212. doi:10.1080/00273170902794214.

    Article  PubMed  Google Scholar 

  20. Luo, W., & Kwok, O.-M. (2012). The consequences of ignoring individual’s mobility in multilevel growth models: a Monte Carlo study. J Educ Behav Stat, 37, 31–56. doi:10.3102/1076998610394366.

    CAS  Article  Google Scholar 

  21. McHugh, R. K., & Barlow, D. H. (2010). The dissemination and implementation of evidence-based psychological treatments: a review of current efforts. Am Psychol, 65, 73–84. doi:10.1037/a0018121.

    Article  PubMed  Google Scholar 

  22. Nakamura, B. J., Chorpita, B. F., Hirsch, M., Daleiden, E., Slavin, L., Amundson, M. J., & Vorsino, W. M. (2011). Large-scale implementation of evidence-based treatments for children 10 years later: Hawaii’s evidence-based services initiative in children’s mental health. Clin Psychol, 18, 24–35. doi:10.1111/j.1468-2850.2010.01231.x.

    Google Scholar 

  23. Patient Protection and Affordable Care Act, 42 U.S.C. § 18001 (2010).

  24. Proctor, E., Silmere, H., Raghavan, R., Hovmand, P., Aarons, G., Bunger, A., & Hensley, M. (2011). Outcomes for implementation research: conceptual distinctions, measurement challenges, and research agenda. Adm Policy Ment Health, 38, 65–76. doi:10.1007/s10488-010-0319-7.

    Article  PubMed  Google Scholar 

  25. Raudenbush, S. W., & Bryk, A. S. (2002). Hierarchical linear models: application and data analysis methods (2nd ed.). Thousand Oaks: Sage Publications.

    Google Scholar 

  26. Raudenbush, S. W., Bryk, A. S., Cheong, Y. K., Congdon, R., & du Toit, M. (2011). HLM 7 [computer software]. Skokie: Scientific software international.

    Google Scholar 

  27. Rhoades, B. L., Bumbarger, B. K., & Moore, J. E. (2012). The role of a state-level prevention support system in promoting high-quality implementation and sustainability of evidence-based programs. Am J Community Psychol, 50, 386–401. doi:10.1007/s10464-012-9502-1.

    Article  PubMed  Google Scholar 

  28. Starin, A. C., Atkins, M. S., Wehrmann, K. C., Mehta, T., Hesson-McInnis, M. S., Marinez-Lora, A., & Mehlinger, R. (2014). Moving science into state child and adolescent mental health systems: Illinois’ evidence-informed practice initiative. J Clin Child Adolesc Psychol, 43, 169–178. doi:10.1080/15374416.2013.848772.

    Article  PubMed  Google Scholar 

  29. Stirman, S. W., Kimberly, J., Cook, N., Calloway, A., Castro, F., & Charns, M. (2012). The sustainability of new programs and innovations: a review of the empirical literature and recommendations for future research. Implement Sci, 7(17), 1–19. doi:10.1186/1748-5908-7-17.

    Google Scholar 

  30. Swain, K., Whitley, R., McHugo, G. J., & Drake, R. E. (2010). The sustainability of evidence-based practices in routine mental health agencies. Community Ment Health J, 46, 119–129. doi:10.1007/s10597-009-9202-y.

    Article  PubMed  Google Scholar 

  31. Trupin, E., & Kerns, S. (2015). Introduction to the special issue: legislation related to children’s evidence-based practice. Adm Policy Ment Health,. doi:10.1007/s10488-015-0666-5.

    Google Scholar 

  32. Weisz, J. R., Jensen-Doss, A., & Hawley, K. M. (2006). Evidence-based youth psychotherapies versus usual clinical care: a meta-analysis of direct comparisons. Am Psychol, 61, 671–689. doi:10.1037/0003-066X.61.7.671.

    Article  PubMed  Google Scholar 

  33. Willging, C. E., Green, A. E., Gunderson, L., Chaffin, M., & Aarons, G. A. (2015). From a “perfect storm” to “smooth sailing” policymaker perspectives on implementation and sustainment of an evidence-based practice in two states. Child Maltreat, 20, 24–36. doi:10.1177/1077559514547384.

    Article  PubMed  Google Scholar 

Download references


This study was funded by a Grant from the National Institute of Mental Health (R01MH100134).

Author information



Corresponding author

Correspondence to Lauren Brookman-Frazee.

Ethics declarations

Conflict of Interests

All authors declare that they have no conflict of interests.

Ethical Approval

All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards.

Informed Consent

A waiver of documented informed consent was obtained given that this study involved secondary analyses of de-identified administrative claims data.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Brookman-Frazee, L., Stadnick, N., Roesch, S. et al. Measuring Sustainment of Multiple Practices Fiscally Mandated in Children’s Mental Health Services. Adm Policy Ment Health 43, 1009–1022 (2016).

Download citation


  • Evidence-based practice
  • Implementation
  • Sustainment outcomes
  • Children’s mental health services