Examining sustainment of multiple evidence-based practices is crucial to understanding the processes and outcomes of system-driven implementation efforts that are increasingly common. This study used administrative claims data to characterize volume and penetration of six practices over 19 fiscal quarters following initial implementation within the context of a system-driven, fiscally mandated implementation effort in Los Angeles County Department of Mental Health’s Prevention and Early Intervention services. Patterns of volume changes over time revealed an overall ramp up of practice claims over time, but variability in patterns of volume and penetration for each practice. Findings varied by the methods used to index and analyze volume and penetration. Furthermore, a number of client case-mix and therapist characteristics were associated with the volume of therapists’ claims for each practice relative to their claims for the other practices.
This is a preview of subscription content, access via your institution.
Buy single article
Instant access to the full article PDF.
Tax calculation will be finalised during checkout.
Subscribe to journal
Immediate online access to all issues from 2019. Subscription will auto renew annually.
Tax calculation will be finalised during checkout.
MAP trainings began in Quarter 2 of FY 2010–2011.
Inclusion of the total number of psychotherapy claims across the six practices per fiscal quarter as a repeated-measures covariate was considered but was not feasible due to high collinearity between total claim volume and time.
Aarons, G. A., Hurlburt, M., & Horwitz, S. M. (2011). Advancing a conceptual model of evidence-based practice implementation in public service sectors. Adm Policy Ment Health, 38, 4–23. doi:10.1007/s10488-010-0327-7.
Beretvas, S. N. (2011). Cross-classified and multiple-membership models. In J. J. Hox & J. K. Roberts (Eds.), Handbook for advanced multilevel analysis (pp. 313–334). New York: Routledge.
Bond, G. R., Drake, R. E., McHugo, G. J., Peterson, A. E., Jones, A. M., & Williams, J. (2014). Long-term sustainability of evidence-based practices in community mental health agencies. Adm Policy Ment Health, 41, 228–236. doi:10.1007/s10488-012-0461-5.
Cafri, G., Hedeker, D., & Aarons, G. A. (2015). An introduction and integration of cross-classified, multiple membership, and dynamic group random-effects models. Psychol Methods,. doi:10.1037/met0000043.
Chambers, D. A., & Rupp, A. (2015). Sharing state mental health data for research: building toward ongoing learning in mental health care systems. Adm Policy Ment Health, 42, 586–587. doi:10.1007/s10488-015-0624-2.
Chorpita, B. F., Bernstein, A., & Daleiden, E. L. (2011). Empirically guided coordination of multiple evidence-based treatments: an illustration of relevance mapping in children’s mental health services. J Consult Clin Psychol, 79, 470–480. doi:10.1037/a0023982.
Chorpita, B. F., & Daleiden, E. L. (2009). Mapping evidence-based treatments for children and adolescents: application of the distillation and matching model to 615 treatments from 322 randomized trials. J Consult Clin Psychol, 77, 566–579. doi:10.1037/a0014565.
Chorpita, B. F., Yim, L. M., Donkervoet, J. C., Arensdorf, A., Amundsen, M. J., McGee, C., & Morelli, P. (2002). Toward large-scale implementation of empirically supported treatments for children: a review and observations by the Hawaii empirical basis to services task force. Clin Psychol, 9, 165–190. doi:10.1111/j.1468-2850.2002.tb00504.x.
Cooper, J. L., Aratani, Y., Knitzer, J., Douglas-Hall, A., Masi, R., Banghart, P. L., Dababnah, S. (2008). Unclaimed children revisited: the status of children’s mental health policy in the United States. Retrieved from Columbia University, Academic Commons. http://www.hdl.handle.net/10022/AC:P:8917.
Daleiden, E. L., Chorpita, B. F., Donkervoet, C., Arensdorf, A. M., Brogan, M., & Hamilton, J. D. (2006). Getting better at getting them better: Health outcomes and evidence-based practice within a system of care. J Am Acad Child Adolesc Psychiatry, 45, 749–756. doi:10.1097/01.chi.0000215154.07142.63.
Enders, C. K. (2010). Applied missing data analysis. New York: Guilford Press.
Garland, A. F., Brookman-Frazee, L., Hurlburt, M. S., Accurso, E. C., Zoffness, R. J., Haine-Schlagel, R., & Ganger, W. (2010). Mental health care for children with disruptive behavior problems: a view inside therapists’ offices. Psychiatr Serv, 61, 788–795. doi:10.1176/ps.2010.61.8.788.
Greenhalgh, T., Robert, G., Macfarlane, F., Bate, P., & Kyriakidou, O. (2004). Diffusion of innovations in service organizations: systematic review and recommendations. Milbank Q, 82, 581–629. doi:10.1111/j.0887-378X.2004.00325.x.
Hoagwood, K. E., Essock, S., Morrissey, J., Libby, A., Donahue, S., Druss, B., & Zerzan, J. (2015). Use of pooled state administrative data for mental health services research. Adm Policy Ment Health, 43(1), 67–78. doi:10.1007/s10488-014-0620-y.
Hoagwood, K. E., Olin, S. S., Horwitz, S., McKay, M., Cleek, A., Gleacher, A., & Hogan, M. (2014). Scaling up evidence-based practices for children and families in New York State: toward evidence-based policies on implementation for state mental health systems. J Clin Child Adolesc Psychol, 43, 145–157. doi:10.1080/15374416.2013.869749.
Lau, A. S., & Brookman-Frazee, L. (2016). The 4KEEPS study: Identifying predictors of sustainment of multiple practices fiscally mandated in children’s mental health services. Implementation Science, 11, 31. doi:10.1186/s13012-016-0388-4.
Los Angeles County Department of Mental Health (2011). Status report on the MHSA PEI plan: systems leadership team meeting. http://www.file.lacounty.gov/dmh/cms1_165911.pdf. Accessed 15 Sept 2015.
Los Angeles County Department of Mental Health (n.d.). About CDMH. http://www.dmh.lacounty.gov/wps/portal/dmh/aboutdmh. Accessed 28 Sept 2015.
Luo, W., & Kwok, O.-M. (2009). The impacts of ignoring a crossed factor in analyzing cross-classified data. Multivariate Behav Res, 44, 182–212. doi:10.1080/00273170902794214.
Luo, W., & Kwok, O.-M. (2012). The consequences of ignoring individual’s mobility in multilevel growth models: a Monte Carlo study. J Educ Behav Stat, 37, 31–56. doi:10.3102/1076998610394366.
McHugh, R. K., & Barlow, D. H. (2010). The dissemination and implementation of evidence-based psychological treatments: a review of current efforts. Am Psychol, 65, 73–84. doi:10.1037/a0018121.
Nakamura, B. J., Chorpita, B. F., Hirsch, M., Daleiden, E., Slavin, L., Amundson, M. J., & Vorsino, W. M. (2011). Large-scale implementation of evidence-based treatments for children 10 years later: Hawaii’s evidence-based services initiative in children’s mental health. Clin Psychol, 18, 24–35. doi:10.1111/j.1468-2850.2010.01231.x.
Patient Protection and Affordable Care Act, 42 U.S.C. § 18001 (2010).
Proctor, E., Silmere, H., Raghavan, R., Hovmand, P., Aarons, G., Bunger, A., & Hensley, M. (2011). Outcomes for implementation research: conceptual distinctions, measurement challenges, and research agenda. Adm Policy Ment Health, 38, 65–76. doi:10.1007/s10488-010-0319-7.
Raudenbush, S. W., & Bryk, A. S. (2002). Hierarchical linear models: application and data analysis methods (2nd ed.). Thousand Oaks: Sage Publications.
Raudenbush, S. W., Bryk, A. S., Cheong, Y. K., Congdon, R., & du Toit, M. (2011). HLM 7 [computer software]. Skokie: Scientific software international.
Rhoades, B. L., Bumbarger, B. K., & Moore, J. E. (2012). The role of a state-level prevention support system in promoting high-quality implementation and sustainability of evidence-based programs. Am J Community Psychol, 50, 386–401. doi:10.1007/s10464-012-9502-1.
Starin, A. C., Atkins, M. S., Wehrmann, K. C., Mehta, T., Hesson-McInnis, M. S., Marinez-Lora, A., & Mehlinger, R. (2014). Moving science into state child and adolescent mental health systems: Illinois’ evidence-informed practice initiative. J Clin Child Adolesc Psychol, 43, 169–178. doi:10.1080/15374416.2013.848772.
Stirman, S. W., Kimberly, J., Cook, N., Calloway, A., Castro, F., & Charns, M. (2012). The sustainability of new programs and innovations: a review of the empirical literature and recommendations for future research. Implement Sci, 7(17), 1–19. doi:10.1186/1748-5908-7-17.
Swain, K., Whitley, R., McHugo, G. J., & Drake, R. E. (2010). The sustainability of evidence-based practices in routine mental health agencies. Community Ment Health J, 46, 119–129. doi:10.1007/s10597-009-9202-y.
Trupin, E., & Kerns, S. (2015). Introduction to the special issue: legislation related to children’s evidence-based practice. Adm Policy Ment Health,. doi:10.1007/s10488-015-0666-5.
Weisz, J. R., Jensen-Doss, A., & Hawley, K. M. (2006). Evidence-based youth psychotherapies versus usual clinical care: a meta-analysis of direct comparisons. Am Psychol, 61, 671–689. doi:10.1037/0003-066X.61.7.671.
Willging, C. E., Green, A. E., Gunderson, L., Chaffin, M., & Aarons, G. A. (2015). From a “perfect storm” to “smooth sailing” policymaker perspectives on implementation and sustainment of an evidence-based practice in two states. Child Maltreat, 20, 24–36. doi:10.1177/1077559514547384.
This study was funded by a Grant from the National Institute of Mental Health (R01MH100134).
Conflict of Interests
All authors declare that they have no conflict of interests.
All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards.
A waiver of documented informed consent was obtained given that this study involved secondary analyses of de-identified administrative claims data.
About this article
Cite this article
Brookman-Frazee, L., Stadnick, N., Roesch, S. et al. Measuring Sustainment of Multiple Practices Fiscally Mandated in Children’s Mental Health Services. Adm Policy Ment Health 43, 1009–1022 (2016). https://doi.org/10.1007/s10488-016-0731-8
- Evidence-based practice
- Sustainment outcomes
- Children’s mental health services