Abstract
Therapist discontinuation of delivering an evidence-based practice (EBP) is a critical outcome in the community implementation of EBPs. This mixed methods study examined factors associated with therapist discontinuation within a large reimbursement-driven implementation of multiple EBPs in public children’s mental health services. The study integrated quantitative survey data from 748 therapists across 65 agencies, and qualitative interviews from a subset of 79 therapists across 14 agencies. Therapists adopted, on average, 2.41 EBPs (SD = 1.05, range = 1–5), and nearly half (n = 355, 47.5%) reported discontinuing at least one EBP. Multi-level models were used to predict the binary outcome of discontinuation, and qualitative analyses were used to expand upon quantitative findings. Quantitative models revealed that therapist factors, including fewer direct service hours per week, a greater number of EBPs adopted, higher emotional exhaustion, and more negative attitudes toward EBPs in general were associated with discontinuation. In addition, EBP-specific factors including more negative perceptions of the particular EBP and lower self-efficacy for delivering the specific EBP predicted discontinuation. Themes from interview responses highlighted the importance of fit of the EBP with the agency’s client base, as well as therapist perceptions of adequate EBP training supports, and the alignment of an EBP with therapists’ professional goals. Together, the findings suggest the need for strategic sustainment planning interventions that target EBP fit (i.e., fit between adopted EBPs and agency target population, fit between EBP and therapist preferences and career goals) and support therapist self-efficacy in delivering EBPs.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Aarons, G. A. (2004). Mental health provider attitudes toward adoption of evidence-based practice: The evidence-based practice attitude scale (EBPAS). Mental Health Services Research,6(2), 61–74. https://doi.org/10.1023/b:mhsr.0000024351.12294.65.
Aarons, G. A., Hurlburt, M., & Horwitz, S. M. (2011). Advancing a conceptual model of evidence-based practice implementation in public service sectors. Administration and Policy in Mental Health and Mental Health Services Research,38(1), 4–23. https://doi.org/10.1007/s10488-010-0327-7.
Aarons, G. A., & Palinkas, L. A. (2007). Implementation of evidence-based practice in child welfare: Service provider perspectives. Administration and Policy in Mental Health and Mental Health Services Research,34(4), 411–419. https://doi.org/10.1007/s10488-007-0121-3.
Aarons, G. A., & Sawitzky, A. C. (2006). Organizational culture and climate and mental health provider attitudes toward evidence-based practice. Psychological Services,3(1), 61–72. https://doi.org/10.1037/1541-1559.3.1.61.
Barnett, M., Brookman-Frazee, L., Regan, J., Saifan, D., Stadnick, N., & Lau, A. S. (2017). How intervention and implementation characteristics relate to community therapists’ attitudes toward evidence-based practices: A mixed methods study. Administration and Policy in Mental Health and Mental Health Services Research,44(6), 824–837. https://doi.org/10.1007/s10488-017-0795-0.
Beidas, R. S., & Kendall, P. C. (2010). Training therapists in evidence-based practice: A critical review of studies from a systems-contextual perspective. Clinical Psychology: Science and Practice,17(1), 1–30. https://doi.org/10.1111/j.1468-2850.2009.01187.x.
Beidas, R. S., Marcus, S., Aarons, G. A., Hoagwood, K. E., Schoenwald, S., Evans, A. C., et al. (2015). Predictors of community therapists’ use of therapy techniques in a large public mental health system. JAMA Pediatrics,169(4), 374–382. https://doi.org/10.1001/jamapediatrics.2014.3736.
Beidas, R. S., Marcus, S., Wolk, C. B., Powell, B., Aarons, G. A., Evans, A. C., et al. (2016). A prospective examination of clinician and supervisor turnover within the context of implementation of evidence-based practices in a publicly-funded mental health system. Administration and Policy in Mental Health and Mental Health Services Research,43(5), 640–649. https://doi.org/10.1007/s10488-015-0673-6.
Beveridge, R. M., Fowles, T. R., Masse, J. J., McGoron, L., Smith, M. A., Parrish, B. P., et al. (2015). State-wide dissemination and implementation of parent–child interaction therapy (PCIT): Application of theory. Children and Youth Services Review,48, 38–48. https://doi.org/10.1016/j.childyouth.2014.11.013.
Blasinsky, M., Goldman, H. H., & Unützer, J. (2006). Project IMPACT: A report on barriers and facilitators to sustainability. Administration and Policy in Mental Health and Mental Health Services Research,33(6), 718–729. https://doi.org/10.1007/s10488-006-0086-7.
Bond, G. R., Drake, R. E., McHugo, G. J., Peterson, A. E., Jones, A. M., & Williams, J. (2014). Long-term sustainability of evidence-based practices in community mental health agencies. Administration and Policy in Mental Health and Mental Health Services Research,41(2), 228–236. https://doi.org/10.1007/s10488-012-0461-5.
Brookman-Frazee, L., Roesch, S., Chlebowski, C., Baker-Ericzen, M., & Ganger, W. (2019). Effectiveness of training therapists to deliver an individualized mental health intervention for children with ASD in publicly funded mental health services: A cluster randomized clinical trial. JAMA Psychiatry. https://doi.org/10.1001/jamapsychiatry.2019.0011.
Brookman-Frazee, L., Zhan, C., Stadnick, N., Sommerfeld, D., Roesch, S., Aarons, G. A., et al. (2018). Using survival analysis to understand patterns of sustainment within a system-driven implementation of multiple evidence-based practices for children’s mental health services. Frontiers in Public Health. https://doi.org/10.3389/fpubh.2018.00054.
Bunger, A. C., Doogan, N., Hanson, R. F., & Birken, S. A. (2018). Advice-seeking during implementation: A network study of clinicians participating in a learning collaborative. Implementation Science,13, 101–113. https://doi.org/10.1186/s13012-018-0797-7.
Bunger, A. C., Hanson, R. F., Doogan, N. J., Powell, B. J., Cao, Y., & Dunn, J. (2016). Can learning collaboratives support implementation by rewiring professional networks? Administration and Policy in Mental Health and Mental Health Services Research,43, 79–92. https://doi.org/10.1007/s10488-014-0621-x.
Cashel, M. L. (2002). Child and adolescent psychological assessment: Current clinical practices and the impact of managed care. Professional Psychology: Research and Practice,33(5), 446–453. https://doi.org/10.1037/0735-7028.33.5.446.
Chorpita, B. F., Bernstein, A., & Daleiden, E. L. (2011). Empirically guided coordination of multiple evidence-based treatments: An illustration of relevance mapping in children’s mental health services. Journal of Consulting and Clinical Psychology,79(4), 470–480. https://doi.org/10.1037/a0023982.
Cook, J. M., Thompson, R., & Schnurr, P. P. (2014). Perceived characteristics of intervention scale: Development and psychometric properties. Assessment,22(6), 704–714. https://doi.org/10.1177/1073191114561254.
Ehrhart, M. G., Aarons, G. A., & Farahnak, L. R. (2014). Assessing the organizational context for EBP implementation: The development and validity testing of the Implementation Climate Scale (ICS). Implementation Science. https://doi.org/10.1186/s13012-014-0157-1.
Friend, S., Flattum, C. F., Simpson, D., Nederhoff, D. M., & Neumark-Sztainer, D. (2014). The researchers have left the building: What contributes to sustaining school-based interventions following the conclusion of formal research support? Journal of School Health,84(5), 326–333. https://doi.org/10.1111/josh.12149.
Glaser, B. G., & Strauss, A. L. (1967). The discovery of grounded theory: Strategies for qualitative research. New Brunswick, NJ: Aldine Transaction.
Glisson, C., Landsverk, J., Schoenwald, S., Kelleher, K., Hoagwood, K. E., Mayberg, S., et al. (2007). Assessing the organizational social context (OSC) of mental health services: Implications for research and practice. Administration and Policy in Mental Health and Mental Health Services Research,35(1), 98–113. https://doi.org/10.1007/s10488-007-0148-5.
Glisson, C., & Schoenwald, S. K. (2005). The ARC organizational and community intervention strategy for implementing evidence-based children’s mental health treatments. Mental Health Services Research,7(4), 243–259. https://doi.org/10.1007/s11020-005-7456-1.
Gnjidic, D., & Elshaug, A. G. (2015). De-adoption and its 43 related terms: Harmonizing low-value care terminology. BMC Medicine,13, 273–276. https://doi.org/10.1186/s12916-015-0511-4.
Hamilton, A. B., Chinman, M., Cohen, A. N., Oberman, R. S., & Young, A. S. (2015). Implementation of consumer providers into mental health intensive case management teams. The Journal of Behavioral Health Services & Research,42(1), 100–108. https://doi.org/10.1007/s11414-013-9365-8.
Hawley, K. M., Cook, J. R., & Jensen-Doss, A. (2009). Do noncontingent incentives increase survey response rates among mental health providers? A randomized trial comparison. Administration and Policy in Mental Health and Mental Health Services Research,36(5), 343–348. https://doi.org/10.1007/s10488-009-0225-z.
Herschell, A. D., Kolko, D. J., Scudder, A. T., Taber-Thomas, S., Schaffner, K. F., Hiegel, S. A., et al. (2015). Protocol for a statewide randomized controlled trial to compare three training models for implementing an evidence-based treatment. Implementation Science,10, 133–148. https://doi.org/10.1186/s13012-015-0324-z.
Herschell, A. D., McNeil, C. B., Urquiza, A. J., McGrath, J. M., Zebell, N. M., Timmer, S. G., et al. (2009). Evaluation of a treatment manual and workshops for disseminating, parent–child interaction therapy. Administration and Policy in Mental Health and Mental Health Services Research,36(1), 63–81. https://doi.org/10.1007/s10488-008-0194-7.
Hill, C. E., Thompson, B. J., & Williams, E. N. (1997). A guide to conducting consensual qualitative research. The Counseling Psychologist,25(4), 517–572. https://doi.org/10.1177/0011000097254001.
Hoagwood, K. E., Olin, S. S., Horwitz, S., McKay, M., Cleek, A., Gleacher, A., et al. (2014). Scaling up evidence-based practices for children and families in New York State: Toward evidence-based policies on implementation for state mental health systems. Journal of Clinical Child & Adolescent Psychology,43(2), 145–157. https://doi.org/10.1080/15374416.2013.869749.
Hunter, S. B., Han, B., Slaughter, M. E., Godley, S. H., & Garner, B. R. (2015). Associations between implementation characteristics and evidence-based practice sustainment: A study of the adolescent community reinforcement approach. Implementation Science,10(1), 173. https://doi.org/10.1186/s13012-015-0364-4.
Kim, J. J., Brookman-Frazee, L., Gellatly, R., Stadnick, N., Barnett, M. L., & Lau, A. S. (2018). Predictors of burnout among community therapists in the sustainment phase of a system-driven implementation of multiple evidence-based practices in children’s mental health. Professional Psychology: Research and Practice,49(2), 132–141. https://doi.org/10.1037/pro0000182.
Lau, A. S., & Brookman-Frazee, L. (2016). The 4KEEPS study: Identifying predictors of sustainment of multiple practices fiscally mandated in children’s mental health services. Implementation Science,11(1), 31–38. https://doi.org/10.1186/s13012-016-0388-4.
Massatti, R. R., Sweeney, H. A., Panzano, P. C., & Roth, D. (2008). The de-adoption of innovative mental health practices (IMHP): Why organizations choose not to sustain an IMHP. Administration and Policy in Mental Health and Mental Health Services Research,35(1–2), 50–65. https://doi.org/10.1007/s10488-007-0141-z.
McHugh, R. K., & Barlow, D. H. (2010). The dissemination and implementation of evidence-based psychological treatments: A review of current efforts. American Psychologist,65(2), 73–84. https://doi.org/10.1037/a0018121.
McIntosh, K., Filter, K. J., Bennett, J. L., Ryan, C., & Sugai, G. (2010). Principles of sustainable prevention: Designing scale-up of school-wide positive behavior support to promote durable systems. Psychology in the Schools,47(1), 5–21. https://doi.org/10.1002/pits.20448.
McNeil, C. B., & Hembree-Kligin, T. L. (2010). Parent-child interaction therapy (2nd ed.). New York: Springer.
Miller, W. R., Yahne, C. E., Moyers, T. B., Martinez, J., & Pirritano, M. (2004). A randomized trial of methods to help clinicians learn motivational interviewing. Journal of Consulting and Clinical Psychology,72(6), 1050–1062. https://doi.org/10.1037/0022-006X.72.6.1050.
Morse, G., Salyers, M. P., Rollins, A. L., Monroe-DeVita, M., & Pfahler, C. (2012). Burnout in mental health services: A review of the problem and its remediation. Administration and Policy in Mental Health and Mental Health Services Research,39(5), 341–352. https://doi.org/10.1007/s10488-011-0352-1.
Mulkens, S., de Vos, C., de Graaff, A., & Waller, G. (2018). To deliver or not to deliver cognitive behavioral therapy for eating disorders: Replication and extension of our understanding of why therapists fail to do what they should do. Behaviour Research and Therapy,106, 57–63. https://doi.org/10.1016/j.brat.2018.05.004.
Nadeem, E., & Ringle, V. A. (2016). De-adoption of an evidence-based trauma intervention in schools: A retrospective report from an urban school district. School Mental Health,8(1), 132–143. https://doi.org/10.1007/s12310-016-9179-y.
Nadeem, E., Weiss, D., Olin, S. S., Hoagwood, K. E., & Horwitz, S. M. (2016). Using a theory-guided learning collaborative model to improve implementation of EBPs in a state children’s mental health system: A pilot study. Administration and Policy in Mental Health and Mental Health Services Research,43(6), 978–990. https://doi.org/10.1007/s10488-016-0735-4.
Nakamura, B. J., Mueller, C. W., Higa-McMillan, C., Okamura, K. H., Chang, J. P., Slavin, L., et al. (2014). Engineering youth service system infrastructure: Hawaii’s continued efforts at large-scale implementation through knowledge management strategies. Journal of Clinical Child & Adolescent Psychology,43(2), 179–189. https://doi.org/10.1080/15374416.2013.812039.
Nehrig, N., Prout, T. A., & Aafjes-van Doorn, K. (2019). Whose anxiety are we treating, anyway? Journal of Clinical Psychology,75(10), 1866–1878. https://doi.org/10.1002/jclp.22824.
Nilsen, P., Timpka, T., Nordenfelt, L., & Lindqvist, K. (2005). Towards improved understanding of injury prevention program sustainability. Safety Science,43(10), 815–833. https://doi.org/10.1016/j.ssci.2005.08.015.
Niven, D. J., Mrklas, K. J., Holodinsky, J. K., Straus, S. E., Hemmelgarn, B. R., Jeffs, L. P., et al. (2015). Towards understanding the de-adoption of low-value clinical practices: A scoping review. BMC Medicine,13, 255–275. https://doi.org/10.1186/s12916-015-0488-z.
Novins, D. K., Green, A. E., Legha, R. K., & Aarons, G. A. (2013). Dissemination and implementation of evidence-based practices for child and adolescent mental health: A systematic review. Journal of the American Academy of Child and Adolescent Psychiatry,52(10), 1009–1025. https://doi.org/10.1016/j.jaac.2013.07.012.
Okamura, K. H., Benjamin Wolk, C. L., Kang-Yi, C. D., Stewart, R., Rubin, R. M., Weaver, S., et al. (2018). The price per prospective consumer of providing therapist training and consultation in seven evidence-based treatments within a large public behavioral health system: An example cost-analysis metric. Frontiers in Public Health,5, 21–29. https://doi.org/10.3389/fpubh.2017.00356.
Palinkas, L. A. (2014). Qualitative and mixed methods in mental health services and implementation research. Journal of Clinical Child & Adolescent Psychology,43(6), 851–861. https://doi.org/10.1080/15374416.2014.910791.
Palinkas, L. A., Aarons, G. A., Horwitz, S., Chamberlain, P., Hurlburt, M., & Landsverk, J. (2011). Mixed method designs in implementation research. Administration and Policy in Mental Health and Mental Health Services Research,38(1), 44–53. https://doi.org/10.1007/s10488-010-0314-z.
Patterson, M. G., West, M. A., Shackleton, V. J., Dawson, J. F., Lawthom, R., Maitlis, S., et al. (2005). Validating the organizational climate measure: Links to managerial practices, productivity and innovation. Journal of Organizational Behavior,26(4), 379–408. https://doi.org/10.1002/job.312.
Peterson, A. E., Bond, G. R., Drake, R. E., McHugo, G. J., Jones, A. M., & Williams, J. R. (2014). Predicting the long-term sustainability of evidence-based practices in mental health care: An 8-year longitudinal analysis. The Journal of Behavioral Health Services & Research,41(3), 337–346. https://doi.org/10.1007/s11414-013-9347-x.
Proctor, E., Silmere, H., Raghavan, R., Hovmand, P., Aarons, G., Bunger, A., et al. (2011). Outcomes for implementation research: Conceptual distinctions, measurement challenges, and research agenda. Administration and Policy in Mental Health and Mental Health Services Research,38(2), 65–76. https://doi.org/10.1007/s10488-010-0319-7.
Regan, J., Lau, A. S., Barnett, M., Stadnick, N., Hamilton, A., Pesanti, K., et al. (2017). Agency responses to a system-driven implementation of multiple evidence-based practices in children’s mental health services. BMC Health Services Research,17(1), 671–684. https://doi.org/10.1186/s12913-017-2613-5.
Rodriguez, A., Lau, A. S., Wright, B., Regan, J., & Brookman-Frazee, L. (2018). Mixed-method analysis of program leader perspectives on the sustainment of multiple child evidence-based practices in a system-driven implementation. Implementation Science,13(1), 44–57. https://doi.org/10.1186/s13012-018-0737-6.
Spradley, J. (1979). The ethnographic interview. New York: Holt, Rinehart and Winston.
Starin, A. C., Atkins, M. S., Wehrmann, K. C., Mehta, T., Hesson-McInnis, M. S., Marinez-Lora, A., et al. (2014). Moving science into state child and adolescent mental health systems: Illinois’ evidence-informed practice initiative. Journal of Clinical Child & Adolescent Psychology,43(2), 169–178. https://doi.org/10.1080/15374416.2013.848772.
Stewart, R. E., Adams, D. R., Mandell, D. S., Hadley, T. R., Evans, A. C., Rubin, R., et al. (2016). The perfect storm: Collision of the business of mental health and the implementation of evidence-based practices. Psychiatric Services,67(2), 159–161. https://doi.org/10.1176/appi.ps.201500392.
Stirman, S. W., Gutiérrez-Colina, A., Toder, K., Esposito, G., Barg, F., Castro, F., et al. (2013). Clinicians’ perspectives on Cognitive Therapy in community mental health settings: Implications for training and implementation. Administration and Policy in Mental Health and Mental Health Services Research,40(4), 274–285. https://doi.org/10.1007/s10488-012-0418-8.
Stirman, S. W., Kimberly, J., Cook, N., Calloway, A., Castro, F., & Charns, M. (2012). The sustainability of new programs and innovations: A review of the empirical literature and recommendations for future research. Implementation Science,7(1), 17–36. https://doi.org/10.1186/1748-5908-7-17.
Willms, D. G., Best, J. A., Taylor, D. W., Gilbert, J. R., Wilson, D. M. C., Lindsay, E. A., et al. (1990). A systematic approach for using qualitative methods in primary prevention research. Medical Anthropology Quarterly,4(4), 391–409. https://doi.org/10.1525/maq.1990.4.4.02a00020.
Funding
Funding for this research project, including the study design and data collection, analysis and interpretation of data, was supported by NIMH Grant R01 MH100134.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
There are no potential conflicts of interest for any of authors.
Informed Consent
Written or verbal informed consent was obtained from all participants if they agreed to participate.
Research Involved in Human Rights
This research involved human subjects.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Lau, A.S., Lind, T., Crawley, M. et al. When Do Therapists Stop Using Evidence-Based Practices? Findings from a Mixed Method Study on System-Driven Implementation of Multiple EBPs for Children. Adm Policy Ment Health 47, 323–337 (2020). https://doi.org/10.1007/s10488-019-00987-2
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10488-019-00987-2