When Do Therapists Stop Using Evidence-Based Practices? Findings from a Mixed Method Study on System-Driven Implementation of Multiple EBPs for Children

  • Anna S. Lau
  • Teresa LindEmail author
  • Morgan Crawley
  • Adriana Rodriguez
  • Ashley Smith
  • Lauren Brookman-Frazee
Original Article


Therapist discontinuation of delivering an evidence-based practice (EBP) is a critical outcome in the community implementation of EBPs. This mixed methods study examined factors associated with therapist discontinuation within a large reimbursement-driven implementation of multiple EBPs in public children’s mental health services. The study integrated quantitative survey data from 748 therapists across 65 agencies, and qualitative interviews from a subset of 79 therapists across 14 agencies. Therapists adopted, on average, 2.41 EBPs (SD = 1.05, range = 1–5), and nearly half (n = 355, 47.5%) reported discontinuing at least one EBP. Multi-level models were used to predict the binary outcome of discontinuation, and qualitative analyses were used to expand upon quantitative findings. Quantitative models revealed that therapist factors, including fewer direct service hours per week, a greater number of EBPs adopted, higher emotional exhaustion, and more negative attitudes toward EBPs in general were associated with discontinuation. In addition, EBP-specific factors including more negative perceptions of the particular EBP and lower self-efficacy for delivering the specific EBP predicted discontinuation. Themes from interview responses highlighted the importance of fit of the EBP with the agency’s client base, as well as therapist perceptions of adequate EBP training supports, and the alignment of an EBP with therapists’ professional goals. Together, the findings suggest the need for strategic sustainment planning interventions that target EBP fit (i.e., fit between adopted EBPs and agency target population, fit between EBP and therapist preferences and career goals) and support therapist self-efficacy in delivering EBPs.


Evidence-based practice Sustainment Discontinuation 



Funding for this research project, including the study design and data collection, analysis and interpretation of data, was supported by NIMH Grant R01 MH100134.

Compliance with Ethical Standards

Conflict of interest

There are no potential conflicts of interest for any of authors.

Informed Consent

Written or verbal informed consent was obtained from all participants if they agreed to participate.

Research Involved in Human Rights

This research involved human subjects.


  1. Aarons, G. A. (2004). Mental health provider attitudes toward adoption of evidence-based practice: The evidence-based practice attitude scale (EBPAS). Mental Health Services Research,6(2), 61–74. Scholar
  2. Aarons, G. A., Hurlburt, M., & Horwitz, S. M. (2011). Advancing a conceptual model of evidence-based practice implementation in public service sectors. Administration and Policy in Mental Health and Mental Health Services Research,38(1), 4–23. Scholar
  3. Aarons, G. A., & Palinkas, L. A. (2007). Implementation of evidence-based practice in child welfare: Service provider perspectives. Administration and Policy in Mental Health and Mental Health Services Research,34(4), 411–419. Scholar
  4. Aarons, G. A., & Sawitzky, A. C. (2006). Organizational culture and climate and mental health provider attitudes toward evidence-based practice. Psychological Services,3(1), 61–72. Scholar
  5. Barnett, M., Brookman-Frazee, L., Regan, J., Saifan, D., Stadnick, N., & Lau, A. S. (2017). How intervention and implementation characteristics relate to community therapists’ attitudes toward evidence-based practices: A mixed methods study. Administration and Policy in Mental Health and Mental Health Services Research,44(6), 824–837. Scholar
  6. Beidas, R. S., & Kendall, P. C. (2010). Training therapists in evidence-based practice: A critical review of studies from a systems-contextual perspective. Clinical Psychology: Science and Practice,17(1), 1–30. Scholar
  7. Beidas, R. S., Marcus, S., Aarons, G. A., Hoagwood, K. E., Schoenwald, S., Evans, A. C., et al. (2015). Predictors of community therapists’ use of therapy techniques in a large public mental health system. JAMA Pediatrics,169(4), 374–382. Scholar
  8. Beidas, R. S., Marcus, S., Wolk, C. B., Powell, B., Aarons, G. A., Evans, A. C., et al. (2016). A prospective examination of clinician and supervisor turnover within the context of implementation of evidence-based practices in a publicly-funded mental health system. Administration and Policy in Mental Health and Mental Health Services Research,43(5), 640–649. Scholar
  9. Beveridge, R. M., Fowles, T. R., Masse, J. J., McGoron, L., Smith, M. A., Parrish, B. P., et al. (2015). State-wide dissemination and implementation of parent–child interaction therapy (PCIT): Application of theory. Children and Youth Services Review,48, 38–48. Scholar
  10. Blasinsky, M., Goldman, H. H., & Unützer, J. (2006). Project IMPACT: A report on barriers and facilitators to sustainability. Administration and Policy in Mental Health and Mental Health Services Research,33(6), 718–729. Scholar
  11. Bond, G. R., Drake, R. E., McHugo, G. J., Peterson, A. E., Jones, A. M., & Williams, J. (2014). Long-term sustainability of evidence-based practices in community mental health agencies. Administration and Policy in Mental Health and Mental Health Services Research,41(2), 228–236. Scholar
  12. Brookman-Frazee, L., Roesch, S., Chlebowski, C., Baker-Ericzen, M., & Ganger, W. (2019). Effectiveness of training therapists to deliver an individualized mental health intervention for children with ASD in publicly funded mental health services: A cluster randomized clinical trial. JAMA Psychiatry. Scholar
  13. Brookman-Frazee, L., Zhan, C., Stadnick, N., Sommerfeld, D., Roesch, S., Aarons, G. A., et al. (2018). Using survival analysis to understand patterns of sustainment within a system-driven implementation of multiple evidence-based practices for children’s mental health services. Frontiers in Public Health. Scholar
  14. Bunger, A. C., Doogan, N., Hanson, R. F., & Birken, S. A. (2018). Advice-seeking during implementation: A network study of clinicians participating in a learning collaborative. Implementation Science,13, 101–113. Scholar
  15. Bunger, A. C., Hanson, R. F., Doogan, N. J., Powell, B. J., Cao, Y., & Dunn, J. (2016). Can learning collaboratives support implementation by rewiring professional networks? Administration and Policy in Mental Health and Mental Health Services Research,43, 79–92. Scholar
  16. Cashel, M. L. (2002). Child and adolescent psychological assessment: Current clinical practices and the impact of managed care. Professional Psychology: Research and Practice,33(5), 446–453. Scholar
  17. Chorpita, B. F., Bernstein, A., & Daleiden, E. L. (2011). Empirically guided coordination of multiple evidence-based treatments: An illustration of relevance mapping in children’s mental health services. Journal of Consulting and Clinical Psychology,79(4), 470–480. Scholar
  18. Cook, J. M., Thompson, R., & Schnurr, P. P. (2014). Perceived characteristics of intervention scale: Development and psychometric properties. Assessment,22(6), 704–714. Scholar
  19. Ehrhart, M. G., Aarons, G. A., & Farahnak, L. R. (2014). Assessing the organizational context for EBP implementation: The development and validity testing of the Implementation Climate Scale (ICS). Implementation Science. Scholar
  20. Friend, S., Flattum, C. F., Simpson, D., Nederhoff, D. M., & Neumark-Sztainer, D. (2014). The researchers have left the building: What contributes to sustaining school-based interventions following the conclusion of formal research support? Journal of School Health,84(5), 326–333. Scholar
  21. Glaser, B. G., & Strauss, A. L. (1967). The discovery of grounded theory: Strategies for qualitative research. New Brunswick, NJ: Aldine Transaction.Google Scholar
  22. Glisson, C., Landsverk, J., Schoenwald, S., Kelleher, K., Hoagwood, K. E., Mayberg, S., et al. (2007). Assessing the organizational social context (OSC) of mental health services: Implications for research and practice. Administration and Policy in Mental Health and Mental Health Services Research,35(1), 98–113. Scholar
  23. Glisson, C., & Schoenwald, S. K. (2005). The ARC organizational and community intervention strategy for implementing evidence-based children’s mental health treatments. Mental Health Services Research,7(4), 243–259. Scholar
  24. Gnjidic, D., & Elshaug, A. G. (2015). De-adoption and its 43 related terms: Harmonizing low-value care terminology. BMC Medicine,13, 273–276. Scholar
  25. Hamilton, A. B., Chinman, M., Cohen, A. N., Oberman, R. S., & Young, A. S. (2015). Implementation of consumer providers into mental health intensive case management teams. The Journal of Behavioral Health Services & Research,42(1), 100–108. Scholar
  26. Hawley, K. M., Cook, J. R., & Jensen-Doss, A. (2009). Do noncontingent incentives increase survey response rates among mental health providers? A randomized trial comparison. Administration and Policy in Mental Health and Mental Health Services Research,36(5), 343–348. Scholar
  27. Herschell, A. D., Kolko, D. J., Scudder, A. T., Taber-Thomas, S., Schaffner, K. F., Hiegel, S. A., et al. (2015). Protocol for a statewide randomized controlled trial to compare three training models for implementing an evidence-based treatment. Implementation Science,10, 133–148. Scholar
  28. Herschell, A. D., McNeil, C. B., Urquiza, A. J., McGrath, J. M., Zebell, N. M., Timmer, S. G., et al. (2009). Evaluation of a treatment manual and workshops for disseminating, parent–child interaction therapy. Administration and Policy in Mental Health and Mental Health Services Research,36(1), 63–81. Scholar
  29. Hill, C. E., Thompson, B. J., & Williams, E. N. (1997). A guide to conducting consensual qualitative research. The Counseling Psychologist,25(4), 517–572. Scholar
  30. Hoagwood, K. E., Olin, S. S., Horwitz, S., McKay, M., Cleek, A., Gleacher, A., et al. (2014). Scaling up evidence-based practices for children and families in New York State: Toward evidence-based policies on implementation for state mental health systems. Journal of Clinical Child & Adolescent Psychology,43(2), 145–157. Scholar
  31. Hunter, S. B., Han, B., Slaughter, M. E., Godley, S. H., & Garner, B. R. (2015). Associations between implementation characteristics and evidence-based practice sustainment: A study of the adolescent community reinforcement approach. Implementation Science,10(1), 173. Scholar
  32. Kim, J. J., Brookman-Frazee, L., Gellatly, R., Stadnick, N., Barnett, M. L., & Lau, A. S. (2018). Predictors of burnout among community therapists in the sustainment phase of a system-driven implementation of multiple evidence-based practices in children’s mental health. Professional Psychology: Research and Practice,49(2), 132–141. Scholar
  33. Lau, A. S., & Brookman-Frazee, L. (2016). The 4KEEPS study: Identifying predictors of sustainment of multiple practices fiscally mandated in children’s mental health services. Implementation Science,11(1), 31–38. Scholar
  34. Massatti, R. R., Sweeney, H. A., Panzano, P. C., & Roth, D. (2008). The de-adoption of innovative mental health practices (IMHP): Why organizations choose not to sustain an IMHP. Administration and Policy in Mental Health and Mental Health Services Research,35(1–2), 50–65. Scholar
  35. McHugh, R. K., & Barlow, D. H. (2010). The dissemination and implementation of evidence-based psychological treatments: A review of current efforts. American Psychologist,65(2), 73–84. Scholar
  36. McIntosh, K., Filter, K. J., Bennett, J. L., Ryan, C., & Sugai, G. (2010). Principles of sustainable prevention: Designing scale-up of school-wide positive behavior support to promote durable systems. Psychology in the Schools,47(1), 5–21. Scholar
  37. McNeil, C. B., & Hembree-Kligin, T. L. (2010). Parent-child interaction therapy (2nd ed.). New York: Springer.CrossRefGoogle Scholar
  38. Miller, W. R., Yahne, C. E., Moyers, T. B., Martinez, J., & Pirritano, M. (2004). A randomized trial of methods to help clinicians learn motivational interviewing. Journal of Consulting and Clinical Psychology,72(6), 1050–1062. Scholar
  39. Morse, G., Salyers, M. P., Rollins, A. L., Monroe-DeVita, M., & Pfahler, C. (2012). Burnout in mental health services: A review of the problem and its remediation. Administration and Policy in Mental Health and Mental Health Services Research,39(5), 341–352. Scholar
  40. Mulkens, S., de Vos, C., de Graaff, A., & Waller, G. (2018). To deliver or not to deliver cognitive behavioral therapy for eating disorders: Replication and extension of our understanding of why therapists fail to do what they should do. Behaviour Research and Therapy,106, 57–63. Scholar
  41. Nadeem, E., & Ringle, V. A. (2016). De-adoption of an evidence-based trauma intervention in schools: A retrospective report from an urban school district. School Mental Health,8(1), 132–143. Scholar
  42. Nadeem, E., Weiss, D., Olin, S. S., Hoagwood, K. E., & Horwitz, S. M. (2016). Using a theory-guided learning collaborative model to improve implementation of EBPs in a state children’s mental health system: A pilot study. Administration and Policy in Mental Health and Mental Health Services Research,43(6), 978–990. Scholar
  43. Nakamura, B. J., Mueller, C. W., Higa-McMillan, C., Okamura, K. H., Chang, J. P., Slavin, L., et al. (2014). Engineering youth service system infrastructure: Hawaii’s continued efforts at large-scale implementation through knowledge management strategies. Journal of Clinical Child & Adolescent Psychology,43(2), 179–189. Scholar
  44. Nehrig, N., Prout, T. A., & Aafjes-van Doorn, K. (2019). Whose anxiety are we treating, anyway? Journal of Clinical Psychology,75(10), 1866–1878. Scholar
  45. Nilsen, P., Timpka, T., Nordenfelt, L., & Lindqvist, K. (2005). Towards improved understanding of injury prevention program sustainability. Safety Science,43(10), 815–833. Scholar
  46. Niven, D. J., Mrklas, K. J., Holodinsky, J. K., Straus, S. E., Hemmelgarn, B. R., Jeffs, L. P., et al. (2015). Towards understanding the de-adoption of low-value clinical practices: A scoping review. BMC Medicine,13, 255–275. Scholar
  47. Novins, D. K., Green, A. E., Legha, R. K., & Aarons, G. A. (2013). Dissemination and implementation of evidence-based practices for child and adolescent mental health: A systematic review. Journal of the American Academy of Child and Adolescent Psychiatry,52(10), 1009–1025. Scholar
  48. Okamura, K. H., Benjamin Wolk, C. L., Kang-Yi, C. D., Stewart, R., Rubin, R. M., Weaver, S., et al. (2018). The price per prospective consumer of providing therapist training and consultation in seven evidence-based treatments within a large public behavioral health system: An example cost-analysis metric. Frontiers in Public Health,5, 21–29. Scholar
  49. Palinkas, L. A. (2014). Qualitative and mixed methods in mental health services and implementation research. Journal of Clinical Child & Adolescent Psychology,43(6), 851–861. Scholar
  50. Palinkas, L. A., Aarons, G. A., Horwitz, S., Chamberlain, P., Hurlburt, M., & Landsverk, J. (2011). Mixed method designs in implementation research. Administration and Policy in Mental Health and Mental Health Services Research,38(1), 44–53. Scholar
  51. Patterson, M. G., West, M. A., Shackleton, V. J., Dawson, J. F., Lawthom, R., Maitlis, S., et al. (2005). Validating the organizational climate measure: Links to managerial practices, productivity and innovation. Journal of Organizational Behavior,26(4), 379–408. Scholar
  52. Peterson, A. E., Bond, G. R., Drake, R. E., McHugo, G. J., Jones, A. M., & Williams, J. R. (2014). Predicting the long-term sustainability of evidence-based practices in mental health care: An 8-year longitudinal analysis. The Journal of Behavioral Health Services & Research,41(3), 337–346. Scholar
  53. Proctor, E., Silmere, H., Raghavan, R., Hovmand, P., Aarons, G., Bunger, A., et al. (2011). Outcomes for implementation research: Conceptual distinctions, measurement challenges, and research agenda. Administration and Policy in Mental Health and Mental Health Services Research,38(2), 65–76. Scholar
  54. Regan, J., Lau, A. S., Barnett, M., Stadnick, N., Hamilton, A., Pesanti, K., et al. (2017). Agency responses to a system-driven implementation of multiple evidence-based practices in children’s mental health services. BMC Health Services Research,17(1), 671–684. Scholar
  55. Rodriguez, A., Lau, A. S., Wright, B., Regan, J., & Brookman-Frazee, L. (2018). Mixed-method analysis of program leader perspectives on the sustainment of multiple child evidence-based practices in a system-driven implementation. Implementation Science,13(1), 44–57. Scholar
  56. Spradley, J. (1979). The ethnographic interview. New York: Holt, Rinehart and Winston.Google Scholar
  57. Starin, A. C., Atkins, M. S., Wehrmann, K. C., Mehta, T., Hesson-McInnis, M. S., Marinez-Lora, A., et al. (2014). Moving science into state child and adolescent mental health systems: Illinois’ evidence-informed practice initiative. Journal of Clinical Child & Adolescent Psychology,43(2), 169–178. Scholar
  58. Stewart, R. E., Adams, D. R., Mandell, D. S., Hadley, T. R., Evans, A. C., Rubin, R., et al. (2016). The perfect storm: Collision of the business of mental health and the implementation of evidence-based practices. Psychiatric Services,67(2), 159–161. Scholar
  59. Stirman, S. W., Gutiérrez-Colina, A., Toder, K., Esposito, G., Barg, F., Castro, F., et al. (2013). Clinicians’ perspectives on Cognitive Therapy in community mental health settings: Implications for training and implementation. Administration and Policy in Mental Health and Mental Health Services Research,40(4), 274–285. Scholar
  60. Stirman, S. W., Kimberly, J., Cook, N., Calloway, A., Castro, F., & Charns, M. (2012). The sustainability of new programs and innovations: A review of the empirical literature and recommendations for future research. Implementation Science,7(1), 17–36. Scholar
  61. Willms, D. G., Best, J. A., Taylor, D. W., Gilbert, J. R., Wilson, D. M. C., Lindsay, E. A., et al. (1990). A systematic approach for using qualitative methods in primary prevention research. Medical Anthropology Quarterly,4(4), 391–409. Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2019

Authors and Affiliations

  1. 1.Department of PsychologyUniversity of California Los AngelesLos AngelesUSA
  2. 2.Department of PsychiatryUniversity of California San DiegoLa JollaUSA
  3. 3.Child and Adolescent Services Research CenterSan DiegoUSA

Personalised recommendations