Skip to main content

Advertisement

SpringerLink
Log in

Decision-Making in Mental Health Care: Measuring Provider and Supervisor Use of Evidence

  • Original Article
  • Published:
Administration and Policy in Mental Health and Mental Health Services Research Aims and scope Submit manuscript

Abstract

Use of evidence to inform clinical decisions has been shown to improve the quality and effectiveness of services. This study piloted an observational coding system for understanding providers and supervisors’ use of evidence in their clinical decision-making. The Action Cycle and Use of Evidence Behavioral Observation Coding System (ACE-BOCS) is based on Graham et al. (Contin Educ Health Prof 26:13–24, 2006) conceptual framework for knowledge management, which articulates a sequence relevant to integrating evidence into decisions and actions, including identifying and selecting a problem and choosing, planning, and rehearsing a solution or action. Using the ACE-BOCS, two coders rated the extensiveness with which evidence was used to inform decisions made in clinical supervision sessions. In these clinical supervision sessions, supervisor-provider dyads discussed cases (N = 30; age range 8–19 years; 80% Latino/a or Hispanic ethnicity) that were identified as potentially being at risk for low treatment engagement in school mental health services. Results indicated that the ACE-BOCS can reliably and validly measure use of evidence and distinguish between strategic and indiscriminate use of evidence. The ACE-BOCS has value and utility for studying use of evidence, as it incorporates multiple actions related to service delivery and has the potential to be adapted for other aspects of mental healthcare decision-making beyond clinical supervision, as well as decision making within fields outside of mental health.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1

Similar content being viewed by others

Notes

  1. The ACE-BOCS is intended to measure, among other actions, identification of a problem that needs addressing, identification of a practice, and match between the identified problem and practice as well as several other “use of evidence” behaviors. Match between the identified problem and practice should be determined based on the best available evidence (e.g., randomized clinical trials, theory, provider expertise, etc.). However, mismatch between the identified problem and practice does not necessarily mean that the identified practice is not effective for treating the identified problem—only that existing evidence does not support that practice for addressing that problem.

References

  • Bailin, A., Bearman, S. K., & Sale, R. (2018). Clinical supervision of mental health professionals serving youth: Format and microskills. Administration and Policy in Mental Health and Mental Health Services Research. https://doi.org/10.1007/s10488-018-0865-y.

    Article  PubMed  Google Scholar 

  • Becker, K. D., Boustani, M., Gellatly, R., & Chorpita, B. F. (2018). Forty years of engagement research in children’s mental health services: Multidimensional measurement and practice elements. Journal of Clinical Child and Adolescent Psychology,47(1), 1–23. https://doi.org/10.1080/15374416.2017.1326121.

    Article  PubMed  Google Scholar 

  • Becker, K. D., Park, A. L., Boustani, M. M., & Chorpita, B. F. (2019). A pilot study to examine the feasibility and acceptability of a coordinated intervention design to address treatment engagement challenges in school mental health services. Journal of School Psychology, 76, 78–88. https://doi.org/10.1016/j.jsp.2019.07.013.

    Article  PubMed  Google Scholar 

  • Brennan, S. E., McKenzie, J. E., Turner, T., Redman, S., Makkar, S., Williamson, A., et al. (2017). Development and validation of SEER (Seeking, Engaging with and Evaluating Research): A measure of policymakers’ capacity to engage with and use research. Health Research Policy and Systems,15(1), 1–19. https://doi.org/10.1186/s12961-016-0162-8.

    Article  PubMed  PubMed Central  Google Scholar 

  • Chambless, D. L., & Hollon, S. D. (1998). Defining empirically supported therapies. Journal of Consulting and Clinical Psychology,66(1), 7–18.

    Article  CAS  Google Scholar 

  • Chorpita, B. F., Becker, K. D., & Park, A. L. (2018). Action cycle and use of evidence behavioral observation coding system (ACE-BOCS) [Measurement instrument]. Unpublished instrument.

  • Chorpita, B. F., & Daleiden, E. L. (2009). Mapping evidence-based treatments for children and adolescents: Application of the distillation and matching model to 615 treatments from 322 randomized trials. Journal of Consulting and Clinical Psychology,77(3), 566–579. https://doi.org/10.1037/a0014565.

    Article  PubMed  Google Scholar 

  • Chorpita, B. F., Daleiden, E. L., & Weisz, J. R. (2005). Identifying and selecting the common elements of evidence based interventions: A distillation and matching model. Mental Health Services Research,7(1), 5–20. https://doi.org/10.1007/s11020-005-1962-6.

    Article  PubMed  Google Scholar 

  • Chorpita, B. F., & Regan, J. (2009). Dissemination of effective mental health treatment procedures: Maximizing the return on a significant investment. Behaviour Research and Therapy,47(11), 990–993. https://doi.org/10.1016/j.brat.2009.07.002.

    Article  PubMed  Google Scholar 

  • Darmstadt, G. L., Bhutta, Z. A., Cousens, S., Adam, T., Walker, N., De Bernis, L., et al. (2005). Evidence-based, cost-effective interventions: How many newborn babies can we save? The Lancet,365(9463), 977–988. https://doi.org/10.1016/S0140-6736(05)71088-6.

    Article  Google Scholar 

  • Dawson, K., & Berry, M. (2002). Engaging families in child welfare services: An evidence-based approach to best practice. Child Welfare: Journal of Policy, Practice, and Program,81(2), 293–317.

    Google Scholar 

  • Dorsey, S., Kerns, S. E. U., Lucid, L., Pullmann, M. D., Harrison, J. P., Berliner, L., et al. (2018). Objective coding of content and techniques in workplace-based supervision of an EBT in public mental health. Implementation Science,13(1), 19. https://doi.org/10.1186/s13012-017-0708-3.

    Article  PubMed  Google Scholar 

  • Dorsey, S., Pullmann, M. D., Deblinger, E., Berliner, L., Kerns, S. E., Thompson, K., et al. (2013). Improving practice in community-based settings: A randomized trial of supervision–study protocol. Implementation Science,8, 1–11. https://doi.org/10.1186/1748-5908-8-89.

    Article  Google Scholar 

  • Fixsen, D. L., Naoom, S. F., Blase, K. A., Friedman, R. M., & Wallace, F. (2005). Implementation research: A synthesis of the literature. Tampa, FL: University of South Florida: Louis de la Parte Florida Mental Health Institute, The National Implementation Research Network. (FMHI Publication #231).

  • Flaspohler, P., Lesesne, C. A., Puddy, R. W., Smith, E., & Wandersman, A. (2012). Advances in bridging research and practice: Introduction to the second special issue on the interactive system framework for dissemination and implementation. American Journal of Community Psychology,50(3–4), 271–281. https://doi.org/10.1007/s10464-012-9545-3.

    Article  PubMed  PubMed Central  Google Scholar 

  • Florin, J., Ehrenberg, A., Wallin, L., & Gustavsson, P. (2012). Educational support for research utilization and capability beliefs regarding evidence-based practice skills: A national survey of senior nursing students. Journal of Advanced Nursing,68(4), 888–897. https://doi.org/10.1111/j.1365-2648.2011.05792.x.

    Article  PubMed  Google Scholar 

  • Forgatch, M. S., Patterson, G. R., & DeGarmo, D. S. (2005). Evaluating fidelity: Predictive validity for a measure of competent adherence to the Oregon model of parent management training. Behavior Therapy,36(1), 3–13.

    Article  Google Scholar 

  • Gershater-Molko, R. M., Lutzker, J. R., & Wesch, D. (2003). Project SafeCare: Improving health, safety, and parenting skills in families reported for, and at-risk for child maltreatment. Journal of Family Violence,18(6), 377–386. https://doi.org/10.1023/A:1026219920902.

    Article  Google Scholar 

  • Gotimer, D. H., & Crouse, K. (2019). Studying the use of research evidence: A review of methods. New York, NY: William T. Grant Foundation.

    Google Scholar 

  • Graham, I. D., Logan, J., Harrison, M. B., Straus, S. E., Tetroe, J., Caswell, W., et al. (2006). Lost in knowledge translation: Time for a map? Journal of Continuing Education in the Health Professions,26, 13–24. https://doi.org/10.1002/chp.47.

    Article  PubMed  Google Scholar 

  • Gray, M., Sharland, E., Heinsch, M., & Schubert, L. (2014). Connecting research to action: Perspectives on research utilisation. The British Journal of Social Work,45(7), 1952–1967. https://doi.org/10.1093/bjsw/bcu089.

    Article  Google Scholar 

  • Greenhalgh, T., Robert, G., Macfarlane, F., Bate, P., & Kyriakidou, O. (2004). Diffusion of innovations in service organizations: Systematic review and recommendations. The Milbank Quarterly,82, 581–629. https://doi.org/10.1111/j.0887-378X.2004.00325.x.

    Article  PubMed  PubMed Central  Google Scholar 

  • Gyani, A., Shafran, R., & Myles, P. (2014). The gap between science and practice: How therapists make their clinical decisions. Behavior Therapy,45(2), 199–211. https://doi.org/10.1016/j.beth.2013.10.004.

    Article  PubMed  Google Scholar 

  • Herschell, A. D., Kolko, D. J., Baumann, B. L., & Davis, A. C. (2010). The role of therapist training in the implementation of psychosocial treatments: A review and critique with recommendations. Clinical Psychology Review,30, 448–466. https://doi.org/10.1016/j.cpr.2010.02.005.

    Article  PubMed  PubMed Central  Google Scholar 

  • Jensen-Doss, A., Haimes, E. M. B., Smith, A. M., Lyon, A. R., Lewis, C. C., Stanick, C. F., et al. (2018). Monitoring treatment progress and providing feedback is viewed favorably but rarely used in practice. Administration and Policy in Mental Health and Mental Health Services Research,45(1), 48–61. https://doi.org/10.1007/s10488-016-0763-0.

    Article  PubMed  Google Scholar 

  • Jensen-Doss, A., & Hawley, K. M. (2011). Understanding clinicians’ diagnostic practices: Attitudes toward the utility of diagnosis and standardized diagnostic tools. Administration and Policy in Mental Health and Mental Health Services Research,38(6), 476–485. https://doi.org/10.1007/s10488-011-0334-3.

    Article  PubMed  Google Scholar 

  • Koo, T. K., & Li, M. Y. (2016). A guideline of selecting and reporting intraclass correlation coefficients for reliability research. Journal of Chiropractic Medicine,15(2), 155–163. https://doi.org/10.1016/j.jcm.2016.02.012.

    Article  PubMed  PubMed Central  Google Scholar 

  • Leslie, L. K., Maciolek, S., Biebel, K., Debordes-Jackson, G., & Nicholson, J. (2014). Exploring knowledge exchange at the research–policy–practice interface in children’s behavioral health services. Administration and Policy in Mental Health and Mental Health Services Research,41(6), 822–834. https://doi.org/10.1007/s10488-014-0535-7.

    Article  PubMed  Google Scholar 

  • McDonnell, L. M., & Weatherford, M. S. (2013). Evidence use and the Common Core State Standards movement: From problem definition to policy adoption. American Journal of Education,120(1), 1–25. https://doi.org/10.1086/673163.

    Article  Google Scholar 

  • McLeod, B. D., Southam-Gerow, M. A., Tully, C. B., Rodríguez, A., & Smith, M. M. (2013). Making a case for treatment integrity as a psychosocial treatment quality indicator for youth mental health care. Clinical Psychology: Science and Practice,20(1), 14–32. https://doi.org/10.1111/cpsp.12020.

    Article  Google Scholar 

  • McLeod, B. D., & Weisz, J. R. (2010). The therapy process observational coding system for child psychotherapy strategies scale. Journal of Clinical Child & Adolescent Psychology,39(3), 436–443. https://doi.org/10.1080/15374411003691750.

    Article  Google Scholar 

  • National Advisory Mental Health Council Workgroup on Child and Adolescent Mental Health Intervention Development and Deployment. 2001. Blueprint for Change: Research on Child and Adolescent Mental Health. A Report by the National Advisory Mental Health Council’s Workgroup on Child and Adolescent Mental Health Intervention Development and Deployment. Bethesda, MD: National Institutes of Health, National Institute of Mental Health.

  • Ness, E. C. (2010). The role of information in the policy process: Implications for the examination of research utilization in higher education policy. In M. B. Paulsen (Ed.), Higher education: Handbook of theory and research (pp. 1–49). Dordrecht: Springer.

    Google Scholar 

  • Nutley, S. M., Walter, I., & Davies, H. T. O. (2007). Using evidence: How research can inform public services. Bristol: The Policy Press.

    Book  Google Scholar 

  • Palinkas, L., Garcia, A., Aarons, G., Finno-Velasquez, M., Holloway, I., Mackie, T., et al. (2016). Measuring use of research evidence: The Structured Interview for Evidence Use. Research on Social Work Practice,26, 550–564. https://doi.org/10.1177/1049731514560413.

    Article  PubMed  Google Scholar 

  • Park, A. L., Tsai, K. H., Guan, K., & Chorpita, B. F. (2018). Unintended consequences of evidence-based treatment policy reform: Is implementation the goal or the strategy for higher quality care? Administration and Policy in Mental Health and Mental Health Services Research. https://doi.org/10.1007/s10488-018-0853-2.

    Article  PubMed  Google Scholar 

  • Perepletchikova, F., Hilt, L. M., Chereji, E., & Kazdin, A. E. (2009). Barriers to implementing treatment integrity procedures: Survey of treatment outcome researchers. Journal of Consulting and Clinical Psychology,77(2), 212–218. https://doi.org/10.1037/a0015232.

    Article  PubMed  Google Scholar 

  • Perepletchikova, F., & Kazdin, A. E. (2005). Treatment integrity and therapeutic change: Issues and research recommendations. Clinical Psychology: Science and Practice,12(4), 365–383. https://doi.org/10.1093/clipsy.bpi045.

    Article  Google Scholar 

  • Real, K., & Poole, M. S. (2005). Innovation implementation: Conceptualization and measurement in organizational research. In D. A. Noumair & A. B. Shani (Eds.), Research in organizational change and development (pp. 63–134). Bingley: Emerald Group Publishing Limited.

    Google Scholar 

  • Rones, M., & Hoagwood, K. (2000). School-based mental health services: A research review. Clinical Child and Family Psychology Review,3(4), 223–241. https://doi.org/10.1023/A:1026425104386.

    Article  CAS  PubMed  Google Scholar 

  • Rowley, J. (2007). The wisdom hierarchy: Representations of the DIKW hierarchy. Journal of Information Science,33, 163–180. https://doi.org/10.1177/0165551506070706.

    Article  Google Scholar 

  • Schoenwald, S. K., Garland, A. F., Southam-Gerow, M. A., Chorpita, B. F., & Chapman, J. E. (2011). Adherence measurement in treatments for disruptive behavior disorders: Pursuing clear vision through varied lenses. Clinical Psychology: Science and Practice,18, 331–341. https://doi.org/10.1111/j.1468-2850.2011.01264.x.

    Article  Google Scholar 

  • Shlonsky, A., Noonan, E., Littell, J. H., & Montgomery, P. (2011). The role of systematic reviews and the Campbell Collaboration in the realization of evidence-informed practice. Clinical Social Work Journal,39(4), 362–368. https://doi.org/10.1007/s10615-010-0307-0.

    Article  Google Scholar 

  • Slavin, R. E., Lake, C., & Groff, C. (2009). Effective programs in middle and high school mathematics: A best-evidence synthesis. Review of Educational Research,79(2), 839–911. https://doi.org/10.3102/0034654308330968.

    Article  Google Scholar 

  • Triantafillou, P. (2015). The political implications of performance management and evidence-based policymaking. The American Review of Public Administration,45(2), 167–181. https://doi.org/10.1177/0275074013483872.

    Article  Google Scholar 

  • Tseng, V., & Nutley, S. (2014). Building the infrastructure to improve the use and usefulness of research in education. In K. S. Finnigan & A. J. Daly (Eds.), Using research evidence in education (pp. 163–175). Cham: Springer.

    Chapter  Google Scholar 

  • Vale, M. J., Jelinek, M. V., Best, J. D., & Santamaria, J. D. (2002). Coaching patients with coronary heart disease to achieve the target cholesterol: A method to bridge the gap between evidence-based medicine and the “real world”—randomized controlled trial. Journal of Clinical Epidemiology,55(3), 245–252. https://doi.org/10.1016/S0895-4356(01)00460-7.

    Article  PubMed  Google Scholar 

  • Weiss, C. H. (1977). Using social research in public policy making. Lanham: Lexington Books.

    Google Scholar 

  • Weiss, C. H. (1979). The many meanings of research utilization. Public Administration Review,39(5), 426–431.

    Article  Google Scholar 

  • Weiss, C. H., Murphy-Graham, E., Petrosino, A., & Gandhi, A. G. (2008). The fairy godmother—And her warts: Making the dream of evidence-based policy come true. American Journal of Evaluation,29(1), 29–47. https://doi.org/10.1177/1098214007313742.

    Article  Google Scholar 

  • Weiss, J. A., & Weiss, C. H. (1981). Social scientists and decision makers look at the usefulness of mental health research. American Psychologist,36(8), 837–847. https://doi.org/10.1037/0003-066X.36.8.837.

    Article  CAS  PubMed  Google Scholar 

  • Weisz, J. R., Jensen-Doss, A., & Hawley, K. M. (2006). Evidence-based youth psychotherapies versus usual clinical care: A meta-analysis of direct comparisons. The American Psychologist,61(7), 671–689.

    Article  Google Scholar 

  • Weisz, J. R., Kuppens, S., Eckshtain, D., Ugueto, A. M., Hawley, K. M., & Jensen-Doss, A. (2013). Performance of evidence-based youth psychotherapies compared with usual clinical care: A multilevel meta-analysis. JAMA Psychiatry,70(7), 750–761. https://doi.org/10.1001/jamapsychiatry.2013.1176.

    Article  CAS  PubMed  Google Scholar 

  • Wulczyn, F., Alpert, L., Monahan-Price, K., Huhr, S., Palinkas, L. A., & Pinsoneault, L. (2015). Research evidence use in the child welfare system. Child Welfare,94(2), 141–165.

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Psychology, University of California, Los Angeles, Box 951563, Los Angeles, CA, 90095, USA

    Alayna L. Park & Bruce F. Chorpita

  2. University of South Carolina, Columbia, SC, USA

    Kimberly D. Becker

  3. Loma Linda University, Loma Linda, CA, USA

    Maya M. Boustani

Authors
  1. Alayna L. Park
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Kimberly D. Becker
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Maya M. Boustani
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Bruce F. Chorpita
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Bruce F. Chorpita.

Ethics declarations

Conflict of interest

All Authors declare that they have no conflict of interest.

Ethical Approval

Approval by the institutional review board of the University of California, Los Angeles as well as by those institutional review boards of participating service agencies that requested independent reviews was obtained before the study was conducted. All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards.

Informed Consent

Informed consent was obtained from all individual participants included in the study.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Park, A.L., Becker, K.D., Boustani, M.M. et al. Decision-Making in Mental Health Care: Measuring Provider and Supervisor Use of Evidence. Adm Policy Ment Health 47, 344–356 (2020). https://doi.org/10.1007/s10488-019-00989-0

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10488-019-00989-0

Keywords

Search

Navigation