Abstract
Standardized assessment measures are important for accurate diagnosis of mental health problems and for treatment planning and evaluation. However, little is known about youth mental health providers’ typical use of standardized measures across disciplines and outside the context of evidence-based practice initiatives. A multidisciplinary national survey examined the frequency with which 674 youth mental health providers administer standardized and unstandardized measures, and the extent to which organizational (i.e., implementation climate, rigid hierarchical organizational structure) and provider (i.e., attitudes toward standardized assessment measures, highest degree, practice setting) characteristics are associated with standardized measure use. Providers used unstandardized measures far more frequently than standardized measures. Providers’ perceptions (a) that standardized measures are practical or feasible, (b) that their organization supports and values evidence-based practices, and (c) that their organization has a rigid hierarchical structure predicted greater use of standardized measures. Working in schools predicted less frequent SMU, while working in higher education and other professional settings predicted more frequent SMU. Standardized measures were not routinely used in this community-based sample. A rigid hierarchical organizational structure may be conducive to more frequent administration of standardized measures, but it is unclear whether such providers actually utilize these measures for clinical decision-making. Alternative strategies to promote standardized measure use may include promoting organizational cultures that value empirical data and encouraging use of standardized measures and training providers to use pragmatic standardized measures for clinical decision making.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Data Availability
Participants did not consent to the sharing of data at the time of informed consent, and thus, data are not available. Survey materials are available upon request.
Notes
Given the low factor loadings of the formal mental status exam and the formal clinician ratings, we reran the model without these two variables. Fit remained good (RMSEA = .031, CFI = .939, TLI = .931) and the pattern of results remained the same, with the exception that working in schools no longer predicted SMU.
References
Aarons, G. A. (2004). Mental health provider attitudes toward adoption of evidence-based practice: The Evidence-Based Practice Attitude Scale. Mental Health Services Research, 6(2), 61–74.
Aarons, G. A., Ehrhart, M. G., Farahnak, L. R., & Hurlburt, M. S. (2015). Leadership and organizational change for implementation (LOCI): A randomized mixed method pilot study of a leadership and organization development intervention for evidence-based practice implementation. Implementation Science, 10(1), 1–12. https://doi.org/10.1186/s13012-014-0192-y
Aarons, G. A., Hurlburt, M., & Horwitz, S. M. (2011). Advancing a conceptual model of evidence-based practice implementation in public service sectors. Administration and Policy in Mental Health and Mental Health Services Research, 38(1), 4–23. https://doi.org/10.1007/s10488-010-0327-7
Aarons, G. A., & Palinkas, L. A. (2007). Implementation of evidence-based practice in child welfare: Service provider perspectives. Administration and Policy in Mental Health and Mental Health Services Research, 34(4), 411–419. https://doi.org/10.1007/s10488-007-0121-3
Aarons, G. A., Sommerfeld, D. H., & Walrath-Greene, C. M. (2009). Evidence-based practice implementation: The impact of public versus private sector organization type on organizational support, provider attitudes, and adoption of evidence-based practice. Implementation Science, 4(1), 83. https://doi.org/10.1186/1748-5908-4-83
Ajzen, I. (1991). The theory of planned behavior. Organizational Behavior and Human Decision Processes, 50(2), 179–211. https://doi.org/10.1016/0749-5978(91)90020-T
Basco, M. R., Bostic, J. Q., Davies, D., Rush, A. J., Witte, B., Hendrickse, W., & Barnett, V. (2000). Methods to improve diagnostic accuracy in a community mental health setting. American Journal of Psychiatry, 157(10), 1599–1605. https://doi.org/10.1176/appi.ajp.157.10.1599
Batty, M. J., Moldavsky, M., Foroushani, P. S., Pass, S., Marriott, M., Sayal, K., & Hollis, C. (2013). Implementing routine outcome measures in child and adolescent mental health services: From present to future practice. Child and Adolescent Mental Health, 18(2), 82–87. https://doi.org/10.1111/j.1475-3588.2012.00658.x
Beidas, R. S., Stewart, R. E., Walsh, L., Lucas, S., Downey, M. M., Jackson, K., Fernandez, T., & Mandell, D. S. (2015). Free, brief, and validated: Standardized instruments for low-resource mental health settings. Cognitive and Behavioral Practice, 22(1), 5–19. https://doi.org/10.1016/j.cbpra.2014.02.002
Bickman, L., Kelley, S. D., Breda, C., de Andrade, A. R., & Riemer, M. (2011). Effects of routine feedback to clinicians on mental health outcomes of youths: Results of a randomized trial. Psychiatric Services, 62(12), 1423–1429. https://doi.org/10.1037/e530282011-001
Bickman, L., Rosof-Williams, J., Salzer, M. S., Summerfelt, W. T., Noser, K., Wilson, S. J., & Karver, M. S. (2000). What information do clinicians value for monitoring adolescent client progress and outcomes? Professional Psychology: Research and Practice, 31, 70–74. https://doi.org/10.1037/0735-7028.31.1.70
Cannon, J. A., Warren, J. S., Nelson, P. L., & Burlingame, G. M. (2010). Change trajectories for the Youth Outcome Questionnaire Self-Report: Identifying youth at risk for treatment failure. Journal of Clinical Child & Adolescent Psychology, 39(3), 289–301. https://doi.org/10.1080/15374411003691727
Cashel, M. L. (2002). Child and adolescent psychological assessment: Current clinical practices and the impact of managed care. Professional Psychology: Research and Practice, 33, 446–453. https://doi.org/10.1037//0735-7028.33.5.446
Cho, E., Wood, P. K., Taylor, E. K., Hausman, E. M., Andrews, J. H., & Hawley, K. M. (2019). Evidence-based treatment strategies in youth mental health services: Results from a national survey of providers. Administration and Policy in Mental Health and Mental Health Services Research, 46(1), 71–81. https://doi.org/10.1007/s10488-018-0896-4
Connors, E. H., Arora, P., Curtis, L., & Stephan, S. H. (2015). Evidence-based assessment in school mental health. Cognitive and Behavioral Practice, 22(1), 60–73. https://doi.org/10.1016/j.cbpra.2014.03.008
Cook, J. R., Hausman, E. M., Jensen-Doss, A., & Hawley, K. M. (2017). Assessment practices of child clinicians: Results from a national survey. Assessment, 24 (2), 210–221. https://doi.org/10.1177/1073191115604353
Damschroder, L. J., Aron, D. C., Keith, R. E., Kirsh, S. R., Alexander, J. A., & Lowery, J. C. (2009). Fostering implementation of health services research findings into practice: A consolidated framework for advancing implementation science. Implementation Science, 4(1), 50. https://doi.org/10.1186/1748-5908-4-50
Dillman, D. A. (2000). Mail and internet surveys: The tailored design method (2nd ed.). Wiley.
Ehrhart, M. G., Aarons, G. A., & Farahnak, L. R. (2014). Assessing the organizational context for EBP implementation: The development and validity testing of the Implementation Climate Scale (ICS). Implementation Science, 9(1), 1–11. https://doi.org/10.1186/s13012-014-0157-1
Garland, A. F., Kruse, M., & Aarons, G. A. (2003). Clinicians and outcome measurements: What’s the use? Journal of Behavioral Health Services & Research, 30, 393–405. https://doi.org/10.1007/bf02287427
Geller, B., Tillman, R., Craney, J. L., & Bolhofner, K. (2004). Four-year prospective outcome and natural history of mania in childrenwith a prepubertal and early adolescent bipolar disorder phenotype. Archives of General Psychiatry, 61(5), 459–467. https://doi.org/10.1001/archpsyc.61.5.459
Gilbody, S. M., House, A. O., & Sheldon, T. A. (2002). Psychiatrists in the UK do not use outcomes measures: National survey. The British Journal of Psychiatry, 180(2), 101–103. https://doi.org/10.1192/bjp.180.2.101
Glisson, C., & James, L. R. (2002). The cross-level effects of culture and climate in human service teams. Journal of Organizational Behavior, 23, 767–794. https://doi.org/10.1002/job.162
Glisson, C., & Schoenwald, S. K. (2005). The ARC organizational and community intervention strategy for implementing evidence-based children’s mental health treatments. Mental Health Services Research, 7(4), 243–259. https://doi.org/10.1007/s11020-005-7456-1
Glisson, C., Schoenwald, S. K., Kelleher, K., Landsverk, J., Hoagwood, K. E., Mayberg, S., & Green, P. (2008). Therapist turnover and new program sustainability in mental health clinics as a function of organizational culture, climate, and service structure. Administrative Policy and Mental Health, 35, 124–133. https://doi.org/10.1007/s10488-007-0152-9
Hatfield, D. R., & Ogles, B. M. (2004). The use of outcome measures by psychologists in clinical practice. Professional Psychology: Research and Practice, 35, 485–491. https://doi.org/10.1037/0735-7028.35.5.485
Hu, L. T., & Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling: A Multidisciplinary Journal, 6(1), 1–55. https://doi.org/10.1080/10705519909540118
Hunsley, J., & Mash, E. J. (2007). Evidence-based assessment. Annual Review of Clinical Psychology, 3, 29–51. https://doi.org/10.1146/annurev.clinpsy.3.022806.091419
Jenkins, M. M., Youngstrom, E. A., Washburn, J. J., & Youngstrom, J. K. (2011). Evidence-based strategies improve assessment of pediatric bipolar disorder by community practitioners. Professional Psychology: Research and Practice, 42(2), 121. https://doi.org/10.1037/a0022506
Jensen, A. L., & Weisz, J. R. (2002). Assessing match and mismatch between practitioner-generated and standardized interview-generated diagnoses for clinic-referred children and adolescents. Journal of Consulting and Clinical Psychology, 70, 158–168. https://doi.org/10.1037/0022-006x.70.1.158
Jensen-Doss, A., Haimes, E. M. B., Smith, A. M., Lyon, A. R., Lewis, C. C., Stanick, C. F., & Hawley, K. M. (2018a). Monitoring treatment progress and providing feedback is viewed favorably but rarely used in practice. Administration and Policy in Mental Health and Mental Health Services Research, 45(1), 48–61. https://doi.org/10.1007/s10488-016-0763-0
Jensen-Doss, A., & Hawley, K. M. (2010). Understanding barriers to evidence-based assessment: Clinician attitudes toward standardized assessment tools. Journal of Clinical Child & Adolescent Psychology, 39(6), 885–896. https://doi.org/10.1080/15374416.2010.517169
Jensen-Doss, A., Smith, A. M., Becker-Haimes, E. M., Mora Ringle, V., Walsh, L. M., Nanda, M., Walsh, S. L., Maxwell, C. A., & Lyon, A. R. (2018). Individualized progress measures are more acceptable to clinicians than standardized measures: Results of a national survey. Administration and Policy in Mental Health and Mental Health Services Research, 45(3), 392–403. https://doi.org/10.1007/s10488-017-0833-y
Jensen-Doss, A., & Weisz, J. R. (2008). Diagnostic agreement predicts treatment process and outcomes in youth mental health clinics. Journal of Consulting and Clinical Psychology, 76(5), 711–722. https://doi.org/10.1037/0022-006x.76.5.711
Johnston, C., & Gowers, S. (2005). Routine outcome measurement: A survey of UK child and adolescent mental health services. Child and Adolescent Mental Health, 10(3), 133–139. https://doi.org/10.1111/j.1475-3588.2005.00357.x
Kazdin, A. E., Siegal, T. C., & Bass, D. (1990). Drawing on clinical practice to inform research on child and adolescent psychotherapy: Survey of practitioners. Professional Psychology: Research and Practice, 21, 189–198. https://doi.org/10.1037//0735-7028.21.3.189
Kramer, T. L., Robbins, J. M., Phillips, S. D., Miller, T. L., & Burns, B. J. (2003). Detection and outcomes of substance use disorders in adolescents seeking mental health treatment. Journal of the American Academy of Child and Adolescent Psychiatry, 42, 1318–1326. https://doi.org/10.1097/01.chi.0000084833.67701.44
Krosnick, J. A. (1999). Survey research. Annual Review of Psychology, 50, 537–567. https://doi.org/10.1146/annurev.psych.50.1.537
Lewczyk, C. M., Garland, A. F., Hurlburt, M. S., Gearity, J., & Hough, R. L. (2003). Comparing DISC-IV and clinician diagnoses among youths receiving public mental health services. Journal of the American Academy of Child & Adolescent Psychiatry, 42, 349–356. https://doi.org/10.1097/00004583-200303000-00016
Lloyd, C. E., Duncan, C., & Cooper, M. (2019). Goal measures for psychotherapy: A systematic review of self-report, idiographic instruments. Clinical Psychology: Science and Practice. https://doi.org/10.1111/cpsp.12281
Lui, J. H., Brookman-Frazee, L., Smith, A., Lind, T., Terrones, L., Rodriguez, A., Motamedi, M., Villodas, M., & Lau, A. S. (2021). Implementation facilitation strategies to promote routine progress monitoring among community therapists. Psychological Services, 19(2), 343–352. https://doi.org/10.1037/ser0000456
Lyon, A. R., & Lewis, C. C. (2016). Designing health information technologies for uptake: Development and implementation of measurement feedback systems in mental health service delivery: Introduction to the special section. Administration and Policy in Mental Health and Mental Health Services Research, 43, 344–349. https://doi.org/10.1007/s10488-015-0704-3
Lyon, A. R., Lewis, C. C., Boyd, M. R., Hendrix, E., & Liu, F. (2016). Capabilities and characteristics of digital measurement feedback systems: Results from a comprehensive review. Administration and Policy in Mental Health and Mental Health Services Research, 43, 441–466. https://doi.org/10.1007/s10488-016-0719-4
Lyon, A. R., Pullmann, M. D., Whitaker, K., Ludwig, K., Wasse, J. K., & McCauley, E. (2019). A digital feedback system to support implementation of measurement-based care by school-based mental health clinicians. Journal of Clinical Child & Adolescent Psychology, 48(sup1), S168–S179. https://doi.org/10.1080/15374416.2017.1280808
Muthén, L. K., & Muthén, B. O. (1998–2012). Mplus user’s guide. Seventh edition. Muthén & Muthén.
Palinkas, L. A., & Aarons, G. A. (2009). A view from the top: Executive and management challenges in a statewide implementation of an evidence based practice to reduce child neglect. International Journal of Child Health and Human Development, 2(1), 47–55.
Powell, B. J., Mettert, K. D., Dorsey, C. N., Weiner, B. J., Stanick, C. F., Lengnick-Hall, R., & Lewis, C. C. (2021). Measures of organizational culture, organizational climate, and implementation climate in behavioral health: A systematic review. Implementation Research and Practice, 2, 1–29. https://doi.org/10.1177/26334895211018862
Proctor, E., Ramsey, A. T., Malone, M. T. S., Hooley, C., & McKay, V. (2019). Training in Implementation Practice Leadership (TRIPLE): Evaluation of a novel practice change strategy in behavioral health organizations. Implementation Science, 14, 1–11. https://doi.org/10.1186/s13012-019-0906-2
Rettew, D. C., Lynch, A. D., Achenbach, T. M., Dumenci, L., & Ivanova, M. Y. (2009). Meta-analyses of agreement between diagnoses made from clinical evaluations and standardized diagnostic interviews. International Journal of Methods in Psychiatric Research, 18(3), 169–184. https://doi.org/10.1002/mpr.289
Sale, R., Bearman, S. K., Woo, R., & Baker, N. (2021). Introducing a measurement feedback system for youth mental health: Predictors and impact of implementation in a community agency. Administration and Policy in Mental Health and Mental Health Services Research, 48, 327–342. https://doi.org/10.1007/s10488-020-01076-5
Scott, K., & Lewis, C. C. (2015). Using measurement-based care to enhance any treatment. Cognitive and Behavioral Practice, 22(1), 49–59. https://doi.org/10.1016/j.cbpra.2014.01.010
Sichel, C. E., & Connors, E. H. (2022). Measurement feedback system implementation in public youth mental health treatment services: A mixed methods analysis. Implementation Science Communications, 3(1), 1–16. https://doi.org/10.1186/s43058-022-00356-5
Stein, B. D., Kogan, J. N., Hutchison, S. L., Magee, E. A., & Sorbero, M. J. (2010). Use of outcomes information in child mental health treatment: Results from a pilot study. Psychiatric Services, 61(12), 1211–1216. https://doi.org/10.1176/appi.ps.61.12.1211
Tuma, J. M., & Pratt, J. M. (1982). Clinical child psychology practice and training: A survey. Journal of Clinical Child Psychology, 11, 27–34. https://doi.org/10.1080/15374418209533058
Ventimiglia, J. A., Marschke, J., Carmichael, P., & Loew, R. (2000). How do clinicians evaluate their practice effectiveness? A survey of clinical social workers. Smith College Studies in Social Work, 70(2), 287–306. https://doi.org/10.1080/00377310009517593
Warren, J. S., Nelson, P. L., Mondragon, S. A., Baldwin, S. A., & Burlingame, G. M. (2010). Youth psychotherapy change trajectories and outcomes in usual care: Community mental health versus managed care settings. Journal of Consulting and Clinical Psychology, 78(2), 144–155. https://doi.org/10.1037/a0018544
Weiner, B. J., Belden, C. M., Bergmire, D. M., & Johnston, M. (2011). The meaning and measurement of implementation climate. Implementation Science, 6, 1–12. https://doi.org/10.1186/1748-5908-6-78
Whiteside, S. P., Sattler, A. F., Hathaway, J., & Douglas, K. V. (2016). Use of evidence-based assessment for childhood anxiety disorders in community practice. Journal of Anxiety Disorders, 39, 65–70. https://doi.org/10.1016/j.janxdis.2016.02.008
Williams, N. J., Becker-Haimes, E. M., Schriger, S., & Beidas, R. S. (2022a). Linking organizational climate for evidence-based practice implementation to observed clinician behavior in patient encounters: A lagged analysis. Implementation Science Communications, 3(64), 1–14. https://doi.org/10.1186/s43058-022-00309-y
Williams, N. J., Glisson, C., Hemmelgarn, A., & Green, P. (2017). Mechanisms of change in the ARC organizational strategy: Increasing mental health clinicians’ EBP adoption through improved organizational culture and capacity. Administration and Policy in Mental Health and Mental Health Services Research, 44(2), 269–283. https://doi.org/10.1007/s10488-016-0742-5
Williams, N. J., Hugh, M. L., Cooney, D. J., Worley, J., & Locke, J. (2022b). Testing a theory of implementation leadership and climate across autism evidence-based interventions of varying complexity. Behavior Therapy, 53(5), 900–912. https://doi.org/10.1016/j.beth.2022.03.001
Williams, N. J., Ramirez, N. V., Esp, S., Watts, A., & Marcus, S. C. (2022c). Organization-level variation in therapists’ attitudes toward and use of measurement-based care. Administration and Policy in Mental Health and Mental Health Services Research, 49(6), 927–942. https://doi.org/10.1007/s10488-022-01206-1
Williams, N. J., Wolk, C. B., Becker-Haimes, E. M., & Beidas, R. S. (2020). Testing a theory of strategic implementation leadership, implementation climate, and clinicians’ use of evidence-based practice: a 5-year panel analysis. Implementation Science. https://doi.org/10.1186/s13012-020-0970-7
Funding
This research was supported in part by R03 MH077752 from the National Institute of Mental Health to Kristin M. Hawley.
Author information
Authors and Affiliations
Contributions
Study conception and design, material preparation, and data collection were performed by JRC and KMH. Data analyses were performed by JRC and EC. All authors contributed to the writing of the manuscript and approved the final manuscript.
Corresponding author
Ethics declarations
Competing interests
The authors have no relevant financial or non-financial interests to disclose.
Ethical Approval
All procedures performed in studies involving human participants were in accordance with the ethical standards of the University of Missouri Institutional Review Board and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards.
Consent to Participate
This study was approved for a waiver of signed consent. All participants were provided with a consent statement.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Cho, E., Cook, J.R. & Hawley, K.M. A Structural Model of Organization and Clinician Factors Associated with Standardized Measure Use in a National Survey of Youth Mental Health Providers. Adm Policy Ment Health 50, 876–887 (2023). https://doi.org/10.1007/s10488-023-01286-7
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10488-023-01286-7