Skip to main content

Advertisement

SpringerLink
Log in

Facilitators and Barriers of Implementing a Measurement Feedback System in Public Youth Mental Health

  • Original Paper
  • Published:
Administration and Policy in Mental Health and Mental Health Services Research Aims and scope Submit manuscript

Abstract

This study examines implementation facilitators and barriers of a statewide roll-out of a measurement feedback system (MFS) in a youth public mental health system. 76 % of all state care coordinators (N = 47) completed interviews, which were coded via content analysis until saturation. Facilitators (e.g., recognition of the MFS’s clinical utility) and barriers (e.g., MFS’s reliability and validity) emerged paralleling the Exploration, Adoption/Preparation, Implementation, and Sustainment framework outlined by Aarons et al. (Adm Policy Mental Health Mental Health Serv Res, 38:4–23, 2011). Sustainment efforts may leverage innovation fit, individual adopter, and system related facilitators.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

Notes

  1. The Youth Form version of the Ohio Scales refers to the “Wellness” sub-scale while the Parent Form version calls this similar set of 4 items the “Hopefulness” sub-scale.

  2. Initially, the care coordinators were trained to administer the Problem Severity, Functioning, Satisfaction, and Hopefulness/Wellness sub-scales of (48 total items) only to Youth and Parents/Primary Caregivers. However, in response to care coordinators’ concerns about scheduling and administration time difficulties, administration moving into the Implementation and Sustainment phases was restricted to only the Problem Severity and Hopefulness/Wellness sub-scales (24 items).

References

  • Aarons, G. A., Hurlburt, M., & Horwitz, S. M. (2011). Advancing a conceptual model of EBP implementation in public service sectors. Administration and Policy in Mental Health and Mental Health Services Research, 38, 4–23.

    Article  PubMed  Google Scholar 

  • Aarons, G. A., & Palinkas, L. A. (2007). Implementation of evidence-based practice in child welfare: Service provider perspectives. Administration and Policy in Mental Health and Mental Health Services Research, 34(4), 411–419.

    Article  PubMed  Google Scholar 

  • Aarons, G. A., Wells, R. S., Zagursky, G., Fettes, D. L., & Palinkas, L. A. (2009). Implementing evidence-based practice in community mental health agencies: A multiple stakeholder analysis. American Journal of Public Health, 99(11), 2087–2095.

    Article  PubMed  PubMed Central  Google Scholar 

  • American Psychological Association (APA). (2006). Evidence-based practice in psychology. American Psychologist, 61(4), 271–285.

    Article  Google Scholar 

  • Bickman, L. (2008). A measurement feedback system (MFS) is necessary to improve mental health outcomes. Journal of the American Academy of Child and Adolescent Psychiatry, 47(10), 1114–1119.

    Article  PubMed  Google Scholar 

  • Bickman, L., Douglas, K. S., & Athay, M. (2012). The technology of measurement feedback systems. Couple and Family Psychology: Research and Practice, 1(4), 274–284.

    Article  PubMed  PubMed Central  Google Scholar 

  • Biegel, D. E., Shafran, R. D., Johnsen, J. A., & Friedman, S. (2001). Facilitators and barriers to support group participation for family caregivers of adults with mental illness. Community Mental Health Journal, 40(2), 151–166.

    Article  Google Scholar 

  • Bruns, E. J., Hyde, K. L., Sather, A., Hook, A. N., & Lyon, A. R. (2015). Applying user input to the design and testing of an electronic behavioral health information system for wraparound car coordination. Administration and Policy in Mental Health and Mental Health Services Research. doi:10.1007/s10488-015-0658-5.

    PubMed Central  Google Scholar 

  • Chorpita, B. F., Bernstein, A., & Daleiden, E. L. (2008). Driving with roadmaps and dashboards: Using information resources to structure the decision models in service organizations. Administration and Policy in Mental Health and Mental Health Services Research, 35(1–2), 114–123.

    Article  PubMed  Google Scholar 

  • Chorpita, B. F., Daleiden, E. L., & Bernstein, A. (2015). At the intersection of health information technology and decision support: Measurement Feedback Systems…and beyond. Administration and Policy in Mental Health and Mental Health Services Research. Advance online publication. Retrieved from http://www.ncbi.nlm.nih.gov/pubmed/26604202.

  • Chorpita, B. F., & Donkervoet, C. M. (2005). Implementation of the Felix Decree in Hawai‘i: The impact of policy and practice development efforts on service delivery. In R. G. Steele & M. C. Roberts (Eds.), Handbook of mental health services for children, adolescents, and families (pp. 317–332). New York: Kluwer.

    Chapter  Google Scholar 

  • Chorpita, B. F., & Mueller, C. W. (2008). Toward new models for research, community, and consumer partnerships: Some guiding principles and an illustration. Clinical psychology: Science and Practice, 15(2), 144–148.

    Google Scholar 

  • Chorpita, B. F., Yim, L. M., Donkervoet, J. C., Arsendorf, A., Amundsen, M. J., McGee, C., & Morelli, P. (2002a). Toward large-scale implementation of empirically supported treatments for children: A review and observations by the Hawai‘i empirical basis to services task force. Clinical Psychology: Science and Practice, 9, 165–190.

    Google Scholar 

  • Chorpita, B. F., Yim, L. M., Donkervoet, J. C., Arsendorf, A., Amundsen, M. J., McGee, C., et al. (2002b). Toward large-scale implementation of empirically supported treatments for children: A review and observations by the Hawai‘i empirical basis to services task force. Clinical Psychology: Science and Practice, 9, 165–190.

    Google Scholar 

  • Connors, E. H., Arora, P., Curtis, L., & Stephan, S. H. (2015). Evidence-based assessment in school mental health. Cognitive and Behavioral Practice, 22, 60–73.

    Article  Google Scholar 

  • Daleiden, E. L., & Chorpita, B. F. (2005). From data to wisdom: Quality improvement strategies supporting large-scale implementation of evidence based services. Child & Adolescent Psychiatric Clinics of North America, 14, 329–349.

    Article  Google Scholar 

  • Daleiden, E. L., Chorpita, B. F., Donkervoet, C., Arensdorf, A. M., & Brogan, M. (2006). Getting better at getting them better: Health outcomes and evidence-based practice within a system of care. Journal of the American Academy of Child Adolescent Psychiatry, 45(6), 749–756.

    Article  PubMed  Google Scholar 

  • Damanpour, F. (1991). Organization innovation: A meta-analysis of effects of determinants and moderators. Academy of Management Journal, 34(3), 555–590.

    Article  Google Scholar 

  • Damshroeder, L. J., Aron, D. C., Keith, R. E., Kirsh, S. R., Alexander, J. A., & Lowery, J. C. (2009). Fostering implementation of health services research findings into practice: A consolidated framework for advancing implementation science. Implementation Science, 4, 50.

    Article  Google Scholar 

  • Department of Health (DOH), Child and Adolescent Mental Health Division (2012). Child and adolescent mental health performance standards, State of Hawai‘i. Clinical Services Office and Performance Manage Office. Retrieved from http://health.hawaii.gov/camhd/files/2013/07/Orange-Book.pdf.

  • Douglas, K. S., Breda, C., de Andrade, A. R., & Rimer, M. (2011). Effects of routine feedback to clinicians on mental health outcomes of youths: Results of a randomized trial. Psychiatric Services, 62(12), 1423–1429.

    Article  Google Scholar 

  • Duncan, K., & Pozehl, B. (2000). Effects of performance feedback on patient pain outcomes. Clinical Nursing Research, 9(4), 379–397.

    Article  CAS  PubMed  Google Scholar 

  • Fixsen, D., Blase, K., Naoom, S., & Wallace, F. (2009). Core implementation components. Research on Social Work Practice, 19(5), 531.

    Article  Google Scholar 

  • Forman, J., & Damschroder, L. (2008). Qualitative content analyses. In L. Jacoby & L. A. Siminoff (Eds.), Empirical methods for bioethics: A primer (pp. 39–62). Oxford: Elsevier.

    Google Scholar 

  • Fulton, B. D., Scheffler, R. M., Sparkes, S. P., Yoonkyung, A., Vujicic, M., & Soucat, A. (2011). Health workforce skill mix and task shifting in low income countries: a recent evidence. Human Resources for Health. doi:10.1186/1478-4491-9-1.

    PubMed  PubMed Central  Google Scholar 

  • Garland, A. F., Kruse, M., & Aarons, G. A. (2003). Clinicians and outcome measurement: What’s the use? Journal of Behavioral Health Services and Research, 30(4), 393–405.

    Article  PubMed  Google Scholar 

  • Gleacher, A. A., Olin, S. S., Nadeem, E., Pollock, M., Ringle, V., Bickman, L., et al. (2015). Implementing a Measurement Feedback System in community mental health clinics: A case study of multilevel barriers and facilitators. Administration and Policy in Mental Health and Mental Health Services Research. doi:10.1007/s10488-015-0642-0.

    PubMed  Google Scholar 

  • Glisson, C. (2007). Assessing and changing organization culture and climate for effective services. Research on Social Work Practice, 17(6), 736–747.

    Article  Google Scholar 

  • Glisson, C., & Schoenwald, S. (2005). The ARC organizational and community intervention strategy for implementing evidence based children’s mental health treatments. Mental Health Services Research, 7(4), 243–259.

    Article  PubMed  Google Scholar 

  • Goebel, L. J. (1997). A peer review feedback method of promoting compliance with preventive care guidelines in a resident ambulatory care clinic. Joint Commission Journal on Quality Improvement, 23, 196–202.

    CAS  PubMed  Google Scholar 

  • Grol, R., Bosch, M. C., Hulscher, M. E. J. L., Eccles, M. P., & Wensing, M. (2007). Planning and studying improvement in patient car: The use of theoretical perspectives. The Milbank Quarterly, 85(1), 93–138.

    Article  PubMed  PubMed Central  Google Scholar 

  • Hatfield, D. R., & Ogles, B. M. (2007). Why some clinicians use outcome measures and others do not. Administration and Policy in Mental Health and Mental Health Services Research, 34, 283–291.

    Article  PubMed  Google Scholar 

  • Hawkins, R. P., Almeida, M. C., Fabry, B., & Reitz, A. (1992). A scale to measure restrictiveness of living environments for troubled children and youths. Hospital & Community Psychiatry, 43, 54–58.

    CAS  Google Scholar 

  • Hill, C. E., Knox, S., Thompson, B. J., Nutt, W. E., Hess, A., & Ladany, N. (2005). Consensual qualitative research: An update. Journal of Counseling Psychology, 52(2), 196–205.

    Article  Google Scholar 

  • Hill, K. A., Mah, A. C., Kotte, A., Belongie, M. L., Keir, S. S., Higa-McMillan, C. K., & Nakamura, B. J. (2015). Case managers’ attitudes toward standardized assessment within the context of a statewide measurement feedback system initiative. Poster accepted at the 49th Annual Convention of the Association for Behavioral and Cognitive Therapies, Chicago, IL.

  • Hoagwood, K. E., Burns, B., Kiser, L., Ringeisen, H., & Schoenwald, S. (2006). Evidence- based-practice in child and adolescent mental health services. Psychiatric Services, 52(9), 1179–1184.

    Article  Google Scholar 

  • Hoagwood, K. E., Olin, S. S., Horwitz, S., McKay, M., Cleek, A., Gleacher, A., & Hogan, M. (2014). Scaling up evidence-based practices for children and families in New York State: Toward evidence-based policies on implementation for state mental health systems. Journal of Clinical Child and Adolescent Psychology, 43(2), 145–157.

    Article  PubMed  PubMed Central  Google Scholar 

  • Hodges, K. (1989). Child and adolescent functional assessment scale. Ypsilanti: Eastern Michigan University.

    Google Scholar 

  • Hodges, S., Ferreira, K., Israel, N., & Mazza, J. (2006). Leveraging change in the Hawai‘i system of care: Site report for case studies of system implementation. Tampa, FL: University of South Florida, Louis de la Parte Florida Mental Health Institute, Department of Child & Family Studies. Retrieved from http://rtckids.fmhi.usf.edu/rtcpubs/study02/HawaiiSiteReports-2007.pdf.

  • Jensen-Doss, A., & Hawley, K. M. (2010). Understanding barriers to evidence-based assessment: Clinician attitudes toward standardized assessment tools. Journal of Clinical Child & Adolescent Psychology, 39(6), 885–896.

    Article  Google Scholar 

  • Jensen-Doss, A., Hawley, K. M., Lopez, M., & Osterberg, L. D. (2009). Using evidence-based treatments: The experiences of youth providers working under a mandate. Professional Psychology: Research and Practice, 40, 417–424.

    Article  Google Scholar 

  • Keir, S., Jackson, D., Nakamura, B., Ku, J., & Mueller, C. (2012). Ohio Scales assessment project: Pilot implementation of parent and youth forms, February 1 to April 30, 2012. http://hawaii.gov/health/mental-health/camhd/resources/index.html. Retrieved 10 Sep 2012

  • Klein, K. J., & Sorra, J. S. (1996). The challenge of innovation implementation. Academy of Management Review, 21(4), 1055–1080.

    Google Scholar 

  • Kotte, A., Mah, A. C., Hill, K. A., Keir, S. S., Higa-McMillan, C. K., & Nakamura, B. J. (2015). Organizational and mental health provider characteristics associated with implementation of evidence-based practices (EBPs) and monitoring feedback systems (MFS). In A. Kotte (Chair) & K. Hawley (Discussant), Case manager attitudes of standardized assessment predict implementation of the Ohio Scales measurement feedback system in a public mental health system. In Symposium conducted at the Forty-Ninth Annual Convention of the Association of Behavioral and Cognitive Therapies, Chicago, IL.

  • Lambert, M. J. (2010). Yes, it is time for clinicians to routinely monitor treatment outcome. In B. Duncan, S. D. Miller, B. E. Wampold, & M. A. Hubble (Eds.), The heart and soul of change: Delivering what works in therapy (2nd ed., pp. 239–266). Washington, DC: American Psychological Association.

    Chapter  Google Scholar 

  • Landes, S. J., Carlson, E. B., Ruzek, J. I., Wang, D., Hugo, E., DeGaetano, N., & Lindley, S. E. (2014). Provider-driven development of a Measurement Feedback System to enhance measurement-based care in VA mental health. Cognitive and Behavioral Practice, 22(1), 87–100.

    Article  Google Scholar 

  • Latham, G. P., & Locke, E. A. (1991). Self-regulation through goal setting. Organizational Behavior and Human Decision Processes, 50(2), 212–247.

    Article  Google Scholar 

  • Lincoln, Y. S., & Guba, E. G. (2003). Paradigmatic controversies, contradictions, and emerging confluences. In N. K. Denzin & Y. S. Lincoln (Eds.), The landscape of qualitative research: Theories and issues (2nd ed., pp. 253–291). Thousand Oaks, CA: Sage.

    Google Scholar 

  • Lyon, A. R., & Lewis, C. C. (2015). Designing health information technologies for uptake: Development and implementation of Measurement Feedback Systems in mental health service delivery. Introduction to the special section. Administration and Policy in Mental Health and Mental Health Services Research. doi: 10.1007/s10488-015-0637-x.

  • Lyon, A. R., Ludwig, K., Wasse, J. K., Bergstrom, A., Hendrix, E., & McCauley, E. (2016). Determinants and functions of standardized assessment use among school mental health clinicians: A mixed methods evaluation. Administration and Policy in Mental Health and Mental Health Services Research, 43, 122–134.

    Article  PubMed  Google Scholar 

  • Lyon, A. R., Wasse, J. K., Ludwig, K., Zachry, M., Bruns, E. J., Unützer, J., & McCauley, E. (2015). The contextualized technology adaptation process (CTAP): Optimizing health information technology to improve mental health systems. Administration and Policy in Mental Health and Mental Health Services Research. doi:10.1007/s10488-015-0637-x.

    Google Scholar 

  • Maurana, C. A., & Goldenberg, K. (1996). A successful academic-community partnership to improve the public’s health. Academic Medicine, 71(5), 425–431.

    Article  CAS  PubMed  Google Scholar 

  • Mendenhall, E., De Silva, M. J., Hanlon, C., Petersen, I., Shidhaye, R., Jordans, M., & Lund, C. (2014). Acceptability and feasibility of using noon-specialist health workers to deliver mental health care: Stakeholder perceptions from the PRIME district sites in Ethiopia, India, Nepal, South Africa, and Uganda. Social Science and Medicine, 118, 33–42.

    Article  PubMed  PubMed Central  Google Scholar 

  • Murray, L. K., Dorsey, S., Bolton, P., Jordans, M. J. D., Rahman, A., Bass, J., & Verdeli, H. (2011). Building Capacity in mental health interventions in low resource countries: An apprenticeship model for training local providers. Journal of Mental Health Systems. doi:10.1186/1752-4458-5-30.

    Google Scholar 

  • Nakamura, B. J., Higa-McMillan, C. K., Okamura, K. H., & Shimabukuro, S. (2011). Knowledge of and attitudes towards evidence-based practices in community child mental health practitioners. Administration and Policy in Mental Health and Mental Health Services Research, 38, 287–300.

    Article  PubMed  Google Scholar 

  • Nakamura, B. J., Mueller, C. W., Higa-McMillan, C., Okamura, K. H., Chang, J. P., Slavin, L., & Shimabukuro, S. (2014a). Engineering youth service system infrastructure: Hawai‘i’s continued efforts at large-scale implementation through knowledge management strategies. Journal of Clinical Child and Adolescent Psychology, 43(2), 179–189.

    Article  PubMed  Google Scholar 

  • Nakamura, B. J., Slavin, S., Shimabukuro, S., & Keir, S. (2014b). Building and advancing an evidence-based service system in Hawai‘i. In R. Beidas & P. C. Kendall (Eds.), Child and adolescent therapy: Dissemination and implementation of empirically supported treatments (pp. 2014–2219). New York: Oxford University Press.

    Google Scholar 

  • Office of the National Coordinator for Health Information Technology (ONC). (2011). Federal health information technology strategic plan, 2011-2015. Retrieved from https://www.healthit.gov/sites/default/files/utility/final-federal-health-it-strategic-plan-0911.pdf.

  • Ogles, B. M., Melendez, G., Davis, D. C., & Lunnen, K. M. (2001). The Ohio Scales: Practical outcome assessment. Journal of Child and Family Studies, 10(2C), 199–212.

    Article  Google Scholar 

  • Proctor, E., Silmere, H., Raghavan, R., Hovmand, P., Aarons, G., Nunger, A., et al. (2011). Outcomes for implementation research: Conceptual distinctions, measurement challenges, and research agenda. Administration and Policy in Mental Health and Mental Health Services Research, 38(2), 65–76.

    Article  PubMed  Google Scholar 

  • Sandelowski, M., & Barroso, J. (2003). Writing the proposal for a qualitative research methodology project. Qualitative Health Research, 13(6), 781–820.

    Article  PubMed  Google Scholar 

  • Sawyer, D., Gale, J., & Lambert, D. (2006). Rural and frontier mental and behavioral health care: Barriers, effective policy strategies, best practices. Waite Park: National Association for Rural Mental Health.

    Google Scholar 

  • Senge, P. M. (2006). The fifth discipline: The art and practice of the learning organization. New York: Doubleday/Currency.

    Google Scholar 

  • Shim, M. (2010). Factors influencing child welfare employee’s turnover: Focusing on organizational culture and climate. Children and Youth Services Review, 32, 847–856.

    Article  Google Scholar 

  • Stoltenberg, C. D., McNeill, B., & Delworth, U. (1998). IDM supervision: An integrated developmental model for supervising counselors and therapists. San Francisco: Jossey-Bass.

    Google Scholar 

  • Tabenkin, H., Steinmetz, D., Eilat, Z., Heman, H., Dagan, B., & Epstein, L. (1995). A peer review programme to audit the management of hypertensive patients in family practices in Israel. Family Practice, 12(3), 309–312.

    Article  CAS  PubMed  Google Scholar 

  • Whitaker, D. J., Rogers-Brown, J. S., Cowart-Osborne, M., Self-Brown, S., & Lutzker, J. R. (2015). Public child welfare staff knowledge, attitudes, and referral behaviors for an evidence based parenting program. Psychosocial Intervention, 24(2), 89–95.

    Article  Google Scholar 

  • Zmud, R. W. (1982). Diffusion of modern software practices: Influence of centralization and formalization. Management Science, 28(12), 1421–1431.

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. University of Hawai‘i at Mānoa, Honolulu, HI, USA

    Amelia Kotte, Kaitlin A. Hill, Albert C. Mah, Priya A. Korathu-Larson, Sonia Izmirian & Brad J. Nakamura

  2. University of Hawai‘i at Hilo, Hilo, HI, USA

    Janelle R. Au & Charmaine K. Higa-McMillan

  3. Child and Adolescent Mental Health Division, Hawai‘i Department of Health, Honolulu, HI, USA

    Scott S. Keir

Authors
  1. Amelia Kotte
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Kaitlin A. Hill
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Albert C. Mah
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Priya A. Korathu-Larson
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Janelle R. Au
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Sonia Izmirian
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Scott S. Keir
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Brad J. Nakamura
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Charmaine K. Higa-McMillan
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Charmaine K. Higa-McMillan.

Appendix: Qualitative Interview Questions

Appendix: Qualitative Interview Questions

  1. 1.

    First, can you please describe your experience with ongoing assessment of youth on your caseload using the Ohio Scales?

    1. a.

      PROBE: How do you feel about ongoing assessment using the Ohio Scales?

  2. 2.

    What benefits have you noticed to using the Ohio Scales on an ongoing basis?

  3. 3.

    What kind of support have you received for administering the Ohio Scales on an ongoing basis?

  4. 4.

    What challenges or barriers have you experienced with ongoing assessment using the Ohio Scales? Be sure to ask 4c.

    1. a.

      PROBE: When/where/how did you experience this challenge?

    2. b.

      PROBE: How feasible has it been to implement ongoing assessment using the Ohio Scales in your everyday work?

    3. c.

      PROBE: Can you walk me through your process for administering and collecting the Ohio Scales from youth and their caregivers? How long does it take to complete each activity? For example, it is my understanding that Honolulu has and continues to have a high face-to-face meeting rate with their families. I think I also understand that administration of the Ohio Scales may not be feasible at these monthly face-to-face meetings. Could you please walk me through the times when you met with family for their face-to-face, but didn’t give the scale?” Also, are you reading it to you families at every administration?

    4. d.

      PROBE: How have you made the time in your schedule to collect ongoing assessments using the Ohio Scales?

  5. 5.

    Can you please describe your experiences using the Reports to monitor youth progress?

    1. a.

      PROBE: How do you feel about using the Reports to monitor youth progress?

  6. 6.

    What benefits have you noticed to using the Reports?

  7. 7.

    What kind of support have you received for using the Reports to monitor youth progress?

    1. a.

      PROBE: Would you like more support? What kind?

  8. 8.

    What challenges or barriers have you experienced with using the Reports? Be sure to ask 8c.

    1. a.

      PROBE: When/where/how did you experience this challenge?

    2. b.

      PROBE: How feasible has it been to implement the Reports in your everyday work?

    3. c.

      PROBE: Can you walk me through your process for using the Reports to monitor youth progress? Can you tell me how and how often you use the Reports during UMR meetings and clinical supervision with the MHS1, CD, and CP?

    4. d.

      PROBE: How have you made the time in your schedule to utilize the Reports?

  9. 9.

    Is there anything else you’d like to add?

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Kotte, A., Hill, K.A., Mah, A.C. et al. Facilitators and Barriers of Implementing a Measurement Feedback System in Public Youth Mental Health. Adm Policy Ment Health 43, 861–878 (2016). https://doi.org/10.1007/s10488-016-0729-2

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10488-016-0729-2

Keywords

Search

Navigation