Skip to main content

Advertisement

SpringerLink
Log in

Implementing a Measurement Feedback System in Community Mental Health Clinics: A Case Study of Multilevel Barriers and Facilitators

  • Original Article
  • Published:
Administration and Policy in Mental Health and Mental Health Services Research Aims and scope Submit manuscript

Abstract

Measurement feedback systems (MFSs) have been proposed as a means of improving practice. The present study examined the implementation of a MFS, the Contextualized Feedback System (CFS), in two community-based clinic sites. Significant implementation differences across sites provided a basis for examining factors that influenced clinician uptake of CFS. Following the theoretical implementation framework of Aarons et al. (Adm Policy Mental Health Mental Health Serv Res 38(1):4–23, 2011), we coded qualitative data collected from eighteen clinicians (13 from Clinic U and 5 from Clinic R) who participated in semi-structured interviews about their experience with CFS implementation. Results suggest that clinicians at both clinics perceived more barriers than facilitators to CFS implementation. Interestingly, clinicians at the higher implementing clinic reported a higher proportion of barriers to facilitators (3:1 vs. 2:1); however, these clinicians also reported a significantly higher level of organizational and leadership supports for CFS implementation. Implications of these findings are discussed.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • Aarons, G. A., Hurlburt, M., & Horwitz, S. M. (2011). Advancing a conceptual model of evidence-based practice implementation in public service sectors. Administration and Policy in Mental Health and Mental Health Services Research, 38(1), 4–23.

    Article  PubMed  PubMed Central  Google Scholar 

  • Aarons, G. A., & Palinkas, L. A. (2007). Implementation of evidence-based practice in child welfare: Service provider perspectives. Administration and Policy in Mental Health and Mental Health Services Research, 34(4), 411–419.

    Article  PubMed  Google Scholar 

  • Aarons, G. A., & Sawitzky, A. C. (2006). Organizational climate partially mediates the effect of culture on work attitudes and staff turnover in mental health services. Administration and Policy in Mental Health and Mental Health Services Research, 33(3), 289–301.

    Article  PubMed  PubMed Central  Google Scholar 

  • Aarons, G. A., Wells, R. S., Zagursky, K., Fettes, D. L., & Palinkas, L. A. (2009). Implementing evidence-based practice in community mental health agencies: A multiple stakeholder analysis. American Journal of Public Health, 99(11), 2087–2095.

    Article  PubMed  PubMed Central  Google Scholar 

  • American Psychological Association (APA) Presidential Task Force on Evidence-Based Practice. (2006). Evidence-based practice in psychology. American Psychologist, 61(4), 271–285.

    Article  Google Scholar 

  • Arco, L. (1997). Improving program outcome with process-based performance feedback. Journal of Organizational Behavior Management, 17(1), 37–64.

    Article  Google Scholar 

  • Bernard, H. H. R., & Ryan, G. W. (2009). Analyzing qualitative data: Systematic approaches. Thousand Oaks: SAGE publications.

    Google Scholar 

  • Bickman, L., Douglas Kelley, S., & Athay, M. (2012). The technology of measurement feedback systems. Couple and Family Psychology: Research and Practice, 1(4), 274–284.

    Article  PubMed  PubMed Central  Google Scholar 

  • Bickman, L., Douglas Kelley, S., Breda, C., de Andrade, A. R., & Riemer, M. (2011). Effects of routine feedback to clinicians on mental health outcomes of youths: results of a randomized trial. Psychiatric Services, 62(12), 1423–1429.

    Article  PubMed  Google Scholar 

  • Brunette, M., Asher, D., Whitley, R., Lutz, W., Wieder, B., Jones, A., et al. (2008). Implementation of integrated dual disorders treatment: a qualitative analysis of facilitators and barriers. Psychiatric Services, 59(9), 989–995.

    Article  PubMed  Google Scholar 

  • Bruns, E. J., Hoagwood, K. E., & Hamilton, J. D. (2008). State implementation of evidence-based practice for youths, part I: Responses to the state of the evidence. Journal of the American Academy of Child and Adolescent Psychiatry, 47(4), 369–373.

    Article  PubMed  Google Scholar 

  • Carr, J. Z., Schmidt, A. M., Ford, J. K., & DeShon, R. P. (2003). Climate perceptions matter: a meta-analytic path analysis relating molar climate, cognitive and affective states, and individual level work outcomes. Journal of Applied Psychology, 88(4), 605.

    Article  PubMed  Google Scholar 

  • Cebul, R. D. (2008). Using electronic medical records to measure and improve performance. Transactions of the American Clinical Climatological Association, 119, 65–76.

    PubMed  PubMed Central  Google Scholar 

  • Chorpita, B. F., Bernstein, A., & Daleiden, E. L. (2008). Driving with roadmaps and dashboards: Using information resources to structure the decision models in service organizations. Administration and Policy in Mental Health and Mental Health Services Research, 35(1–2), 114–123.

    Article  PubMed  Google Scholar 

  • Chorpita, B. F., Bernstein, A., & Daleiden, E. L. (2011). Empirically guided coordination of multiple evidence-based treatments: An illustration of relevance mapping in children’s mental health services. Journal of Consulting and Clinical Psychology, 79(4), 470–480.

    Article  PubMed  Google Scholar 

  • Duncan, K., & Pozehl, B. (2000). Effects of performance feedback on patient pain outcomes. Clinical Nursing Research, 9(4), 379–397.

    Article  CAS  PubMed  Google Scholar 

  • Fixsen, D., Blasé, K., Metz, A., & Van Dyke, M. (2013). Statewide implementation of evidence-based programs. Exceptional Children, 79(2), 213–230.

    Google Scholar 

  • Friese, S. (2011). ATLAS. ti 6 user manual. Berlin, Germany, ATLAS. ti Scientific Software Development GmbH.

  • Furman, C. E., Adamek, M. S., & Furman, A. G. (1992). The use of an auditory device to transmit feedback to student therapists. Journal of Music Therapy, 29(1), 40–53.

    Article  Google Scholar 

  • Glisson, C. (2002). The organizational context of children’s mental health services. Clinical Child and Family Psychology Review, 5(4), 233–253.

    Article  PubMed  Google Scholar 

  • Glisson, C. (2008). Interventions with organizations: The ARC model. In K. Sowers & C. Dulmus (Eds.), The comprehensive handbook of social work and social welfare. New Jersey: Wiley.

    Google Scholar 

  • Glisson, C., & Durick, M. (1988). Predictors of job satisfaction and organizational commitment in human service organizations. Administrative Science Quarterly, 33(1), 61–81.

    Article  Google Scholar 

  • Glisson, C., & Green, P. (2006). The effects of organizational culture and climate on the access to mental health care in child welfare and juvenile justice systems. Administration and Policy in Mental Health and Mental Health Services Research, 33(4), 433–448.

    Article  PubMed  Google Scholar 

  • Glisson, C., & Hemmelgarn, A. (1998). The effects of organizational climate and inter-organizational coordination on the quality and outcomes of children’s service systems. Child Abuse and Neglect, 22, 401–421.

    Article  CAS  PubMed  Google Scholar 

  • Glisson, C., Hemmelgarn, A., Green, P., Dukes, D., Atkinson, S., & Williams, N. J. (2012). Randomized trial of the availability, responsiveness, and continuity (ARC) organizational intervention with community-based mental health programs and clinicians serving youth. Journal of the American Academy of Child & Adolescent Psychiatry, 51(8), 780–787.

    Article  Google Scholar 

  • Glisson, C., Hemmelgarn, A., Green, P., & Williams, N. J. (2013). Randomized trial of the availability, responsiveness and continuity (ARC) organizational intervention for improving youth outcomes in community mental health programs. Journal of the American Academy of Child and Adolescent Psychiatry, 52(5), 493–500.

    Article  PubMed  Google Scholar 

  • Glisson, C., & James, L. R. (2002). The cross–level effects of culture and climate in human service teams. Journal of Organizational Behavior, 23, 767–794.

    Article  Google Scholar 

  • Glisson, C., Landsverk, J., Schoenwald, S., Kelleher, K., Hoagwood, K. E., Mayberg, S., et al. (2008). Assessing the organizational social context (OSC) of mental health services: Implications for research and practice. Administration and Policy in Mental Health and Mental Health Services Research, 35(1), 98–113.

    Article  PubMed  Google Scholar 

  • Glisson, C., Schoenwald, S. K., Hemmelgarn, A., Green, P., Dukes, D., Armstrong, K. S., et al. (2010). Randomized trial of MST and ARC in a two-level evidence-based treatment implementation strategy. Journal of Consulting and Clinical Psychology, 78(4), 537.

    Article  PubMed  PubMed Central  Google Scholar 

  • Goebel, L. J. (1997). A peer review feedback method of promoting compliance with preventive care guidelines in a resident ambulatory care clinic. Joint Commission Journal on Quality Improvement, 23, 196–202.

    CAS  PubMed  Google Scholar 

  • Greener, J. M., Joe, G. W., Simpson, D. D., Rowan-Szal, G. A., & Lehman, W. E. (2007). Influence of organizational functioning on client engagement in treatment. Journal of Substance Abuse Treatment, 33(2), 139.

    Article  PubMed  PubMed Central  Google Scholar 

  • Guzzo, R. A., Jette, R. D., & Katzell, R. A. (1985). The effects of psychologically based intervention programs on worker productivity: A meta-analysis. Personnel Psychology, 38(2), 275–291.

    Article  Google Scholar 

  • Hall, D. T., & Yip, K. (2014). Career cultures and climates in organizations. In B. Schneider & K. Barbera (Eds.), The oxford handbook of organizational climate and culture. New York: Oxford University Press.

    Google Scholar 

  • Holmboe, E., Scranton, R., Sumption, K., & Hawkins, R. (1998). Effect of medical record audit and feedback on residents’ compliance with preventive health care guidelines. Academic Medicine, 73(8), 901–903.

    Article  CAS  PubMed  Google Scholar 

  • Howe, A. (1996). Detecting psychological distress: can general practitioners improve their own performance? British Journal of General Practice, 46(408), 407–410.

    CAS  PubMed  PubMed Central  Google Scholar 

  • Jensen-Doss, A., & Hawley, K. M. (2010). Understanding barriers to evidence-based assessment: Clinician attitudes toward standardized assessment tools. Journal of Clinical Child & Adolescent Psychology, 39(6), 885–896.

    Article  Google Scholar 

  • Klein, K. J., & Sorra, J. S. (1996). The challenge of innovation implementation. Academy of Management Review, 21(4), 1055–1080.

    Google Scholar 

  • Kluger, A. N., & Denisi, A. (1996). The effects of feedback interventions on performance: a historical review, a meta-analysis and a preliminary feedback intervention theory. Psychological Bulletin, 119(2), 254–284.

    Article  Google Scholar 

  • Lambert, M., Hansen, N., & Finch, A. (2001a). Patient-focused research: Using treatment outcome data to enhance treatment effects. Journal of Consulting and Clinical Psychology, 69, 159–172.

    Article  CAS  PubMed  Google Scholar 

  • Lambert, M. J., Harmon, C., Slade, K., Whipple, J. L., & Hawkins, E. J. (2005). Providing feedback to psychotherapists on their patients’ progress: Clinical results and practice suggestions. Journal of Clinical Psychology, 61(2), 165–174.

    Article  PubMed  Google Scholar 

  • Lambert, M. J., Whipple, J. L., Smart, D. W., Vermeersch, D. A., Nielsen, S. L., & Hawkin, E. J. (2001b). The effects of providing therapists with feedback on patient progress during psychotherapy: Are outcomes enhanced? Psychotherapy Research, 11, 49–68.

    Article  CAS  PubMed  Google Scholar 

  • Langley, A. K., Nadeem, E., Kataoka, S. H., Stein, B. D., & Jaycox, L. H. (2010). Evidence-based mental health programs in schools: Barriers and facilitators of successful implementation. School mental health, 2(3), 105–113.

    Article  PubMed  PubMed Central  Google Scholar 

  • Leshan, L. A., Fitzsimmons, M., Marbella, A., & Gottlieb, M. (1997). Increasing clinical prevention efforts in a family practice residency program through CQI methods. The Joint Commission Journal on Quality Improvement, 23(7), 391–400.

    CAS  PubMed  Google Scholar 

  • Martin, A. M., Fishman, R., Baxter, L., & Ford, T. (2011). Practitioners’ attitudes towards the use of standardized diagnostic assessment in routine practice: a qualitative study in two child and adolescent mental health services. Clinical Child Psychology and Psychiatry, 16(3), 407–420.

    Article  PubMed  Google Scholar 

  • Mazonson, P. D., Mathias, S. D., Fifer, S. K., Buesching, D. P., Malek, P., & Patrick, D. L. (1996). The mental health patient profile: Does it change primary care physicians’ practice patterns? The Journal of the American Board of Family Practice, 9(5), 336–345.

    CAS  PubMed  Google Scholar 

  • Mortenson, B. P., & Witt, J. C. (1998). The use of weekly performance feedback to increase teacher implementation of a pre-referral academic intervention. School Psychology Review, 27, 613–627.

    Google Scholar 

  • Neuman, G. A., Edwards, J. E., & Raju, N. S. (1989). Organizational development interventions: A meta-analysis of their effects on satisfaction and other attitudes. Personnel Psychology, 42(3), 461–489.

    Article  Google Scholar 

  • New Freedom Commission on Mental Health (2003). Achieving the promise: Transforming mental health care in America: Final report. President’s New Freedom Commission on Mental Health. http://govinfo.library.unt.edu/mentalhealthcommission/reports/reports.htm.

  • New York State (NYS) Office of mental health (OMH) (2011). NYS OMH dashboard. http://bi.omh.ny.gov/cmhp/dashboard.

  • Olin, S. S., Williams, N., Pollock, M., Armusewicz, K., Kutash, K., Glisson, C., & Hoagwood, K. E. (2014). Quality indicators for family support services and their relationship to organizational social context. Administration and Policy in Mental Health and Mental Health Services Research, 41(1), 43–54.

    Article  PubMed  Google Scholar 

  • Osborne, D., & Gaebler, T. (1992). Reinventing Government: How the entrepreneurial spirit is transforming the public sector. New York: Plume.

    Google Scholar 

  • Panzano, P., & Roth, D. (2006). The decision to adopt evidence-based and other innovative mental health practices: Risky business? Psychiatric Services, 57(8), 1153–1161.

    Article  PubMed  Google Scholar 

  • Parker, C. P., Baltes, B. B., Young, S. A., Huff, J. W., Altmann, R. A., Lacost, H. A., et al. (2003). Relationships between psychological climate perceptions and work outcomes: A meta-analytic review. Journal of Organizational Behavior, 24(4), 389–416.

    Article  Google Scholar 

  • Peters, T. J., & Waterman, R. H. (1982). In search of excellence. New York: Harper & Row Inc.

    Google Scholar 

  • Robertson, P. J., Roberts, D. R., & Porras, J. I. (1993). Dynamics of planned organizational change: Assessing empirical support for a theoretical model. Academy of Management Journal, 36(3), 619–634.

    Article  Google Scholar 

  • Robinson, M. B., Thompson, E., & Black, N. A. (1996). Evaluation of the effectiveness of guidelines, audit and feedback: Improving the use of intravenous thrombolysis in patients with suspected acute myocardial infarction. International Journey of Quality Health Care, 8, 211–222.

    CAS  Google Scholar 

  • Rokstad, K., Straand, J., & Fugelli, P. (1995). Can drug treatment be improved by feedback on prescribing profiles combined with therapeutic recommendations? A prospective, controlled trial in general practice. Journal of Clinical Epidemiology, 48, 1061–1068.

    Article  CAS  PubMed  Google Scholar 

  • Rose, D. J., & Church, R. J. (1998). Learning to teach: The acquisition and maintenance of teaching skills. Journal of Behavioral Education, 8(1), 5–35.

    Article  Google Scholar 

  • Sapyta, J., Riemer, M., & Bickman, L. (2005). Feedback to clinicians: Theory, research, and practice. Journal of Clinical Psychology, 61(2), 145–153.

    Article  PubMed  Google Scholar 

  • Sheridan, J. E. (1992). Organizational culture and employee retention. Academy of Management Journal, 35(5), 1036–1056.

    Article  Google Scholar 

  • Shim, M. (2010). Factors influencing child welfare employee’s turnover: Focusing on organizational culture and climate. Children and Youth Services Review, 32(6), 847–856.

    Article  Google Scholar 

  • Tabenkin, H., Steinmetz, D., Eilat, Z., Heman, H., Dagan, B., & Epstein, L. (1995). A peer review programme to audit the management of hypertensive patients in family practices in Israel. Family Practice, 12(3), 309–312.

    Article  CAS  PubMed  Google Scholar 

  • Tuckman, B. W., & Yates, D. (1980). Evaluating the student feedback strategy for changing teacher style. Journal of Educational Research, 74(2), 74–77.

    Article  Google Scholar 

  • Unsworth, G., Cowie, H., & Green, A. (2012). Therapists’ and clients’ perceptions of routine outcome measurement in the NHS: A qualitative study. Counselling and Psychotherapy Research: Linking research with practice, 12(1), 71–80.

    Article  Google Scholar 

  • Wisdom, J. P., Chor, K. H. B., Hoagwood, K. E., & Horwitz, S. M. (2013). Innovation adoption: A review of theories and constructs. Administration and Policy in Mental Health and Mental Health Services Research, 41(4), 480–502.

    Article  Google Scholar 

  • Wolpert, M., Curtis-Tyler, K., & Edbrooke-Childs, J. (2014). A qualitative exploration of patient and clinician views on patient reported outcome measures in child mental health and diabetes services. Administration and Policy in Mental Health and Mental Health Services Research,. doi:10.1007/s10488-014-0586-9.

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. New York University Langone Medical Center, New York, USA

    Alissa A. Gleacher, Serene S. Olin, Erum Nadeem, Michele Pollock, Vanesa Ringle & Kimberly Hoagwood

  2. Vanderbilt University, Nashville, USA

    Leonard Bickman & Susan Douglas

Authors
  1. Alissa A. Gleacher
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Serene S. Olin
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Erum Nadeem
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Michele Pollock
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Vanesa Ringle
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Leonard Bickman
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Susan Douglas
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Kimberly Hoagwood
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Kimberly Hoagwood.

Appendices

Appendix: Interview Guide

Hi, my name is________. I’m speaking with you today because we want to get more information about aspects of the CFS project that have gone well for you, or aspects that could be improved. We will be recording these calls for note-taking purposes, but all of your responses are confidential and won’t be shared with other members of your agency. Your answers will be used to better understand this quality improvement initiative. We really appreciate your time and willingness to speak with us. Do you have any questions before we get started?

Let’s start with some general questions about your overall experience during this project.

General Overview

  1. 1.

    How would you describe your experience using CFS? What do you like or dislike?

  2. 2.

    What supports have been helpful when using CFS?

  3. 3.

    What have been some of the barriers to successful implementation?

Quality of Training, User manuals, and Technical Assistance

  1. 1.

    What do you think of the support provided by the Columbia/Vanderbilt team? If subject needs prompting, ask the subject specifically about the phases of CFS contextualization:

    1. a.

      Training

    2. b.

      Consultation

    3. c.

      Technical support

  2. 2.

    Would you work in collaboration with the Columbia/Vanderbilt team again? Why or why not?

Management Support

  1. 1.

    How well has your agency supported the implementation of CFS? (For example: giving time, training, administrative support) If subject needs prompting, ask the following:

    1. a.

      Do you think CFS is a priority at the management level? Why or why not?

    2. b.

      How much do you feel like other clinic obligations or initiatives impact your CFS use?

  2. 2.

    How have your co-workers responded to CFS?

  3. 3.

    Is there someone at your agency that particularly helped support CFS? What does he or she do that is helpful?

Technical Issues

  1. 1.

    How did you feel about the design of the program? (For example: the layout, interface, graphics, etc.) If subject needs prompting, ask the following:

    1. a.

      What are some of the technical difficulties, if any, with the program?

    2. b.

      Which components of CFS do you feel most comfortable using?

    3. c.

      Which components of CFS do you feel least comfortable using?

  2. 2.

    What could have been done differently to make you feel more comfortable with the technical aspects of using CFS?

Now I’m going to ask you about the clinical application of CFS and how the program fits with your clinical practice.

Clinical Use

  1. 1.

    Do you think the measures reported by the program accurately reflect your client’s current state? Does it mesh with what you see in session?

  2. 2.

    Is there sufficient time in the course of a session for clients and caregivers to complete the CFS questionnaires? Is there sufficient time to discuss their responses?

  3. 3.

    Did you feel comfortable interpreting CFS data? If not, what would be helpful to increase your comfort level?

  4. 4.

    How does using CFS inform your clinical practice? Can you think of a time when it has (or hasn’t) changed the way you handled a case?

  5. 5.

    Overall, do you believe a program like CFS could be helpful to you in a clinical setting? Why or why not?

  6. 6.

    How do the kids you work with respond to the use of CFS? If subject needs prompting, ask the following:

    1. a.

      What aspects of the program do kids struggle with?

    2. b.

      What aspects of the program do kids use successfully?

  7. 7.

    How do caregivers respond to the use of CFS? If subject needs prompting, ask the following:

    1. a.

      What aspects of the program do caregivers struggle with?

    2. b.

      What aspects of the program do caregivers use successfully?

  8. 8.

    How could CFS be better adapted to suit the needs of your clients and their caregivers (the population you work with)?

Other

  1. 1.

    Some of your new clients were randomly assigned to have data available all the time, and some only had data available every six months. What did you think about the randomization process? How did it affect your engagement with CFS?

  2. 2.

    Is there any other feedback you’d like to offer regarding your experience with CFS?

Those are all of the questions I have for you today. Thank you again for participating! We have one final piece of this interview–it’s an online survey that you should be able to complete quickly. I’ll send you the link to the survey now. It would be great if you could complete the survey as soon as possible, while your experience with CFS is still fresh in your mind.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Gleacher, A.A., Olin, S.S., Nadeem, E. et al. Implementing a Measurement Feedback System in Community Mental Health Clinics: A Case Study of Multilevel Barriers and Facilitators. Adm Policy Ment Health 43, 426–440 (2016). https://doi.org/10.1007/s10488-015-0642-0

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10488-015-0642-0

Keywords

Search

Navigation