Implementing a Measurement Feedback System in Community Mental Health Clinics: A Case Study of Multilevel Barriers and Facilitators

  • Alissa A. Gleacher
  • Serene S. Olin
  • Erum Nadeem
  • Michele Pollock
  • Vanesa Ringle
  • Leonard Bickman
  • Susan Douglas
  • Kimberly Hoagwood
Original Article
First Online:

Abstract

Measurement feedback systems (MFSs) have been proposed as a means of improving practice. The present study examined the implementation of a MFS, the Contextualized Feedback System (CFS), in two community-based clinic sites. Significant implementation differences across sites provided a basis for examining factors that influenced clinician uptake of CFS. Following the theoretical implementation framework of Aarons et al. (Adm Policy Mental Health Mental Health Serv Res 38(1):4–23, 2011), we coded qualitative data collected from eighteen clinicians (13 from Clinic U and 5 from Clinic R) who participated in semi-structured interviews about their experience with CFS implementation. Results suggest that clinicians at both clinics perceived more barriers than facilitators to CFS implementation. Interestingly, clinicians at the higher implementing clinic reported a higher proportion of barriers to facilitators (3:1 vs. 2:1); however, these clinicians also reported a significantly higher level of organizational and leadership supports for CFS implementation. Implications of these findings are discussed.

Keywords

Mixed methods Implementation Measurement feedback 
This is a preview of subscription content, log in to check access

References

  1. Aarons, G. A., Hurlburt, M., & Horwitz, S. M. (2011). Advancing a conceptual model of evidence-based practice implementation in public service sectors. Administration and Policy in Mental Health and Mental Health Services Research, 38(1), 4–23.CrossRefPubMedPubMedCentralGoogle Scholar
  2. Aarons, G. A., & Palinkas, L. A. (2007). Implementation of evidence-based practice in child welfare: Service provider perspectives. Administration and Policy in Mental Health and Mental Health Services Research, 34(4), 411–419.CrossRefPubMedGoogle Scholar
  3. Aarons, G. A., & Sawitzky, A. C. (2006). Organizational climate partially mediates the effect of culture on work attitudes and staff turnover in mental health services. Administration and Policy in Mental Health and Mental Health Services Research, 33(3), 289–301.CrossRefPubMedPubMedCentralGoogle Scholar
  4. Aarons, G. A., Wells, R. S., Zagursky, K., Fettes, D. L., & Palinkas, L. A. (2009). Implementing evidence-based practice in community mental health agencies: A multiple stakeholder analysis. American Journal of Public Health, 99(11), 2087–2095.CrossRefPubMedPubMedCentralGoogle Scholar
  5. American Psychological Association (APA) Presidential Task Force on Evidence-Based Practice. (2006). Evidence-based practice in psychology. American Psychologist, 61(4), 271–285.CrossRefGoogle Scholar
  6. Arco, L. (1997). Improving program outcome with process-based performance feedback. Journal of Organizational Behavior Management, 17(1), 37–64.CrossRefGoogle Scholar
  7. Bernard, H. H. R., & Ryan, G. W. (2009). Analyzing qualitative data: Systematic approaches. Thousand Oaks: SAGE publications.Google Scholar
  8. Bickman, L., Douglas Kelley, S., & Athay, M. (2012). The technology of measurement feedback systems. Couple and Family Psychology: Research and Practice, 1(4), 274–284.CrossRefPubMedPubMedCentralGoogle Scholar
  9. Bickman, L., Douglas Kelley, S., Breda, C., de Andrade, A. R., & Riemer, M. (2011). Effects of routine feedback to clinicians on mental health outcomes of youths: results of a randomized trial. Psychiatric Services, 62(12), 1423–1429.CrossRefPubMedGoogle Scholar
  10. Brunette, M., Asher, D., Whitley, R., Lutz, W., Wieder, B., Jones, A., et al. (2008). Implementation of integrated dual disorders treatment: a qualitative analysis of facilitators and barriers. Psychiatric Services, 59(9), 989–995.CrossRefPubMedGoogle Scholar
  11. Bruns, E. J., Hoagwood, K. E., & Hamilton, J. D. (2008). State implementation of evidence-based practice for youths, part I: Responses to the state of the evidence. Journal of the American Academy of Child and Adolescent Psychiatry, 47(4), 369–373.CrossRefPubMedGoogle Scholar
  12. Carr, J. Z., Schmidt, A. M., Ford, J. K., & DeShon, R. P. (2003). Climate perceptions matter: a meta-analytic path analysis relating molar climate, cognitive and affective states, and individual level work outcomes. Journal of Applied Psychology, 88(4), 605.CrossRefPubMedGoogle Scholar
  13. Cebul, R. D. (2008). Using electronic medical records to measure and improve performance. Transactions of the American Clinical Climatological Association, 119, 65–76.PubMedPubMedCentralGoogle Scholar
  14. Chorpita, B. F., Bernstein, A., & Daleiden, E. L. (2008). Driving with roadmaps and dashboards: Using information resources to structure the decision models in service organizations. Administration and Policy in Mental Health and Mental Health Services Research, 35(1–2), 114–123.CrossRefPubMedGoogle Scholar
  15. Chorpita, B. F., Bernstein, A., & Daleiden, E. L. (2011). Empirically guided coordination of multiple evidence-based treatments: An illustration of relevance mapping in children’s mental health services. Journal of Consulting and Clinical Psychology, 79(4), 470–480.CrossRefPubMedGoogle Scholar
  16. Duncan, K., & Pozehl, B. (2000). Effects of performance feedback on patient pain outcomes. Clinical Nursing Research, 9(4), 379–397.CrossRefPubMedGoogle Scholar
  17. Fixsen, D., Blasé, K., Metz, A., & Van Dyke, M. (2013). Statewide implementation of evidence-based programs. Exceptional Children, 79(2), 213–230.Google Scholar
  18. Friese, S. (2011). ATLAS. ti 6 user manual. Berlin, Germany, ATLAS. ti Scientific Software Development GmbH.Google Scholar
  19. Furman, C. E., Adamek, M. S., & Furman, A. G. (1992). The use of an auditory device to transmit feedback to student therapists. Journal of Music Therapy, 29(1), 40–53.CrossRefGoogle Scholar
  20. Glisson, C. (2002). The organizational context of children’s mental health services. Clinical Child and Family Psychology Review, 5(4), 233–253.CrossRefPubMedGoogle Scholar
  21. Glisson, C. (2008). Interventions with organizations: The ARC model. In K. Sowers & C. Dulmus (Eds.), The comprehensive handbook of social work and social welfare. New Jersey: Wiley.Google Scholar
  22. Glisson, C., & Durick, M. (1988). Predictors of job satisfaction and organizational commitment in human service organizations. Administrative Science Quarterly, 33(1), 61–81.CrossRefGoogle Scholar
  23. Glisson, C., & Green, P. (2006). The effects of organizational culture and climate on the access to mental health care in child welfare and juvenile justice systems. Administration and Policy in Mental Health and Mental Health Services Research, 33(4), 433–448.CrossRefPubMedGoogle Scholar
  24. Glisson, C., & Hemmelgarn, A. (1998). The effects of organizational climate and inter-organizational coordination on the quality and outcomes of children’s service systems. Child Abuse and Neglect, 22, 401–421.CrossRefPubMedGoogle Scholar
  25. Glisson, C., Hemmelgarn, A., Green, P., Dukes, D., Atkinson, S., & Williams, N. J. (2012). Randomized trial of the availability, responsiveness, and continuity (ARC) organizational intervention with community-based mental health programs and clinicians serving youth. Journal of the American Academy of Child & Adolescent Psychiatry, 51(8), 780–787.CrossRefGoogle Scholar
  26. Glisson, C., Hemmelgarn, A., Green, P., & Williams, N. J. (2013). Randomized trial of the availability, responsiveness and continuity (ARC) organizational intervention for improving youth outcomes in community mental health programs. Journal of the American Academy of Child and Adolescent Psychiatry, 52(5), 493–500.CrossRefPubMedGoogle Scholar
  27. Glisson, C., & James, L. R. (2002). The cross–level effects of culture and climate in human service teams. Journal of Organizational Behavior, 23, 767–794.CrossRefGoogle Scholar
  28. Glisson, C., Landsverk, J., Schoenwald, S., Kelleher, K., Hoagwood, K. E., Mayberg, S., et al. (2008). Assessing the organizational social context (OSC) of mental health services: Implications for research and practice. Administration and Policy in Mental Health and Mental Health Services Research, 35(1), 98–113.CrossRefPubMedGoogle Scholar
  29. Glisson, C., Schoenwald, S. K., Hemmelgarn, A., Green, P., Dukes, D., Armstrong, K. S., et al. (2010). Randomized trial of MST and ARC in a two-level evidence-based treatment implementation strategy. Journal of Consulting and Clinical Psychology, 78(4), 537.CrossRefPubMedPubMedCentralGoogle Scholar
  30. Goebel, L. J. (1997). A peer review feedback method of promoting compliance with preventive care guidelines in a resident ambulatory care clinic. Joint Commission Journal on Quality Improvement, 23, 196–202.PubMedGoogle Scholar
  31. Greener, J. M., Joe, G. W., Simpson, D. D., Rowan-Szal, G. A., & Lehman, W. E. (2007). Influence of organizational functioning on client engagement in treatment. Journal of Substance Abuse Treatment, 33(2), 139.CrossRefPubMedPubMedCentralGoogle Scholar
  32. Guzzo, R. A., Jette, R. D., & Katzell, R. A. (1985). The effects of psychologically based intervention programs on worker productivity: A meta-analysis. Personnel Psychology, 38(2), 275–291.CrossRefGoogle Scholar
  33. Hall, D. T., & Yip, K. (2014). Career cultures and climates in organizations. In B. Schneider & K. Barbera (Eds.), The oxford handbook of organizational climate and culture. New York: Oxford University Press.Google Scholar
  34. Holmboe, E., Scranton, R., Sumption, K., & Hawkins, R. (1998). Effect of medical record audit and feedback on residents’ compliance with preventive health care guidelines. Academic Medicine, 73(8), 901–903.CrossRefPubMedGoogle Scholar
  35. Howe, A. (1996). Detecting psychological distress: can general practitioners improve their own performance? British Journal of General Practice, 46(408), 407–410.PubMedPubMedCentralGoogle Scholar
  36. Jensen-Doss, A., & Hawley, K. M. (2010). Understanding barriers to evidence-based assessment: Clinician attitudes toward standardized assessment tools. Journal of Clinical Child & Adolescent Psychology, 39(6), 885–896.CrossRefGoogle Scholar
  37. Klein, K. J., & Sorra, J. S. (1996). The challenge of innovation implementation. Academy of Management Review, 21(4), 1055–1080.Google Scholar
  38. Kluger, A. N., & Denisi, A. (1996). The effects of feedback interventions on performance: a historical review, a meta-analysis and a preliminary feedback intervention theory. Psychological Bulletin, 119(2), 254–284.CrossRefGoogle Scholar
  39. Lambert, M., Hansen, N., & Finch, A. (2001a). Patient-focused research: Using treatment outcome data to enhance treatment effects. Journal of Consulting and Clinical Psychology, 69, 159–172.CrossRefPubMedGoogle Scholar
  40. Lambert, M. J., Harmon, C., Slade, K., Whipple, J. L., & Hawkins, E. J. (2005). Providing feedback to psychotherapists on their patients’ progress: Clinical results and practice suggestions. Journal of Clinical Psychology, 61(2), 165–174.CrossRefPubMedGoogle Scholar
  41. Lambert, M. J., Whipple, J. L., Smart, D. W., Vermeersch, D. A., Nielsen, S. L., & Hawkin, E. J. (2001b). The effects of providing therapists with feedback on patient progress during psychotherapy: Are outcomes enhanced? Psychotherapy Research, 11, 49–68.CrossRefPubMedGoogle Scholar
  42. Langley, A. K., Nadeem, E., Kataoka, S. H., Stein, B. D., & Jaycox, L. H. (2010). Evidence-based mental health programs in schools: Barriers and facilitators of successful implementation. School mental health, 2(3), 105–113.CrossRefPubMedPubMedCentralGoogle Scholar
  43. Leshan, L. A., Fitzsimmons, M., Marbella, A., & Gottlieb, M. (1997). Increasing clinical prevention efforts in a family practice residency program through CQI methods. The Joint Commission Journal on Quality Improvement, 23(7), 391–400.PubMedGoogle Scholar
  44. Martin, A. M., Fishman, R., Baxter, L., & Ford, T. (2011). Practitioners’ attitudes towards the use of standardized diagnostic assessment in routine practice: a qualitative study in two child and adolescent mental health services. Clinical Child Psychology and Psychiatry, 16(3), 407–420.CrossRefPubMedGoogle Scholar
  45. Mazonson, P. D., Mathias, S. D., Fifer, S. K., Buesching, D. P., Malek, P., & Patrick, D. L. (1996). The mental health patient profile: Does it change primary care physicians’ practice patterns? The Journal of the American Board of Family Practice, 9(5), 336–345.PubMedGoogle Scholar
  46. Mortenson, B. P., & Witt, J. C. (1998). The use of weekly performance feedback to increase teacher implementation of a pre-referral academic intervention. School Psychology Review, 27, 613–627.Google Scholar
  47. Neuman, G. A., Edwards, J. E., & Raju, N. S. (1989). Organizational development interventions: A meta-analysis of their effects on satisfaction and other attitudes. Personnel Psychology, 42(3), 461–489.CrossRefGoogle Scholar
  48. New Freedom Commission on Mental Health (2003). Achieving the promise: Transforming mental health care in America: Final report. President’s New Freedom Commission on Mental Health. http://govinfo.library.unt.edu/mentalhealthcommission/reports/reports.htm.
  49. New York State (NYS) Office of mental health (OMH) (2011). NYS OMH dashboard. http://bi.omh.ny.gov/cmhp/dashboard.
  50. Olin, S. S., Williams, N., Pollock, M., Armusewicz, K., Kutash, K., Glisson, C., & Hoagwood, K. E. (2014). Quality indicators for family support services and their relationship to organizational social context. Administration and Policy in Mental Health and Mental Health Services Research, 41(1), 43–54.CrossRefPubMedGoogle Scholar
  51. Osborne, D., & Gaebler, T. (1992). Reinventing Government: How the entrepreneurial spirit is transforming the public sector. New York: Plume.Google Scholar
  52. Panzano, P., & Roth, D. (2006). The decision to adopt evidence-based and other innovative mental health practices: Risky business? Psychiatric Services, 57(8), 1153–1161.CrossRefPubMedGoogle Scholar
  53. Parker, C. P., Baltes, B. B., Young, S. A., Huff, J. W., Altmann, R. A., Lacost, H. A., et al. (2003). Relationships between psychological climate perceptions and work outcomes: A meta-analytic review. Journal of Organizational Behavior, 24(4), 389–416.CrossRefGoogle Scholar
  54. Peters, T. J., & Waterman, R. H. (1982). In search of excellence. New York: Harper & Row Inc.Google Scholar
  55. Robertson, P. J., Roberts, D. R., & Porras, J. I. (1993). Dynamics of planned organizational change: Assessing empirical support for a theoretical model. Academy of Management Journal, 36(3), 619–634.CrossRefGoogle Scholar
  56. Robinson, M. B., Thompson, E., & Black, N. A. (1996). Evaluation of the effectiveness of guidelines, audit and feedback: Improving the use of intravenous thrombolysis in patients with suspected acute myocardial infarction. International Journey of Quality Health Care, 8, 211–222.Google Scholar
  57. Rokstad, K., Straand, J., & Fugelli, P. (1995). Can drug treatment be improved by feedback on prescribing profiles combined with therapeutic recommendations? A prospective, controlled trial in general practice. Journal of Clinical Epidemiology, 48, 1061–1068.CrossRefPubMedGoogle Scholar
  58. Rose, D. J., & Church, R. J. (1998). Learning to teach: The acquisition and maintenance of teaching skills. Journal of Behavioral Education, 8(1), 5–35.CrossRefGoogle Scholar
  59. Sapyta, J., Riemer, M., & Bickman, L. (2005). Feedback to clinicians: Theory, research, and practice. Journal of Clinical Psychology, 61(2), 145–153.CrossRefPubMedGoogle Scholar
  60. Sheridan, J. E. (1992). Organizational culture and employee retention. Academy of Management Journal, 35(5), 1036–1056.CrossRefGoogle Scholar
  61. Shim, M. (2010). Factors influencing child welfare employee’s turnover: Focusing on organizational culture and climate. Children and Youth Services Review, 32(6), 847–856.CrossRefGoogle Scholar
  62. Tabenkin, H., Steinmetz, D., Eilat, Z., Heman, H., Dagan, B., & Epstein, L. (1995). A peer review programme to audit the management of hypertensive patients in family practices in Israel. Family Practice, 12(3), 309–312.CrossRefPubMedGoogle Scholar
  63. Tuckman, B. W., & Yates, D. (1980). Evaluating the student feedback strategy for changing teacher style. Journal of Educational Research, 74(2), 74–77.CrossRefGoogle Scholar
  64. Unsworth, G., Cowie, H., & Green, A. (2012). Therapists’ and clients’ perceptions of routine outcome measurement in the NHS: A qualitative study. Counselling and Psychotherapy Research: Linking research with practice, 12(1), 71–80.CrossRefGoogle Scholar
  65. Wisdom, J. P., Chor, K. H. B., Hoagwood, K. E., & Horwitz, S. M. (2013). Innovation adoption: A review of theories and constructs. Administration and Policy in Mental Health and Mental Health Services Research, 41(4), 480–502.CrossRefGoogle Scholar
  66. Wolpert, M., Curtis-Tyler, K., & Edbrooke-Childs, J. (2014). A qualitative exploration of patient and clinician views on patient reported outcome measures in child mental health and diabetes services. Administration and Policy in Mental Health and Mental Health Services Research,. doi: 10.1007/s10488-014-0586-9.Google Scholar

Copyright information

© Springer Science+Business Media New York 2015

Authors and Affiliations

  • Alissa A. Gleacher
    • 1
  • Serene S. Olin
    • 1
  • Erum Nadeem
    • 1
  • Michele Pollock
    • 1
  • Vanesa Ringle
    • 1
  • Leonard Bickman
    • 2
  • Susan Douglas
    • 2
  • Kimberly Hoagwood
    • 1
  1. 1.New York University Langone Medical CenterNew YorkUSA
  2. 2.Vanderbilt UniversityNashvilleUSA

Personalised recommendations