Program Evaluation and Quality Management

Chapter

Abstract

The steady outcry for demonstrating quality outcomes for community psychiatry and public mental health programs is long standing and has been responded to in various ways—from a collective yawn in some quarters to outstanding efforts and advances in others. While most of the laborers in the mental health field are highly conscientious and diligent in providing proven, effective and high-quality services, the ability to demonstrate that this is the case can be quite challenging. The mechanics of program evaluation can be daunting when working at the level of an agency or mental health system. More practical efforts typically examine new or distinct programs of limited size and scope and employ methods usually seen in experimental studies of mental health interventions, or quasi-experimental approaches without a defined comparison or control group. Repeated measures approaches can lend themselves to program evaluation projects where baseline data is collected and follow-ups occur at subsequent intervals comparing the same individuals to themselves over time. In addition to these well-established approaches to program evaluation, other valuable and sometimes less demanding methods have been developed. These include program fidelity measurement, quality of life and treatment satisfaction surveys, organizational readiness and atmosphere surveys, evidence-based practice implementation, and reference-based benchmarking methodologies.

Keywords

Transportation Income Stein 

References

  1. Anderson, R. L., Lyons, J. S., Giles, D. M., Price, J. A., & Estle, G. (2003). Reliability of the child and adolescent needs and strengths-mental health (CANS-MH) scale. Journal of Child and Family Studies, 12, 279–289.CrossRefGoogle Scholar
  2. Bond, G. R., Evans, L., Salyers, M. P., Williams, J., & Kim, H. W. (2000). Measurement of fidelity in psychiatric rehabilitation. Mental Health Services Research, 2, 75–87.PubMedCrossRefGoogle Scholar
  3. Bond, G. R., & Salyers, M. P. (2004). Prediction of outcome from the Dartmouth assertive community treatment fidelity scale. CNS Spectrums, 9, 937–942.PubMedGoogle Scholar
  4. Boulet, J., & Boss, M. W. (1991). Reliability and validity of the brief symptom inventory. Psychological Assessment: A Journal of Consulting and Clinical Psychology, 3, 433–437.CrossRefGoogle Scholar
  5. Corbin, J., & Strauss, A. (1990). Grounded theory research: Procedures, canons, and evaluative criteria. Qualitative Sociology, 13(1), 3–21.CrossRefGoogle Scholar
  6. Deming, W. E. (1913). The new economics for industry, government, education. Cambridge, MA: Massachusetts Institute of Technology, Center for Advanced Engineering Study.Google Scholar
  7. Derogatis, L. R., Rickels, K., & Rock, A. F. (1976). The SCL-90 and the MMPI: A step in the validation of a new self-report scale. The British Journal of Psychiatry, 128, 280–289.PubMedCrossRefGoogle Scholar
  8. Dowdy, D. W., Eid, M. P., Sedrakyan, A., Mendez-Tellez, P. A., Pronovost, P. J., Herridge, M. S., & Needham, D. M. (2005). Quality of life in adult survivors of critical illness: A systematic review of the literature. Intensive Care Medicine, 31, 611–620.PubMedCrossRefGoogle Scholar
  9. Eisen, S. V., Dill, D. L., & Grob, M. C. (1994). Reliability and validity of a brief patient-report instrument for psychiatric outcome evaluation. Hospital & Community Psychiatry, 45, 242–247.Google Scholar
  10. Eisen, S. V., Wilcox, M., Leff, H. S., Schaefer, E., & Culhane, M. A. (1999). Assessing behavioral health outcomes in outpatient programs: Reliability and validity of the Basis-32. Journal of Behavioral Health Services and Research, 26, 5–17.PubMedCrossRefGoogle Scholar
  11. Eisen, S., Shaul, J. A., Leff, H. S., Stringfellow, V., Clarridge, B. R., & Cleary, P. D. (2001). Toward a national consumer survey: Evaluation of the CABHS and MHSIP instruments. Journal of Behavioral Health Services and Research, 28, 347–369.PubMedCrossRefGoogle Scholar
  12. Ganju, V., Beall, M. A., Callahan, N., Dailey, W., Dumont, J., Hornik, J. A., et al. (1996). The MHSIP consumer-oriented mental health report card. Washington, DC: Center for Mental Health Services (SAMHSA, DHHS).Google Scholar
  13. Ganju, V. (2006). Mental health quality and accountability: The role of evidence-based practices and performance measurement. Administration and Policy in Mental Health, 33(6), 659–665.PubMedCrossRefGoogle Scholar
  14. Gortner, S., & Schultz, P. (1988). Approaches to nursing science methods. Image, 20, 22–23.Google Scholar
  15. Gowin, D., & Millman, J. (1981). Toward reform of program evaluation. Educational Evaluation and Policy Analysis, 3(6), 85–87.Google Scholar
  16. Hermann, R. C., Chan, J. A., Provost, S. E., & Chiu, W. T. (2006a). Statistical benchmarks for process measures of quality of care for mental and substance use disorders. Psychiatric Services, 57(10), 1461–1467.PubMedCrossRefGoogle Scholar
  17. Hermann, R. C., Chan, J. A., Zazzali, J. L., & Lerner, D. (2006b). Aligning measurement-based quality improvement with implementation of evidence-based practices. Administration and Policy in Mental Health, 33(6), 636–645.PubMedCrossRefGoogle Scholar
  18. Hermann, R. C., Mattke, S., Somekh, D., Silfverhielm, H., Goldner, E., Glover, G., et al. (2006c). Quality indicators for international benchmarking of mental health care. International Journal for Quality in Health Care, 18(Suppl 1), 31–38.PubMedCrossRefGoogle Scholar
  19. Hill, L. G., Maucione, K., & Hood, B. K. (2007). A focused approach to assessing program fidelity. Prevention Science, 8, 25–34.PubMedCrossRefGoogle Scholar
  20. Jerrell, J. (2006). Psychometrics of the MHSIP adult consumer survey. Journal of Behavioral Health Services and Research, 33(4), 483–488.PubMedCrossRefGoogle Scholar
  21. Lefkovitz, P. M. (2004). Beyond outcomes: Benchmarking in behavioral healthcare. Behavioral Healthcare Tomorrow, 13(1), 32–37.PubMedGoogle Scholar
  22. Lehman, A. (1988). A quality of life interview for the chronically ill. Evaluation and Program Planning, 11(1), 51–62.CrossRefGoogle Scholar
  23. Lutterman, T., Ganju, V., Schacht L., Shaw R., Monihan K., and Huddle M. (2003). National Association of State Mental Health Program Directors, Sixteen State study on mental health performance measures.Google Scholar
  24. McGrew, J. H., Bond, G. R., Dietzen, L., & Salyers, M. (1994). Measuring the fidelity of implementation of a mental health program model. Journal of Consulting and Clinical Psychology, 62, 670–678.PubMedCrossRefGoogle Scholar
  25. McHugo, G. J., Drake, R. E., Teague, G. B., & Xie, H. (1999). Fidelity to assertive community treatment and client outcomes in the New Hampshire dual disorders study. Psychiatric Services, 50, 818–824.PubMedGoogle Scholar
  26. Mowbray, C. T., Holter, M. C., Teague, G. B., & Bybee, D. (2003). Fidelity criteria: Development, measurement, and validation. American Journal of Evaluation, 24, 315–340.Google Scholar
  27. National Institute of Mental Health. (2011). 2012 Budget report. Retrieved from http://www.nimh.nih.gov/about/budget/cj2012.pdf.
  28. National Alliance on Mental Illness. (2007). Assertive community treatment: Investment yields outcomes. Retrieved from http://www.nami.org/Template.cfm?Section=ACT-TA_Center&template=/ContentManagement/ContentDisplay.cfm&ContentID=52382.
  29. Olsen, L. R., Mortensen, E. L., & Bech, P. (2004). The SCL-90 and SCL-90R versions validated by item response models in a Danish community sample. Acta Psychiatrica Scandinavica, 110, 225–229.PubMedCrossRefGoogle Scholar
  30. Praed Foundation. (2007). About the ANSA. Retrieved from http://www.praedfoundation.org/About%20the%20ANSA.html.
  31. Praed Foundation. (2008). Children and adolescent needs and strengths mental health manual. Retrieved from http://www.praedfoundation.org/CANS-MH%20Manual.pdf.
  32. Roberts, M., & Steele, R. (Eds.). (2005). Program evaluation approaches to service delivery in child and family mental health. In Handbook of mental health services for children, adolescents, and families. EBSCO Publishing NetLibrary. [Printed Feb 14, 2011 3:32 pm via University of Alabama-Birmingham, eISBN:9780306485602].Google Scholar
  33. Rushing, S., & Lefkovitz, P. (2008). Learning from top performers. Behavioral Healthcare, 28(2), 30–31.Google Scholar
  34. Schalock, R. L. (1995). Outcome-based evaluation. New York: Plenum.CrossRefGoogle Scholar
  35. Schalock, R. L. (2001). Outcome-based evaluation (2nd ed.). EBSCO Publishing-NetLibrary. [Printed on Feb 28, 2011 3:12 pm via University of Alabama-Birmingham, eISBN:9780306464584].Google Scholar
  36. Schalock, R. L., & Bonham, G. S. (2003). Measuring outcomes and managing for results. Evaluation and Program Planning, 26, 229–235.CrossRefGoogle Scholar
  37. Schalock, R. L., Verdugo, M. A., & Gomez, L. E. (2010). Evidence-based practices in the field of intellectual and developmental disabilities: An international consensus approach. Evaluation and Program Planning, 34, 273–282.PubMedCrossRefGoogle Scholar
  38. Soderberg, P., Tungstrom, S., & Armelius, B. A. (2005). Reliability of global assessment of functioning ratings made by clinical psychiatric staff. Psychiatric Services, 56, 434–438.PubMedCrossRefGoogle Scholar
  39. Spieth, L. E., & Harris, C. V. (1996). Assessment of health-related quality of life in children and adolescents: An integrative review. Journal of Pediatric Psychology, 21, 175–193.PubMedCrossRefGoogle Scholar
  40. Stein, L. I., & Test, M. A. (1980). Alternative to mental hospital treatment. I. Conceptual model, treatment program and clinical evaluation. Archives of General Psychiatry, 37, 392–397.PubMedCrossRefGoogle Scholar
  41. Teague, G. B., Bond, G. R., & Drake, R. E. (1998). Program fidelity in assertive community treatment: Development and use of a measure. The American Journal of Orthopsychiatry, 68, 216–232.PubMedCrossRefGoogle Scholar
  42. Wolff, N., Helminiak, T., Morse, G., Calsyn, R., Klinkenberg, D., & Trusty, M. (1997). Cost-effectiveness evaluation of three approaches to case management for homeless mentally ill clients. The American Journal of Psychiatry, 154(3), 341–348.PubMedGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC 2012

Authors and Affiliations

  1. 1.Department of Psychiatry and Behavioral NeurobiologyUniversity of Alabama at BirminghamBirminghamUSA
  2. 2.Department of Psychiatry and Behavioral NeurobiologyUniversity of Alabama at BirminghamBirminghamUSA
  3. 3.University of Alabama at BirminghamBirminghamUSA

Personalised recommendations