Skip to main content
SpringerLink
Log in

Program Evaluation

  • Chapter
  • First Online:
Textbook of Community Psychiatry

  • 770 Accesses

Abstract

The field of program evaluation is expanding rapidly as has the culture of accountability and data-driven decision-making. Accordingly, the need for demonstrating quality outcomes for community psychiatry and public mental health programs is more important than ever. The mechanics of program evaluation can be daunting and will become moreso as the field of program evaluation expands with more nuanced methodologies and recommendations. In fact, the growth is so significant that, to fully review the wealth of available literature, theory, and texts would be nearly impossible. This chapter aims to provide a summative review with the intention that all professionals who are involved in community mental health will be able to engage with the material and benefit from the discussion. This includes some discussion of basic theory and key components of program evaluation, common types of program evaluation, specific considerations for community mental health, and recommendations concerning sources of data and data analysis choices.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 169.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 219.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 219.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  • Aby, M. A. (2020). Case study of implementing grant-funded integrated care in a community mental health center. Journal of Behavioral Health Services & Research, 47, 293–308. https://doi.org/10.1007/s11414-019-09671-7

    Article  Google Scholar 

  • Altschuld, J. W., & Watkins, R. (2014). A primer on needs assessment: More than 40 years of research and practice. New Directions for Evaluation, 2014(144), 5–18. https://doi.org/10.1002/ev.20099

    Article  Google Scholar 

  • Barnow, B. S. & Greenberg, D. H. (2019). Special issue editors’ essay. Evaluation Review, 43(5), 231–265. https://doi.org/10.1177/0193841X19865076

    Article  PubMed  Google Scholar 

  • Brousselle, A., & Champagne, F. (2011). Program theory evaluation: Logic analysis. Evaluation and Program Planning, 34(1), 69–78. https://doi.org/10.1016/j.evalprogplan.2010.04.001

    Article  PubMed  Google Scholar 

  • Bond, G. R., Evans, L., Salyers, M. P., Williams, J., & Kim, H. W. (2000). Measurement of fidelity in psychiatric rehabilitation. Mental Health Services Research, 2, 75–87.

    Article  Google Scholar 

  • Cabassa, L. J., Gomes, A. P., & Lewis-Fernández, R. (2015). What would it take? Stakeholders’ views and preferences for implementing a health care manager program in community mental health clinics under health care reform. Medical Care Research and Review: MCRR, 72(1) 71–95. https://doi.org/10.1177/1077558714563171

    Article  PubMed  Google Scholar 

  • Coryn, C. L. S., Noakes, L. A., Westine, C. D., & SchrÓ§ter, D. C. (2011). A systematic review of theory-driven evaluation practice from 1990 to 2009. American Journal of Evaluation, 32(2), 199–226. https://doi.org/10.1177/1098214010389321

    Article  Google Scholar 

  • Donabedian, A. (2005). Evaluating the quality of medical care. 1966. The Milbank Quarterly, 83(4), 691–729. (Original work published 1966). https://doi.org/10.1111/j.1468-0009.2005.00397.x

    Article  PubMed  PubMed Central  Google Scholar 

  • Fihn, S. D., Francis, J., Clancy, C., Nielson, C., Nelson, K., Rumsfeld, J., Cullen, T., Bates, J., & Graham, G. L. (2014). Insights from advanced analytics at the Veterans Health Administration. Health Affairs, 33(7), 1203–1211. doi: https://doi.org/10.1377/hlthaff.2014.0054

    Article  PubMed  Google Scholar 

  • Fiscella, K., & Sanders, M.R. (2016). Racial and Ethnic Disparities in the Quality of Health Care. Annual Review of Public Health, 37(1), 375–394. https://doi.org/10.1146/annurev-publhealth-032315-021439

    Article  PubMed  Google Scholar 

  • Fortuna, L. R., Tolou-Shams, M., Robles-Ramamurthy, B., & Porche, M. V. (2020). Inequity and the disproportionate impact of COVID-19 on communities of color in the United States: The need for a trauma-informed social justice response. Psychological Trauma: Theory, Research, Practice, and Policy. https://doi.org/10.1016/S2468-2667(20)30164-X

  • Ganju, V. (2006). Mental health quality and accountability: the role of evidence-based practices and performance measurement. Administration and Policy in Mental Health, 33(6), 659–665.

    Article  Google Scholar 

  • Gao, X., Shen, J., Wu, H., & Kreen, H. Y. (2019). Evaluating program effects: Conceptualizing and demonstrating a typology. Evaluation and Program Planning, 72, 88–96. https://doi.org/10.1016/j.evalprogplan.2018.10.008

    Article  PubMed  Google Scholar 

  • Gertler, P. J., Martinez, S., Premand, P., Rawlings, L. B., & Vermeersch, C. M. (2016). Impact evaluation in practice. The World Bank.

    Book  Google Scholar 

  • Gill, S., Kuwahara, R., & Wilce, M. (2016). Through a culturally competent lens: Why the program evaluation standards matter,. Health Promot Pract., 17(1), 5–8. https://doi.org/10.1177/1524839915616364

    Article  PubMed  PubMed Central  Google Scholar 

  • Gowin, D., & Millman, J. (1981). Toward reform of program evaluation. Educational Evaluation and Policy Analysis, 3(6),85–87.

    Google Scholar 

  • Hermann, R. C., Chan, J. A., Provost, S. E., & Chiu, W. T. (2006a). Statistical benchmarks for process measures of quality of care for mental and substance use disorders. Psychiatric Services, 57(10), 1461–1467.

    Article  Google Scholar 

  • Hermann, R. C., Chan, J. A., Zazzali, J. L., & Lerner, D. (2006b). Aligning measurement-based quality improvement with implementation of evidence-based practices. Administration and Policy in Mental Health, 33(6), 636–645.

    Article  Google Scholar 

  • Hermann, R. C., Mattke, S., Somekh, D., Silfverhielm, H., Goldner, E., Glover, G., et al. (2006c). Quality indicators for international benchmarking of mental health care. International Journal of Quality in Health Care, 18 Suppl 1, 31–38.

    Article  Google Scholar 

  • Hill, L. G., Maucione, K., & Hood, B. K. (2007). A focused approach to assessing program fidelity. Prevention Science, 8, 25–34.

    Article  Google Scholar 

  • Jacobson, M. R., & Azzam, T. (2018). The effects of stakeholder involvement on perceptions of an evaluation’s credibility. Evaluation and Program Planning, 68, 64–73. doi: https://doi.org/10.1016/j.evalprogplan.2018.02.006

    Article  PubMed  Google Scholar 

  • Kilbourne, A. M., Beck, K., Spaeth-Rublee, B., Ramanuj, P., O’Brien, R. W., Tomoyasu, N., & Pincus, H. A. (2018). Measuring and improving the quality of mental health care: A global perspective. World Psychiatry, 17(1), 30–38. doi: https://doi.org/10.1002/wps.20482

    Article  PubMed  PubMed Central  Google Scholar 

  • Luchins, D. (2012). Two Approaches to Improving Mental Health Care: Positivist/quantitative versus skill-based/qualitative. Perspectives in Biology and Medicine, 55, 409–34.

    Article  Google Scholar 

  • McGrew, J. H., Bond, G. R., Dietzen, L., & Salyers, M. (1994). Measuring the fidelity of implementation of a mental health program model. Journal of Consulting and Clinical Psychology, 62, 670–678.

    Article  Google Scholar 

  • McHugo, G. J., Drake, R. E., Teague, G. B., & Xie, H. (1999). Fidelity to assertive community treatment and client outcomes in the New Hampshire dual disorders study. Psychiatric Services, 50, 818–824.

    Article  Google Scholar 

  • Moore, Q., Perez-Johnson, I., & Santillano, R. (2019). Decomposing differences in impacts on survey- and administrative-measured earnings from a job training voucher experiment. Evaluation Review, 42(5–6), 515–549. https://doi.org/10.1177/0193841X18799434

    Article  Google Scholar 

  • National Institute of Mental Health. (2011). 2012 Budget report. Retrieved from http://www.nimh.nih.gov/about/budget/cj2012.pdf.

  • Nesbit, B., Hertz, M., Thigpen, S., Castellanos, T., Brown, M., Porter, J., & Williams, A. (2018). Innovative methods for designing actionable program evaluation. J Public Health Manag Practi., 24(Suppl 1), S12–S22. https://doi.org/10.1097/PHH.0000000000000706

    Article  Google Scholar 

  • Pincus, H. A., Scholle, S. H., Spaeth-Rublee, B., Hepner, K. A., & Brown, J. (2016). Quality measures for mental health and substance use: Gaps, opportunities, and challenges. Health Affairs, 35(6), 1000–1008. https://doi.org/10.1377/hlthaff.2016.0027

    Article  PubMed  Google Scholar 

  • Rabarison, K. M., Marcelin, R. A., Bish, C. L., Chandra, G., Massoudi, M. S., & Greenlund, K. J. (2018). Cost analysis of Prevention Research Centers: Instrument development. J Public Health Manag Pract., 24(5), 440–443. doi: https://doi.org/10.1097/PPH.0000000000000706

    Article  PubMed  PubMed Central  Google Scholar 

  • Roberts, M. & Steele, R. (2005). Program evaluation approaches to service delivery in child and family mental health. In Handbook of Mental Health Services for Children, Adolescents, and Families. EBSCO Publishing NetLibrary; printed on 2/14/2011 3:32 pm via University of Alabama-Birmingham eISBN:9780306485602.

    Google Scholar 

  • Rollins, A. L., Kukla, M., Salyers, M. P., McGrew, J. H., Flanagan, M. E., Leslie, D. L., Hunt, M. G., & McGuire, A. B. (2017). Comparing the costs and acceptability of three fidelity assessment methods for Assertive Community Treatment. Adm Policy Ment Health, 44, 810–816. https://doi.org/10.1007/s10488-016-0785-7

    Article  PubMed  Google Scholar 

  • Rushing, S. & Lefkovitz, P. (2008). Learning from top performers. Behav Healthc, 28(2), 28, 30–31.

    Google Scholar 

  • Schalock, R. L. (1995). Outcome-based evaluation. New York: Plenum Press.

    Book  Google Scholar 

  • Schalock, R. L. (2001). Outcome-based evaluation, second edition. EBSCO Publishing-NetLibrary; printed on 2/28/2011 3:12pm via University of Alabama-Birmingham eISBN:9780306464584.

    Google Scholar 

  • Schalock, R. L., Lee, T., Verdugo, M., Swart, K., Claes, C., van Loon, J., & Lee, C. S. (2014). An evidence-based approach to organization evaluation and change in human service organizations evaluation and program planning. Evaluation and program planning, 45, 110–118.

    Article  Google Scholar 

  • Schalock, R. L., Verdugo, M. A., and Gomez, L. E. (2011). Evidence-based practices in the field of intellectual and developmental disabilities: An international consensus approach. Evaluation and Program Planning, 34(3), 273–82. https://doi.org/10.1016/j.evalprogplan.2010.10.004

    Article  PubMed  Google Scholar 

  • Schmidt, E. M., Krahn, D. D., McGuire, M. H., Tavakoli, S., Wright, D. M., Solares, H. E., Lemke, S., & Trafton, J. (2017). Using organizational and clinical performance data to increase the value of mental health care. Psychological Services, 14(1), 13–22. doi: https://doi.org/10.1037/ser0000098

    Article  PubMed  Google Scholar 

  • Schoenwald, S. K., Garland, A. F., Chapman, J. E., Frazier, S. L., Sheidow, A. J., & Southam-Gerow, M. A. (2011). Toward the effective and efficient measurement of implementation fidelity. Administration and Policy in Mental Health, 38(1), 32–43. doi: https://doi.org/10.1007/s10488-010-0321-0

    Article  PubMed  PubMed Central  Google Scholar 

  • Shearer, J., McCrone, P., & Romeo, R. (2016). Economic evaluation of mental health interventions: A guide to costing approaches. PharmacoEconomics, 34, 651–664. https://doi.org/10.1007/s40273-016-0390-3

    Article  PubMed  Google Scholar 

  • Smith, J. D. & Hasan, M. (2020). Quantitative approaches for the evaluation of implementation research studies. Psychiatry Res., 283, 112521. https://doi.org/10.1016/j.psychres.2019.112521

  • Teague, G. B., Mueser, K. T., & Rapp, C.A. (2012). Advances in fidelity measurement for mental health services research: four measures. Psychiatr Serv., 63(8), 765–771.

    Article  Google Scholar 

  • Treasury Board of Canada, Secretariat. (1998). Program Evaluation Methods. Public Affairs Branch.

    Google Scholar 

  • U.S. Department of Health and Human Services, Administration on Children, Youth, and Families. (2016). Qualitative research methods in program evaluation: Considerations for federal staff. https://www.acf.hhs.gov/sites/default/files/acyf/qualitative_research_methods_in_program_evaluation.pdf

  • U.S. Department of Health and Human Services Centers for Disease Control and Prevention. (1999). Morbidity and Mortality Weekly Report: Framework for program evaluation in public health. https://www.cdc.gov/eval/framework/index.htm

  • U.S. Department of Health and Human Services Centers for Disease Control and Prevention. Office of the Director, Office of Strategy and Innovation. (2011). Introduction to program evaluation for public health programs: A self-study guide. https://www.cdc.gov/eval/guide/index.htm

  • Weisner, T. S. & Hay, M. C. (2015). Practice to research: Integrating evidence-based practices with culture and context. Transcultural Psychiatry, 52(2), 222–243. doi: https://doi.org/10.1177/1363461514557066

    Article  PubMed  Google Scholar 

  • Wolff, N., Helminiak, T., Morse, G., Calsyn, R., Klinkenberg, D., & Trusty, M. (1997). Cost-effectiveness evaluation of three approaches to case management for homeless mentally ill clients. American Journal of Psychiatry, 154(3), 341–348.

    Article  Google Scholar 

  • Youker, B. W., Ford, K., & Bayer, N. (2017). Dismissing the goals: A comparison of four goal-dismissive goal-free evaluations. Evaluations and Program Planning, 64, 1–6. https://doi.org/10.1016/j.evalprogplan.2017.05.007

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. University of Alabama at Birmingham, Birmingham, AL, USA

    Alison R. Thomas, Kathleen Hodgin & Robert Savage

  2. Washington State University, Pullman, WA, USA

    Erinn E. Savage

Authors
  1. Alison R. Thomas
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Erinn E. Savage
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Kathleen Hodgin
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Robert Savage
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Alison R. Thomas .

Editor information

Editors and Affiliations

  1. Clinical Professor of Psychiatry, University of Pittsburgh Medical Center, Pittsburgh, PA, USA

    Wesley E. Sowers

  2. Clinical Professor of Psychiatry, New York University Grossman School of Medicine | NYU Langone Health, New York, NY, USA

    Hunter L. McQuistion

  3. Clinical Professor of Psychiatry, Columbia University, Vagelos College of Physicians and Surgeons, New York, NY, USA

    Jules M. Ranz

  4. Professor Emerita of Psychiatry, Department of Psychiatry and Behavioral Neurobiology, University of Alabama at Birmingham, Birmingham, AL, USA

    Jacqueline Maus Feldman

  5. Vice Chair/Professor, Department of Psychiatry, Case Western Reserve School of Medicine, Ohio, OH, USA

    Patrick S. Runnels

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Thomas, A.R., Savage, E.E., Hodgin, K., Savage, R. (2022). Program Evaluation. In: Sowers, W.E., McQuistion, H.L., Ranz, J.M., Feldman, J.M., Runnels, P.S. (eds) Textbook of Community Psychiatry. Springer, Cham. https://doi.org/10.1007/978-3-031-10239-4_52

Download citation

Publish with us

Policies and ethics

Search

Navigation