The Incremental Value of Self-Reported Mental Health Measures in Predicting Functional Outcomes of Veterans

  • Susan V. Eisen
  • Kathryn A. Bottonari
  • Mark E. Glickman
  • Avron SpiroIII
  • Mark R. Schultz
  • Lawrence Herz
  • Robert Rosenheck
  • Ethan S. Rofman
Article

Abstract

Research on patient-centered care supports use of patient/consumer self-report measures in monitoring health outcomes. This study examined the incremental value of self-report mental health measures relative to a clinician-rated measure in predicting functional outcomes among mental health service recipients. Participants (n = 446) completed the Behavior and Symptom Identification Scale, the Brief Symptom Inventory, and the Veterans/Rand Short Form-36 at enrollment in the study (T1) and 3 months later (T2). Global Assessment of Functioning (GAF) ratings, mental health service utilization, and psychiatric diagnoses were obtained from administrative data files. Controlling for demographic and clinical variables, results indicated that improvement based on the self-report measures significantly predicted one or more functional outcomes (i.e., decreased likelihood of post-enrollment psychiatric hospitalization and increased likelihood of paid employment), above and beyond the predictive value of the GAF. Inclusion of self-report measures may be a useful addition to performance measurement efforts.

Notes

Acknowledgements

This research was supported by VA grant IIR 02-290 from the Veterans Administration Health Services Research & Development (HSR&D) Service. The work was conducted at the Center for Health Quality, Outcomes and Economic Research, a Health Services Research and Development Center of Excellence of the Veterans Administration located at the Edith Nourse Rogers Memorial Veterans Hospital, Bedford, MA. The views expressed in this article are those of the authors and do not necessarily represent the views of the Department of Veterans Affairs. We thank Nicole DelVecchio for her assistance with manuscript preparation.

RH: Incremental Value of Self-Reported Mental Health Measures

References

  1. 1.
    Donabedian A: Explorations in Quality Assessment and Monitoring, Vol. I: The Definition of Quality and Approaches to its Assessment, Ann Arbor, MI, Health Administration Press, 1980.Google Scholar
  2. 2.
    Eisen SV, Grob MC, Dill DL. Outcome Measurement: Tapping the Patient’s Perspective. In Psychiatric Treatment, Advances in Outcome Research, Mirin SM, Gossett J, Grob MC (eds.) Washington, DC: American Psychiatric Press, 213-235, 1991Google Scholar
  3. 3.
    Young A, Grusky O, Jordan D, et al. Routine Outcome Monitoring in a Public Mental Health System: The Impact of Patients Who Leave Care. Psychiatric Services 2000; 51: 85-91.Google Scholar
  4. 4.
    Deegan PE, Drake RE. Shared Decision Making and Medication Management in the Recovery Process. Psychiatric Services 2006; 57:1636-1639.PubMedCrossRefGoogle Scholar
  5. 5.
    Gertais M, Edgman-Levitan S, Daley J, et al. (Eds.) Through the Patient’s Eyes. Understanding and Promoting Patient-Centered Care. San Francisco: Jossey-Bass, 1993.Google Scholar
  6. 6.
    Schauer C, Everett A, del Vecchio P, et al. Promoting the value and practice of shared decision-making in mental health care. Psychiatric Rehabilitation Journal 2007; 31: 54-61.PubMedCrossRefGoogle Scholar
  7. 7.
    Sechrest L, McKnight P, McKnight K. Calibration of measures for psychotherapy outcome research. American Psychologist 1996; 51: 1065-1071.PubMedCrossRefGoogle Scholar
  8. 8.
    Campbell JF. The Consumer Movement and implications for vocational rehabilitation services. Journal of Vocational Rehabilitation 1:67-75, 1991.Google Scholar
  9. 9.
    Ellison ML, Danley KS, Crean T, et al. Involvement of people with psychiatric disabilities in state agencies of vocational rehabilitation: state agency survey. Journal of Rehabilitation Administration 20:319-333, 1996.Google Scholar
  10. 10.
    Chinman M, Allende M, Weingarten R, et al. On the road to collaborative treatment planning: consumer and provider perspective. Journal of Behavioral Health Services & Research 1999; 26: 211-218.CrossRefGoogle Scholar
  11. 11.
    Chinman M, Young AS, Hassell J, et al. Toward the implementation of mental health consumer provider services. Journal of Behavioral Health Services & Research 2006; 33:176-195.CrossRefGoogle Scholar
  12. 12.
    Hansen T, Hatling T, Lidal E, et al. The user perspective: respected or reject in mental health care? Journal of Psychiatric and Mental Health Nursing 2004; 11: 292-297.PubMedCrossRefGoogle Scholar
  13. 13.
    Guthrie D, McIntosh M, Callaly T, et al. Consumer attitudes towards the use of routine outcome measures in a public mental health service: a consumer-driven study. International Journal of Mental Health Nursing 2008; 17: 92-97.PubMedCrossRefGoogle Scholar
  14. 14.
    Garland AF, Kruse M, & Aarons GA. Clinicians and outcome measurement: what’s the use? Journal of Behavioral Health Services & Research 2003; 30: 393-405.CrossRefGoogle Scholar
  15. 15.
    Gilbody SM, House AO, & Sheldon, TA. Outcomes research in mental health. British Journal of Psychiatry 2002; 181: 8-16.PubMedCrossRefGoogle Scholar
  16. 16.
    Gilbody SM, House AO, & Sheldon, TA. Routine administration of Health Related Quality of Life (HRQoL) and needs assessment instruments to improve psychological outcome—a systematic review. Psychological Medicine 2002; 32: 1345-1356.PubMedCrossRefGoogle Scholar
  17. 17.
    Slade M. What outcomes to measure in routine mental health services, and how to assess them: a systematic review. Australian and New Zealand Journal of Psychiatry 2002; 36: 743-753.PubMedCrossRefGoogle Scholar
  18. 18.
    Slade M. Routine outcome assessment in mental health services. Psychological Medicine, 2002.Google Scholar
  19. 19.
    Garb HN. Incremental validity and the assessment of psychopathology in adults. Psychological Assessment 2003; 15: 508-520.PubMedCrossRefGoogle Scholar
  20. 20.
    Haynes SN & Lench HC. Incremental validity of new clinical assessment measures. Psychological Assessment 2003; 15: 456-466.PubMedCrossRefGoogle Scholar
  21. 21.
    Hunsley J. Introduction to the special section on incremental validity and utility in clinical assessment. Psychological Assessment 2003; 15: 443-445.PubMedCrossRefGoogle Scholar
  22. 22.
    Overall JE, Gorham DR. The Brief Psychiatric Rating Scale. Psychological Reports 1962; 10: 799-812.CrossRefGoogle Scholar
  23. 23.
    Spitzer RL, Gibbon M, Williams JB, et al. Global Assessment of Functioning (GAF) scale. In: Sederer LI, Dickey B (eds). Outcomes Assessment in Clinical Practice. Baltimore, Williams & Wilkins, 1996.Google Scholar
  24. 24.
    American Psychiatric Association: Diagnostic and Statistical Manual of Mental Disorders (Text Revision): DSM-IV-TR (ed 4). Washington, DC: American Psychiatric Association, 2000.CrossRefGoogle Scholar
  25. 25.
    Greenberg GA, Rosenheck, RA. Using the GAF as a national mental health outcome measure in the Department of Veterans Affairs. Psychiatric Services 2005; 56: 420-426.PubMedCrossRefGoogle Scholar
  26. 26.
    Greenberg GA, Rosenheck RA. Continuity of care and clinical outcomes in a national health system. Psychiatric Services 2005; 56: 427-433.PubMedCrossRefGoogle Scholar
  27. 27.
    Jones SH, Thornicroft G, Coffey M et al. A Brief Mental Health Outcome Scale: reliability and validity. British Journal of Psychology 1995; 166: 654-659.CrossRefGoogle Scholar
  28. 28.
    Jovanovi AA, Gasi MJ, Ivkovi M, et al. Reliability and validity of DSM-IV Axis V scales in a clinical sample of veterans with posttraumatic stress disorder. Psychiatria Danubina 2008; 20: 286-300.Google Scholar
  29. 29.
    Schwartz, RC. Concurrent validity of the Global Assessment of Functioning Scale for clients with schizophrenia. Psychological Reports 2007; 100: 571-574.PubMedCrossRefGoogle Scholar
  30. 30.
    Sernyak MJ, Leslie D, Rosenheck R. Use of system-wide outcomes monitoring data to compare the effectiveness of atypical neuroleptic medications. American Journal of Psychiatry 2003; 160: 310-315.PubMedCrossRefGoogle Scholar
  31. 31.
    Startup M, Jackson MC, Bendix S. The concurrent validity of the Global Assessment of Functioning. British Journal of Clinical Psychology 2002; 41: 417-422.PubMedCrossRefGoogle Scholar
  32. 32.
    Bates LW, Lyons JA, Shaw JB. Effects of Brief Training on Application of the Global Assessment of Functioning Scale. Psychological Reports 2002; 91: 999-1006.PubMedCrossRefGoogle Scholar
  33. 33.
    Moos RH, McCoy L, Moos BS. Global Assessment of Functioning (GAF) ratings: determinants and role as predictors of one-year treatment outcomes. Journal of Clinical Psychology 2000; 56: 449-461.PubMedCrossRefGoogle Scholar
  34. 34.
    Moos RH, Nichol AC, Moos BS. Global Assessment of Functioning ratings and the allocation and outcomes of mental health services. Psychiatric Services 2002; 53: 730-737.PubMedCrossRefGoogle Scholar
  35. 35.
    Roy-Byrne P, Dagadakis C, Unutzer J, et al. Evidence for limited validity of the Revised Global Assessment of Functioning Scale. Psychiatric Services 1996; 47: 864-866.PubMedGoogle Scholar
  36. 36.
    Söderberg P, Tungström S, Armelius BA. Reliability of Global Assessment of Functioning ratings made by clinical psychiatric staff. Psychiatric Services 2005; 56: 434-438.PubMedCrossRefGoogle Scholar
  37. 37.
    Zastrow A, Faude V, Seyboth F, et al. Risk factors of symptom underestimation by physicians. Journal of Psychosomatic Research 2008; 64: 543-551.PubMedCrossRefGoogle Scholar
  38. 38.
    Ellwood P, Lundberg G. Managed care: a work in progress. Journal of the American Medical Association 1996; 276: 1083-1086.PubMedCrossRefGoogle Scholar
  39. 39.
    Hughner RS & Kleine SS. Variations in Lay Health Theories: Implications for Consumer Health Care Decision Making. Qualitative Health Research 2008; 18: 1687-1703.CrossRefGoogle Scholar
  40. 40.
    Atkinson M, Zibin S, Chuang H. Characterizing quality of life among patients with chronic mental illness: a critical examination of the self-report methodology. American Journal of Psychiatry 1997; 154: 99-105.PubMedGoogle Scholar
  41. 41.
    Dickey B, Wagenaar H. Evaluating mental health care reform: including the clinician, client and family perspective. Journal of Mental Health Administration 1994; 21: 313-319.PubMedCrossRefGoogle Scholar
  42. 42.
    Hamera EK, Schneider JK, Potocky M, et al. Validity of self-administered symptom scales in clients with schizophrenia and schizoaffective disorders. Schizophrenia Research 1996; 19: 213-219.PubMedCrossRefGoogle Scholar
  43. 43.
    Niv N, Cohen AN, Sullivan G, et al. The MIRECC version of the Global Assessment of Functioning scale: reliability and validity. Psychiatric Services 2007; 58: 529-535.PubMedCrossRefGoogle Scholar
  44. 44.
    Hunter R, Cameron R, Norrie J. Using patient-reported outcomes in schizophrenia: the Scottish Schizophrenia Outcomes Study. Psychiatric Services 2009; 60: 240-245.PubMedCrossRefGoogle Scholar
  45. 45.
    Zuvekas SH, Fleishman JA. Self-rated mental health and racial/ethnic disparities in mental health service use. Medical Care 2008; 46: 915-923.PubMedCrossRefGoogle Scholar
  46. 46.
    Statement of The Honorable Robert H. Roswell, MD, Under Secretary for Health, Dept. of Veterans Affairs, March 2, 2004Google Scholar
  47. 47.
    VA Mental Health Consumer Council. VHA mental health strategic plan. Consumer Council Newsletter 2004; 7: 1-2.Google Scholar
  48. 48.
    The President’s New Freedom Commission on Mental Health Report. www.mentalhealthcommission.gov/
  49. 49.
    Zimmerman M, McGlinchey JB, Posternak MA, et al. Remission in depressed outpatients: more than just symptom resolution?. Journal of Psychiatric Research 2008; 42: 797-801.PubMedCrossRefGoogle Scholar
  50. 50.
    Zimmerman M, Ruggero CJ, Chelminski, I, et al. Developing brief scales for use in clinical practice: the reliability and validity of single-item self-report measures of depressive symptom severity, psychosocial impairment due to depression, and quality of life. Journal of Clinical Psychiatry 2006; 67: 1536-1541.PubMedCrossRefGoogle Scholar
  51. 51.
    Burns RB, Moskowitz MA, Ash A, et al. Self-report versus medical record functional status. Medical Care 1992; 30: MS85-MS95.PubMedCrossRefGoogle Scholar
  52. 52.
    Distefano MK, Pryer MW, Garrison JL. Validity of psychiatric patient’s self-reports of rehospitalization. Hospital & Community Psychiatry 1991; 12: 849-850.Google Scholar
  53. 53.
    Russo J, Roy-Byrne P, Jaffe C, et al. The relationship of patient-administered outcome assessments to quality of life and physician ratings: validity of the BASIS-32. Journal of Mental Health Administration 1997; 22:200-214.CrossRefGoogle Scholar
  54. 54.
    Mausbach BT, Harvey PD, Goldman SR, et al. Development of a brief scale of everyday functioning in persons with serious mental illness. Schizophrenia Bulletin 2007; 33:1364-1372.PubMedCrossRefGoogle Scholar
  55. 55.
    Mundt JC, Marks IM, Shear MK, et al. The Work and Social Adjustment Scale: a simple measure of impairment in functioning. British Journal of Psychiatry 2002; 180:461-464.PubMedCrossRefGoogle Scholar
  56. 56.
    Henry AD, & Lucca AM. Facilitators and barriers to employment: the perspectives of people with psychiatric disabilities and employment service providers. Work 2004; 22:169-182.PubMedGoogle Scholar
  57. 57.
    Hendryx MS, Russo JE, Stegner B, et al. Predicting rehospitalization and outpatient services from administration and clinical databases. The Journal of Behavioral Health Services & Research 2003; 30: 342-351.CrossRefGoogle Scholar
  58. 58.
    Stergiopoulos V, Dewa CS, Rouleau K, et al. Collaborative mental health care for the homeless: the role of psychiatry in positive housing and mental health outcomes. Canadian Journal of Psychiatry - Revue Canadienne de Psychiatrie 2008; 53: 61-67.PubMedGoogle Scholar
  59. 59.
    Rosenheck R, Lam JA. Homeless mentally ill clients’ and providers’ perceptions of service needs and clients’ use of services. Psychiatric Services 1997; 48: 381-386.PubMedGoogle Scholar
  60. 60.
    Clements KM, Murphy JM, Eisen SV, et al. Comparison of self-report and clinician-rated measures of psychiatric symptoms and functioning in predicting 1-year hospital readmission. Administration and Policy in Mental Health & Mental Health Services Research 2006; 33: 568-577.CrossRefGoogle Scholar
  61. 61.
    Salyers MP, Godfrey JL, Mueser KT, et al. Measuring illness management outcomes: a psychometric study of clinician and consumer rating scales for illness self-management and recovery. Community Mental Health Journal 2007; 43:459-480.PubMedCrossRefGoogle Scholar
  62. 62.
    Eisen SV, Normand SLT, Belanger A. et al. The revised Behavior and Symptom Identification Scale (BASIS-24): reliability and validity. Medical Care 2004; 42: 1230-1241.PubMedCrossRefGoogle Scholar
  63. 63.
    Derogatis LR. Brief Symptom Inventory (BSI)—Administration, Scoring and Procedures Manual (3 rd edition). Minneapolis: National Computer systems, 1993.Google Scholar
  64. 64.
    Derogatis LR, Savitz KL. The SCL-90-R, Brief Symptom Inventory, and Matching Clinical Rating Scales. In: Maruish M. (ed). The Use of Psychological Testing for Treatment Planning and Outcome Assessment. New Jersey: Lawrence Erlbaum Associates, 1999, 679-724.Google Scholar
  65. 65.
    Ware JE, Sherbourne CD. The MOS 36-item Short-Form Health Survey (SF-36): I conceptual framework and items selection. Medical Care, 1992; 30: 473-483.PubMedCrossRefGoogle Scholar
  66. 66.
    Kazis LE, Miller DR, Clark JA et al. Improving the response choices on the veterans SF-36 health survey role functioning scales. Results from the Veterans Health Study. Journal of Ambulatory Care Management, 2004; 27: 263-280.PubMedGoogle Scholar
  67. 67.
    Spitzer RL Gibbon M, Williams JB, et al. Global Assessment of Functioning (GAF) scale in Outcomes Assessment in Clinical Practice. In: Sederer LI, Dickey B (eds) Baltimore, Williams & Wilkins, 1996.Google Scholar
  68. 68.
    Tabachnick BG, Fidell LS. Using Multivariate Statistics (5th Edition). Boston: Allyn & Bacon, 2006.Google Scholar
  69. 69.
    Barry MM, Zissi A. Quality of life as an outcome measure in evaluating mental health services: a review of the empirical evidence. Social Psychiatry & Psychiatric Epidemiology 1997; 32: 38-47.CrossRefGoogle Scholar
  70. 70.
    Ramirez A, Ekeselius L, Ramklint M. Axis V–Global Assessment of Functioning scale (GAF): further evaluation of the self-report version. European Psychiatry 2008; 23: 575-579.PubMedCrossRefGoogle Scholar
  71. 71.
    Slade M, McCrone P, Kuipers E, et al. Use of standardised outcome measures in adult mental health services: a randomized controlled trial. British Journal of Psychiatry 2006; 18: 330-336.CrossRefGoogle Scholar
  72. 72.
    Rosen AK, Chatterjee S, Glickman ME, Spiro AS, Seal P, Eisen SV. Improving Risk Adjustment of Self-Reported Mental Health Outcomes. Journal of Behavioral Health Services & Research 2010; 37: 291-306.Google Scholar
  73. 73.
    Adler DA, Bungay KM, Cynn DJ, et al. Patient-based health status assessments in an outpatient psychiatry setting. Psychiatric Services 2000;51:341-348.PubMedCrossRefGoogle Scholar
  74. 74.
    Clardy JA, Booth BM, Smith LG, et al. Implementing a statewide outcomes management system for consumers of public mental health services. Psychiatric Services 1998;49:191-195. PubMedGoogle Scholar

Copyright information

© Department of Veterans Affairs 2010

Authors and Affiliations

  • Susan V. Eisen
    • 1
  • Kathryn A. Bottonari
    • 2
  • Mark E. Glickman
    • 1
  • Avron SpiroIII
    • 3
  • Mark R. Schultz
    • 1
  • Lawrence Herz
    • 4
  • Robert Rosenheck
    • 4
  • Ethan S. Rofman
    • 5
  1. 1.Center for Health Quality, Outcomes and Economic Research (CHQOER)Edith Nourse Rogers Memorial Veterans HospitalBedfordUSA
  2. 2.Charlie Norwood VAMCAugustaUSA
  3. 3.Massachusetts Veterans Epidemiology Research and Information Center (MAVERIC)Boston University School of Public HealthBostonUSA
  4. 4.NEPEC, VA Connecticut Healthcare SystemWest HavenUSA
  5. 5.VISN 1Edith Nourse Rogers Memorial Veterans HospitalBedfordUSA

Personalised recommendations