Measuring Clinically Meaningful Change Following Mental Health Treatment

  • Susan V. Eisen
  • Gayatri Ranganathan
  • Pradipta Seal
  • Avron SpiroIII
Regular Article

Abstract

Assessment of clinically meaningful change is useful for treatment planning, monitoring progress, and evaluating treatment response. Outcome studies often assess statistically significant change, which may not be clinically meaningful. Study objectives are to: (1) evaluate responsiveness of the BASIS-24© using three methods for determining clinically meaningful change: reliable change index (RCI), effect size (ES), and standard error of measurement (SEM); and (2) determine which method provides an estimate of clinically meaningful change most concordant with other change measures. BASIS-24© assessments were obtained at two time points for 1,397 inpatients and 850 outpatients. The proportion showing clinically meaningful change using each method was compared to the proportion showing change in global mental health, retrospectively reported change, and clinician-assessed change. BASIS-24© demonstrated responsiveness at both aggregate and individual levels. Regarding clinically meaningful improvement and decline, SEM was most concordant with all three outcome measures; regarding no change, RCI was most concordant with all three measures.

Keywords

clinically meaningful change outcome assessment 

References

  1. 1.
    Newman FL, Ciarlo JA, Carpenter D. Guidelines for selecting psychological instruments for treatment planning and outcome assessment. In: Maruish M, ed. The Use of Psychological Testing for Treatment Planning and Outcomes Assessment 2nd Edition. Mahwah, NJ: Lawrence Erlbaum; 1999:153–170.Google Scholar
  2. 2.
    Guyatt G, Walter S, Norman G. Measuring change over time: assessing the usefulness of evaluative instruments. Journal of Chronic Disease. 1987;40:171–178.CrossRefGoogle Scholar
  3. 3.
    Liang MH. Longitudinal construct validity: Establishment of clinical meaning in patient evaluation instruments. Medical Care. 2000;38(Suppl II):S84–S90.Google Scholar
  4. 4.
    Hays RD, Anderson R, Revicki D. Psychometric considerations in evaluating health-related quality of life measures. Quality of Life Research. 1993;2:441–449.PubMedCrossRefGoogle Scholar
  5. 5.
    Wyrwich KW, Wolinsky FD. Identifying meaningful intra-individual change standards for health-related quality of life measures. Journal of Evaluation in Clinical Practice. 2000;6:39–49.PubMedCrossRefGoogle Scholar
  6. 6.
    McHorney CA, Tarlow AR. Individual patient monitoring in clinical practice: are available health status surveys adequate? Quality of Life Research. 1995;4(4):293–307.PubMedCrossRefGoogle Scholar
  7. 7.
    Hays RD, Brodsky M, Johnston MF, et al. Evaluating the statistical significance of health-related quality-of-life change in individual patients. Evaluation & the Health Professions. 2005;28:160–171.CrossRefGoogle Scholar
  8. 8.
    Wyrwich KW, Bullinger M, Aaronson N, et al. Estimating clinically significant differences in quality of life outcomes. Quality of Life Research. 2005;14:285–295.PubMedCrossRefGoogle Scholar
  9. 9.
    Glossary. Health outcomes methodology. Medical Care. 2000;38(9 Suppl. II,II-7-II-13):2–10).Google Scholar
  10. 10.
    Atkins DC, Bedics JD, McGlinchey JB, et al. Assessing clinical significance: Does it matter which method we use? Journal of Consulting & Clinical Psychology. 2005;73:982–989.CrossRefGoogle Scholar
  11. 11.
    Eisen SV, Normand SLT, Belanger AJ, et al. The revised Behavior and Symptom Identification Scale (BASIS-24©): Reliability and validity. Medical Care. 2004;42:1230–1241.PubMedCrossRefGoogle Scholar
  12. 12.
    Guyatt GH, Norman GR, Juniper EF, et al. A critical look at transition ratings. Journal of Clinical Epidemiology. 2002;55:900–908.PubMedCrossRefGoogle Scholar
  13. 13.
    Spitzer RL, Gibbon M, Endicott J. Global Assessment Scale (GAS), Global Assessment of Functioning (GAF) Scale, Social and Occupational Functioning Assessment Scale (SOFAS). In: Handbook of Psychiatric Measures. Washington DC: American Psychiatric Association; 2000:96–100.Google Scholar
  14. 14.
    Kazdin AE. Assessing the clinical or applied importance of behavior change through social validation. Behavior Modification. 1977;1:427–452.CrossRefGoogle Scholar
  15. 15.
    Streiner DL, Norman GR.Health Measurement Scales. A Practical Guide to Their Development and Use (3rd ed.). New York: Oxford University Press 2003Google Scholar
  16. 16.
    Jaeschke R, Singer J, Guyatt GH. Measurement of health status. Ascertaining the minimally important difference. Controlled Clinical Trials. 1991;10:407–415.CrossRefGoogle Scholar
  17. 17.
    Hurst H, Bolton J. Assessing the clinical significance of change scores recorded on subjective outcome measures. Journal of Manipulative Physiological Therapeutics. 2004;27:26–35.CrossRefGoogle Scholar
  18. 18.
    Ferguson RJ, Robinson AB, Splaine M. Use of the reliable change index to evaluate clinical significance in SF-36 outcomes. Quality of Life Research. 2002;11:509–516.PubMedCrossRefGoogle Scholar
  19. 19.
    Clancy C, Eisenberg J. Outcomes research care: measuring the end results of health care. Science. 1998;282:245–246.PubMedCrossRefGoogle Scholar
  20. 20.
    Cohen J.Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum Associates; 1988.Google Scholar
  21. 21.
    Testa M. Interpreting quality of life clinical trial data for use in the clinical practice of antihypertensive therapy. Journal of Hypertension Supplement. 1987;5:S9–S13.Google Scholar
  22. 22.
    Kazis Le, Anderson JJ, Meenan RF. Effect sizes for interpreting changes in health status. Medical Care. 1989;27(Suppl 3):S178–S189.PubMedCrossRefGoogle Scholar
  23. 23.
    Norman GR, Sloan JA, Wyrwich KW. Interpretation of changes in health-related quality of life. The remarkable universality of half a standard deviation. Medical Care. 2003;5:582–592.CrossRefGoogle Scholar
  24. 24.
    Jacobson NS, Truax P. Clinical significance: a statistical approach to defining meaningful change in psychotherapy research. Journal of Consulting and Clinical Psychology. 1991;59:12–19.PubMedCrossRefGoogle Scholar
  25. 25.
    Jacobson NS, Follette WC, Revenstorf. Psychotherapy outcome research. Methods for reporting variability and evaluating clinical significance. Behavior Therapy. 1984;15:336–352.CrossRefGoogle Scholar
  26. 26.
    Jacobson NS, Roberts LJ, Berns SB, McGlinchey JB. Methods for defining and determining the clinical significance of treatment effects: description, application and alternatives. Journal of Consulting and Clinical Psychology. 1999;67:300–307.PubMedCrossRefGoogle Scholar
  27. 27.
    Speer D. Methodological developments. Clinically significant change: Jacobson and Truax (1991) revisited. Journal of Consulting and Clinical Psychology. 1992;69(3):402–408.CrossRefGoogle Scholar
  28. 28.
    Hageman WJM, Arrindell WA. A further refinement of the reliable change (RC) index by improving the pre–post difference score: introducing RCID. Behaviour Research and Therapy. 1993;31:693–700.PubMedCrossRefGoogle Scholar
  29. 29.
    Anastasi A, Urbina S.Psychological Testing (7th ed.). Upper Saddle River, NJ: Prentice-Hall; 1997.Google Scholar
  30. 30.
    Wyrwich KW, Tierney WM, Wolinsky FD. Further evidence supporting an SEM-based criterion for identifying meaningful intra-individual changes in health-related quality of life. Journal of Clinical Epidemiology. 1999;52:861–873.PubMedCrossRefGoogle Scholar
  31. 31.
    Wyrwich KW, Nienaber NA, Tierney WM, Wolinsky FD. Linking clinical relevance and statistical significance in evaluating intra-individual changes in health-related quality of life. Medical Care. 1999;37:469–478.PubMedCrossRefGoogle Scholar
  32. 32.
    Eisen SV, Dill DL, Grob MC. Reliability and validity of a brief patient-report instrument for psychiatric outcome evaluation. Hospital & Community Psychiatry. 1994;45:242–247.Google Scholar
  33. 33.
    American Psychiatric Association.Diagnostic and Statistical Manual of Mental Disorders. 4th ed. (DSM-IV). Washington D.C.: American Psychiatric Association; 1994.Google Scholar
  34. 34.
    Burlingame GM, Dunn TW, Chen S, et al. Selection of outcome assessment instruments for inpatients with severe and persistent mental illness. Psychiatric Services. 2005;56:444–451.PubMedCrossRefGoogle Scholar
  35. 35.
    Eisen SV, Gerena M, Rangathan G, et al. Reliability and validity of the BASIS-24© Mental Health Survey for Whites, African-Americans and Latinos. Journal of Behavioral Health Services & Research. 2006;33:304–323.CrossRefGoogle Scholar
  36. 36.
    Jerrell JM. Behavior and Symptom Identification Scale 32. Sensitivity to change over time. Journal of Behavioral Health Services & Research. 2005;32:341–346.CrossRefGoogle Scholar
  37. 37.
    Cohen J. A coefficient of agreement for nominal scales. Educational and Psychological Measurement. 1960;20:37–46.CrossRefGoogle Scholar
  38. 38.
    Lambert MJ, Lambert JM. Use of psychological tests for assessing treatment outcome. In: Maruish ME, ed. The Use of Psychological Testing for Treatment Planning and Outcomes Assessment (2nd ed.). Mahwah, NJ: Lawrence Erlbaum; 1999:115–152.Google Scholar
  39. 39.
    Jacobson NS, Roberts LJ, Berns SB, et al. Methods for defining and determining the clinical significance of treatment effects. Journal of Consulting and Clinical Psychology. 1999;67:300–307.PubMedCrossRefGoogle Scholar
  40. 40.
    Ogles BM, Lambert MJ, Sawyer JD. Clinical significance of the National Institute of Mental Health treatment of depression collaborative research program data. Journal of Consulting and Clinical Psychology. 1995;63:321–326.PubMedCrossRefGoogle Scholar
  41. 41.
    Guyatt H, Osoba D, Wu A, et al. Methods to explain the clinical significance of health status measures. Mayo Clinic Proceedings. 2002;77(4):373–383.Google Scholar
  42. 42.
    Jones SH, Thornicroft G, Coffey M, et al. A brief mental health outcome scale-reliability and validity of the Global Assessment of Functioning (GAF). British Journal of Psychiatry. 1995;166:654–650.PubMedCrossRefGoogle Scholar
  43. 43.
    Startup M, Jackson MC, Bendix S. The concurrent validity of the Global Assessment of Functioning (GAF). British Journal of Clinical Psychology. 2002;41:417–422.PubMedCrossRefGoogle Scholar
  44. 44.
    Greenberg GA, Rosenheck RA. Using the GAS as a national mental health outcome measure in the Department of Veterans Affairs. Psychiatric Services. 2005;56:420–426.PubMedCrossRefGoogle Scholar
  45. 45.
    Wilkinson G, Hesdon B, Wild D, et al. Self-report quality of life measure for people with schizophrenia: the SQLS. British Journal of Psychiatry. 2000;177:42–46.PubMedCrossRefGoogle Scholar
  46. 46.
    Srebnick D, Hendryx M, Stevenson J, et al. Development of outcome indicators for monitoring the quality of public mental health care. Psychiatric Services. 1997;48:903–909.Google Scholar
  47. 47.
    Norman GR, Sridhar FG, Guyatt GH. Relation of distribution- and anchor-based approaches in interpretation of changes in health-related quality of life. Medical Care. 2001;39:1039–1047.PubMedCrossRefGoogle Scholar
  48. 48.
    Nunnally JC.Psychometric Theory. New York: McGraw-Hill; 1994.Google Scholar
  49. 49.
    Gilbody SM, House AO, Sheldon T. Routine administration of health related quality of life (HRQoL) and needs assessment instruments to improve psychological outcome—a systematic review. Psychological Medicine. 2002;32:1345–1356.PubMedCrossRefGoogle Scholar
  50. 50.
    Garland AF, Kruse M, Aarons GA. Clinicians and outcome measurement: What’s the use? Journal of Behavioral Health Services & Research. 2003;30:393–405.CrossRefGoogle Scholar

Copyright information

© National Council for Community Behavioral Healthcare 2007

Authors and Affiliations

  • Susan V. Eisen
    • 1
    • 2
  • Gayatri Ranganathan
    • 3
  • Pradipta Seal
    • 1
  • Avron SpiroIII
    • 4
    • 5
  1. 1.Center for Health Quality, Outcomes and Economic Research (CHQOER)Edith Nourse Rogers Memorial Veterans HospitalBedfordUSA
  2. 2.Department of Health Policy and ManagementBoston University School of Public HealthBostonUSA
  3. 3.MetaWorks Inc.MedfordUSA
  4. 4.Massachusetts Veterans Epidemiology Research and Information Center (MAVERIC)VA Boston Healthcare SystemBostonUSA
  5. 5.Department of EpidemiologyBoston University School of Public HealthBostonUSA

Personalised recommendations