Skip to main content

Advertisement

SpringerLink
Log in

Measuring Clinically Meaningful Change Following Mental Health Treatment

  • Regular Article
  • Published:
The Journal of Behavioral Health Services & Research Aims and scope Submit manuscript

Abstract

Assessment of clinically meaningful change is useful for treatment planning, monitoring progress, and evaluating treatment response. Outcome studies often assess statistically significant change, which may not be clinically meaningful. Study objectives are to: (1) evaluate responsiveness of the BASIS-24© using three methods for determining clinically meaningful change: reliable change index (RCI), effect size (ES), and standard error of measurement (SEM); and (2) determine which method provides an estimate of clinically meaningful change most concordant with other change measures. BASIS-24© assessments were obtained at two time points for 1,397 inpatients and 850 outpatients. The proportion showing clinically meaningful change using each method was compared to the proportion showing change in global mental health, retrospectively reported change, and clinician-assessed change. BASIS-24© demonstrated responsiveness at both aggregate and individual levels. Regarding clinically meaningful improvement and decline, SEM was most concordant with all three outcome measures; regarding no change, RCI was most concordant with all three measures.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Similar content being viewed by others

References

  1. Newman FL, Ciarlo JA, Carpenter D. Guidelines for selecting psychological instruments for treatment planning and outcome assessment. In: Maruish M, ed. The Use of Psychological Testing for Treatment Planning and Outcomes Assessment 2nd Edition. Mahwah, NJ: Lawrence Erlbaum; 1999:153–170.

    Google Scholar 

  2. Guyatt G, Walter S, Norman G. Measuring change over time: assessing the usefulness of evaluative instruments. Journal of Chronic Disease. 1987;40:171–178.

    Article  CAS  Google Scholar 

  3. Liang MH. Longitudinal construct validity: Establishment of clinical meaning in patient evaluation instruments. Medical Care. 2000;38(Suppl II):S84–S90.

    Google Scholar 

  4. Hays RD, Anderson R, Revicki D. Psychometric considerations in evaluating health-related quality of life measures. Quality of Life Research. 1993;2:441–449.

    Article  PubMed  CAS  Google Scholar 

  5. Wyrwich KW, Wolinsky FD. Identifying meaningful intra-individual change standards for health-related quality of life measures. Journal of Evaluation in Clinical Practice. 2000;6:39–49.

    Article  PubMed  CAS  Google Scholar 

  6. McHorney CA, Tarlow AR. Individual patient monitoring in clinical practice: are available health status surveys adequate? Quality of Life Research. 1995;4(4):293–307.

    Article  PubMed  CAS  Google Scholar 

  7. Hays RD, Brodsky M, Johnston MF, et al. Evaluating the statistical significance of health-related quality-of-life change in individual patients. Evaluation & the Health Professions. 2005;28:160–171.

    Article  Google Scholar 

  8. Wyrwich KW, Bullinger M, Aaronson N, et al. Estimating clinically significant differences in quality of life outcomes. Quality of Life Research. 2005;14:285–295.

    Article  PubMed  Google Scholar 

  9. Glossary. Health outcomes methodology. Medical Care. 2000;38(9 Suppl. II,II-7-II-13):2–10).

  10. Atkins DC, Bedics JD, McGlinchey JB, et al. Assessing clinical significance: Does it matter which method we use? Journal of Consulting & Clinical Psychology. 2005;73:982–989.

    Article  Google Scholar 

  11. Eisen SV, Normand SLT, Belanger AJ, et al. The revised Behavior and Symptom Identification Scale (BASIS-24©): Reliability and validity. Medical Care. 2004;42:1230–1241.

    Article  PubMed  Google Scholar 

  12. Guyatt GH, Norman GR, Juniper EF, et al. A critical look at transition ratings. Journal of Clinical Epidemiology. 2002;55:900–908.

    Article  PubMed  Google Scholar 

  13. Spitzer RL, Gibbon M, Endicott J. Global Assessment Scale (GAS), Global Assessment of Functioning (GAF) Scale, Social and Occupational Functioning Assessment Scale (SOFAS). In: Handbook of Psychiatric Measures. Washington DC: American Psychiatric Association; 2000:96–100.

  14. Kazdin AE. Assessing the clinical or applied importance of behavior change through social validation. Behavior Modification. 1977;1:427–452.

    Article  Google Scholar 

  15. Streiner DL, Norman GR.Health Measurement Scales. A Practical Guide to Their Development and Use (3rd ed.). New York: Oxford University Press 2003

    Google Scholar 

  16. Jaeschke R, Singer J, Guyatt GH. Measurement of health status. Ascertaining the minimally important difference. Controlled Clinical Trials. 1991;10:407–415.

    Article  Google Scholar 

  17. Hurst H, Bolton J. Assessing the clinical significance of change scores recorded on subjective outcome measures. Journal of Manipulative Physiological Therapeutics. 2004;27:26–35.

    Article  Google Scholar 

  18. Ferguson RJ, Robinson AB, Splaine M. Use of the reliable change index to evaluate clinical significance in SF-36 outcomes. Quality of Life Research. 2002;11:509–516.

    Article  PubMed  Google Scholar 

  19. Clancy C, Eisenberg J. Outcomes research care: measuring the end results of health care. Science. 1998;282:245–246.

    Article  PubMed  CAS  Google Scholar 

  20. Cohen J.Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum Associates; 1988.

    Google Scholar 

  21. Testa M. Interpreting quality of life clinical trial data for use in the clinical practice of antihypertensive therapy. Journal of Hypertension Supplement. 1987;5:S9–S13.

    CAS  Google Scholar 

  22. Kazis Le, Anderson JJ, Meenan RF. Effect sizes for interpreting changes in health status. Medical Care. 1989;27(Suppl 3):S178–S189.

    Article  PubMed  CAS  Google Scholar 

  23. Norman GR, Sloan JA, Wyrwich KW. Interpretation of changes in health-related quality of life. The remarkable universality of half a standard deviation. Medical Care. 2003;5:582–592.

    Article  Google Scholar 

  24. Jacobson NS, Truax P. Clinical significance: a statistical approach to defining meaningful change in psychotherapy research. Journal of Consulting and Clinical Psychology. 1991;59:12–19.

    Article  PubMed  CAS  Google Scholar 

  25. Jacobson NS, Follette WC, Revenstorf. Psychotherapy outcome research. Methods for reporting variability and evaluating clinical significance. Behavior Therapy. 1984;15:336–352.

    Article  Google Scholar 

  26. Jacobson NS, Roberts LJ, Berns SB, McGlinchey JB. Methods for defining and determining the clinical significance of treatment effects: description, application and alternatives. Journal of Consulting and Clinical Psychology. 1999;67:300–307.

    Article  PubMed  CAS  Google Scholar 

  27. Speer D. Methodological developments. Clinically significant change: Jacobson and Truax (1991) revisited. Journal of Consulting and Clinical Psychology. 1992;69(3):402–408.

    Article  Google Scholar 

  28. Hageman WJM, Arrindell WA. A further refinement of the reliable change (RC) index by improving the pre–post difference score: introducing RCID. Behaviour Research and Therapy. 1993;31:693–700.

    Article  PubMed  CAS  Google Scholar 

  29. Anastasi A, Urbina S.Psychological Testing (7th ed.). Upper Saddle River, NJ: Prentice-Hall; 1997.

    Google Scholar 

  30. Wyrwich KW, Tierney WM, Wolinsky FD. Further evidence supporting an SEM-based criterion for identifying meaningful intra-individual changes in health-related quality of life. Journal of Clinical Epidemiology. 1999;52:861–873.

    Article  PubMed  CAS  Google Scholar 

  31. Wyrwich KW, Nienaber NA, Tierney WM, Wolinsky FD. Linking clinical relevance and statistical significance in evaluating intra-individual changes in health-related quality of life. Medical Care. 1999;37:469–478.

    Article  PubMed  CAS  Google Scholar 

  32. Eisen SV, Dill DL, Grob MC. Reliability and validity of a brief patient-report instrument for psychiatric outcome evaluation. Hospital & Community Psychiatry. 1994;45:242–247.

    CAS  Google Scholar 

  33. American Psychiatric Association.Diagnostic and Statistical Manual of Mental Disorders. 4th ed. (DSM-IV). Washington D.C.: American Psychiatric Association; 1994.

    Google Scholar 

  34. Burlingame GM, Dunn TW, Chen S, et al. Selection of outcome assessment instruments for inpatients with severe and persistent mental illness. Psychiatric Services. 2005;56:444–451.

    Article  PubMed  Google Scholar 

  35. Eisen SV, Gerena M, Rangathan G, et al. Reliability and validity of the BASIS-24© Mental Health Survey for Whites, African-Americans and Latinos. Journal of Behavioral Health Services & Research. 2006;33:304–323.

    Article  Google Scholar 

  36. Jerrell JM. Behavior and Symptom Identification Scale 32. Sensitivity to change over time. Journal of Behavioral Health Services & Research. 2005;32:341–346.

    Article  Google Scholar 

  37. Cohen J. A coefficient of agreement for nominal scales. Educational and Psychological Measurement. 1960;20:37–46.

    Article  Google Scholar 

  38. Lambert MJ, Lambert JM. Use of psychological tests for assessing treatment outcome. In: Maruish ME, ed. The Use of Psychological Testing for Treatment Planning and Outcomes Assessment (2nd ed.). Mahwah, NJ: Lawrence Erlbaum; 1999:115–152.

    Google Scholar 

  39. Jacobson NS, Roberts LJ, Berns SB, et al. Methods for defining and determining the clinical significance of treatment effects. Journal of Consulting and Clinical Psychology. 1999;67:300–307.

    Article  PubMed  CAS  Google Scholar 

  40. Ogles BM, Lambert MJ, Sawyer JD. Clinical significance of the National Institute of Mental Health treatment of depression collaborative research program data. Journal of Consulting and Clinical Psychology. 1995;63:321–326.

    Article  PubMed  CAS  Google Scholar 

  41. Guyatt H, Osoba D, Wu A, et al. Methods to explain the clinical significance of health status measures. Mayo Clinic Proceedings. 2002;77(4):373–383.

    Google Scholar 

  42. Jones SH, Thornicroft G, Coffey M, et al. A brief mental health outcome scale-reliability and validity of the Global Assessment of Functioning (GAF). British Journal of Psychiatry. 1995;166:654–650.

    Article  PubMed  CAS  Google Scholar 

  43. Startup M, Jackson MC, Bendix S. The concurrent validity of the Global Assessment of Functioning (GAF). British Journal of Clinical Psychology. 2002;41:417–422.

    Article  PubMed  Google Scholar 

  44. Greenberg GA, Rosenheck RA. Using the GAS as a national mental health outcome measure in the Department of Veterans Affairs. Psychiatric Services. 2005;56:420–426.

    Article  PubMed  Google Scholar 

  45. Wilkinson G, Hesdon B, Wild D, et al. Self-report quality of life measure for people with schizophrenia: the SQLS. British Journal of Psychiatry. 2000;177:42–46.

    Article  PubMed  CAS  Google Scholar 

  46. Srebnick D, Hendryx M, Stevenson J, et al. Development of outcome indicators for monitoring the quality of public mental health care. Psychiatric Services. 1997;48:903–909.

    Google Scholar 

  47. Norman GR, Sridhar FG, Guyatt GH. Relation of distribution- and anchor-based approaches in interpretation of changes in health-related quality of life. Medical Care. 2001;39:1039–1047.

    Article  PubMed  CAS  Google Scholar 

  48. Nunnally JC.Psychometric Theory. New York: McGraw-Hill; 1994.

    Google Scholar 

  49. Gilbody SM, House AO, Sheldon T. Routine administration of health related quality of life (HRQoL) and needs assessment instruments to improve psychological outcome—a systematic review. Psychological Medicine. 2002;32:1345–1356.

    Article  PubMed  CAS  Google Scholar 

  50. Garland AF, Kruse M, Aarons GA. Clinicians and outcome measurement: What’s the use? Journal of Behavioral Health Services & Research. 2003;30:393–405.

    Article  Google Scholar 

Download references

Acknowledgment

The authors thank Colleen McHorney, Ph.D., for suggesting use of the SEM to assess clinically meaningful change, and Joel Reisman for his helpful comments on an earlier version of this manuscript.

This research was supported by grant R01 MH58240 from the National Institute of Mental Health and by the Veterans Affairs Health Services Research & Development program.

Author information

Authors and Affiliations

  1. Center for Health Quality, Outcomes and Economic Research (CHQOER), Edith Nourse Rogers Memorial Veterans Hospital, 200 Springs Road (152), Bedford, MA, 01730, USA

    Susan V. Eisen PhD (Professor, Health Research Scientist) & Pradipta Seal MS (Data Analyst)

  2. Department of Health Policy and Management, Boston University School of Public Health, Boston, MA, USA

    Susan V. Eisen PhD (Professor, Health Research Scientist)

  3. MetaWorks Inc., 10 President’s Landing, Medford, MA, 02155, USA

    Gayatri Ranganathan MS (Biostatistician)

  4. Massachusetts Veterans Epidemiology Research and Information Center (MAVERIC), VA Boston Healthcare System, 150 South Huntington Ave, Boston, MA, 02130, USA

    Avron Spiro III PhD (Health Research Scientist, Associate Professor)

  5. Department of Epidemiology, Boston University School of Public Health, Boston, MA, USA

    Avron Spiro III PhD (Health Research Scientist, Associate Professor)

Authors
  1. Susan V. Eisen PhD
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Gayatri Ranganathan MS
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Pradipta Seal MS
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Avron Spiro III PhD
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Susan V. Eisen PhD.

Additional information

This work was done at the Center for Health Quality, Outcomes and Economic Research, a Health Services Research and Development Center of Excellence of the Veterans Administration. The Center is located at the Edith Nourse Rogers Memorial Veterans Hospital, Bedford, MA.

The views expressed in this article are those of the authors and do not necessarily represent the views of the Department of Veterans Affairs.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Eisen, S.V., Ranganathan, G., Seal, P. et al. Measuring Clinically Meaningful Change Following Mental Health Treatment. J Behav Health Serv Res 34, 272–289 (2007). https://doi.org/10.1007/s11414-007-9066-2

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11414-007-9066-2

Keywords

Search

Navigation