Skip to main content
SpringerLink
Log in

Measuring mental health outcomes with pre-post designs

  • Special Section
  • Published:
The Journal of Behavioral Health Services & Research Aims and scope Submit manuscript

Abstract

The pre-post design has been the workhorse of outcome evaluations for many years. Using data from a study of 984 treated children (ages 5 to 17 years), this article argues that there are two structural problems with the pre-post evaluation of outcome: (1) excessively large intervals of uncertainty for individual outcomes and (2) paradoxical inconsistencies in the evaluation of groups. These problems can be solved by designs with three or more repeated measures analyzed with longitudinal multilevel analytic models.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Berwick DM, Wald DL. Hospital leader's opinions of the HCFA mortality data.Journal of the American Medical Association. 1990;263:247–249.

    Google Scholar 

  2. Brinkley J. US releasing lists of hospitals with abnormal mortality rates.New York Times, March 12, 1986, 1.

  3. Blumberg MS. Risk adjusting health care outcomes: a methodologic review.Medical Care Review. 1986;43(2):351–393.

    Google Scholar 

  4. Campbell DT, Stanley JC.Experimental and Quasi-Experimental Designs for Research. Boston: Houghton Mifflin Company; 1963.

    Google Scholar 

  5. Bickman L, Guthrie P, Foster EM, et al.Evaluating Managed Mental Health Services: The Fort Bragg Experiment. New York: Plenum; 1995.

    Google Scholar 

  6. Bickman L. A continuum of care: more is not always better.American Psychologist. 1996;51(7):689–701.

    Google Scholar 

  7. Hamner KM, Lambert EW, Bickman L. Children's mental health in a continuum of care: clinical outcomes at 18 months for the Fort Bragg demonstration.Journal of Mental Health Administration. 1997;24(4):465–471.

    Google Scholar 

  8. Lambert EW, Guthrie PR. Clinical outcomes of a children's mental health managed care demonstration.Journal of Mental Health Administration. 1996;23(1):51–68.

    Google Scholar 

  9. Newman FL, Tejeda MJ. The need for research that is designed to support decisions in the delivery of mental health services.American Psychologist. 1996;51(10):1040–1049.

    Google Scholar 

  10. Krause MS, Howard KI, Lutz W. Exploring individual change.Journal of Consulting and Clinical Psychology. 1998;66(5):838–845.

    Google Scholar 

  11. Kleinbaum DG, Kupper LL, Muller KE, Nizam A.Applied Regression Analysis and Other Multivariable Methods. Pacific Grove, CA: Brooks/Cole; 1998.

    Google Scholar 

  12. Nunnally JC, Bernstein IH.Psychometric Theory. New York: McGraw-Hill; 1994.

    Google Scholar 

  13. Achenbach TM. Child behavior checklist and related instruments. In: Maruish ME, ed.The Use of Psychological Testing for Treatment Planning and Outcome Assessment. Hillsdale, NJ: Lawrence Erlbaum Associates, Inc; 1994.

    Google Scholar 

  14. Rogosa DR, Willett JB. Understanding correlates of change by modeling individual differences in growth.Psychometrika. 1985;50(2):203–228.

    Google Scholar 

  15. Rogosa D, Brandt D, Zimowski M. A growth curve approach to the measurement of change.Psychological Bulletin. 1982;92(3):726–748.

    Google Scholar 

  16. Lutz W, Martinovich Z, Howard KI. Patient profiling: an application of random coefficient regression models to depicting the response of a patient to outpatient psychotherapy.Journal of Consulting and Clinical Psychology. 1999;67(4):571–577.

    Google Scholar 

  17. Lambert EW, Bickman L. Individual and group outcomes: longitudinal analytic models solving real world problems. Presented at the Mental Health Outcomes Risk-Adjustment: A National Research Development Conference; August 1999; Spokane, WA.

  18. Lord FM. Elementary models for measuring change. In: Harris CW, ed.Problem in Measuring Change. Milwaukee, WI: University of Wisconsin Press; 1963:21–38.

    Google Scholar 

  19. Cronbach LJ, Furby L. How we should measure “change”: or should we?Psychological Bulletin. 1970;74(1):68–80.

    Google Scholar 

  20. Rogosa D. Myths about longitudinal research. In: Schaie KW, Campbell RT, Meredith W, Rawlings RC, eds.Methodological Issues in Aging Research. New York: Springer; 1988:171–209.

    Google Scholar 

  21. Newman FL. Selection of design and statistical procedures for progress and outcome assessment. In: Maruish ME, ed.The Use of Psychological Testing for Treatment Planning and Outcome Assessment. Hillsdale, NJ: Lawrence Erlbaum Associates, Inc; 1994:111–134.

    Google Scholar 

  22. Locascia JJ, Cordray DS. A re-analysis of “Lord's paradox.”Educational and Psychological Measurement. 1983;43(1):115–126.

    Google Scholar 

  23. Gottman JM, Rushe RH. The analysis of change: issues, fallacies, and new ideas.Journal of Consulting and Clinical Psychology. 1993;61(6):907–910.

    Google Scholar 

  24. Kenny DA. Glossary of experimental and quasi-experimental terms (1998). http://nw3.nai.net/~dakenny/glrtm.htm. Accessed on May 15, 2001.

  25. Jin P. Toward a reconceptualization of the law of initial value.Psychological Bulletin. 1992;111(1):176–184.

    Google Scholar 

  26. Diggle P, Liang K-Y, Zeger SL.Analysis of Longitudinal Data. New York: Clarendon Press; 1994.

    Google Scholar 

  27. Kraemer HC, Thiemann S. A strategy to use soft data effectively in randomized controlled clinical trials.Journal of Consulting and Clinical Psychology. 1989;57(1):148–154.

    Google Scholar 

  28. Lambert EW, Bickman L. Longitudinal analytic models solving real world problems. Presented at the 12th Annual Research Conference Proceedings: A System of Care for Children's Mental Health: Expanding the Research Base; February 1999; Clearwater Beach, FL.

  29. Laird NM, Ware JH. Random-effects models for longitudinal data.Biometrics. 1982;38(4):963–974.

    Google Scholar 

  30. Hedeker DR, Gibbons RD, Davis JM. Random regression models for multicenter clinical trials data.Psychopharmacology Bulletin. 1991;27(1):73–77.

    Google Scholar 

  31. Gibbons RD, Hedeker D, Elkin I, et al. Some conceptual and statistical issues in analysis of longitudinal psychiatric data: application to the NIMH treatment of depression collaborative research program dataset.Archives of General Psychiatry. 1993;50(9):739–750.

    Google Scholar 

  32. Bryk AS, Raudenbush SW.Hierarchical Linear Models: Applications and Data Analysis Methods. Newbury Park, CA: Sage; 1992.

    Google Scholar 

  33. Raudenbush SW, Chan WS. Application of a hierarchical linear model to the study of adolescent deviance in an overlapping cohort design.Journal of Consulting and Clinical Psychology. 1993;61(6):941–951.

    Google Scholar 

  34. Verbeke G, Molenberghs G.Linear Mixed Models in Practice: An SAS-Oriented Approach. New York: Springer; 1997.

    Google Scholar 

  35. Littell RC, Milliken GA, Stroup WW, Wolfinger RD.SAS System for Mixed Models. Cary, NC: SAS Institute; 1996.

    Google Scholar 

  36. Nich C, Carroll K. Now you see it, now you don't: a comparison of traditional versus random-effects regression models in the analysis of longitudinal follow-up data from a clinical trial.Journal of Consulting and Clinical Psychology. 1997;65(2):252–261.

    Google Scholar 

  37. Bickman L, Lambert EW, Andrade AR, Penaloza R. The Fort Bragg continuum of care for children and adolescents: mental health outcomes over five years.Journal of Consulting and Clinical Psychology. 2000;68(4):710–716.

    Google Scholar 

  38. Lambert EW, Wahler RG, Andrade AR, Bickman L. Looking for the disorder in conduct disorder.Journal of Abnormal Psychology. 2001; 110(1):110–123.

    Google Scholar 

  39. Hedeker D, Gibbons RD. MIXOR, MIXREG, MIXNO, and MIXPREG programs (1999). TIGGER.UIC.EDU/~HEDEKER/MIXREG. HTML. Accessed on May 15, 2001.

  40. Bryk A, Raudenbush S, Congdon R.HLM Hierarchical Linear and Nonlinear Modeling with the HLM/2L and HLM/3L Programs. Chicago: Scientific Software International, Inc; 1996.

    Google Scholar 

  41. Hedeker D, Gibbons RD. Application of random-effects pattern-mixture models for missing data in longitudinal studies.Psychological Methods. 1997;2(1):64–78.

    Google Scholar 

  42. Hedeker D, Mermelstein RJ. Section III. Methods for analyzing longitudinal data on relapse: application of random-effects regression models in relapse research.Addiction. 1996;91(suppl):S211-S229.

    Google Scholar 

  43. Khattree R, Naik DN.Applied Multivariate Statistics with SAS Software. Carey, NC: SAS Institute; 1995.

    Google Scholar 

  44. Cohen J. A power primer.Psychological Bulletin. 1992;112(1):155–159.

    Google Scholar 

  45. Cohen J.Statistical Power Analysis for the Behavioral Sciences. Hillsdale, NJ: Lawrence Erlbaum Associates; 1988.

    Google Scholar 

  46. Rogosa D, Saner H. Longitudinal data analysis examples with random coefficient models.Journal of Educational and Behavioral Statistics. 1995;20(2):149–170.

    Google Scholar 

  47. Sperry L, Brill PL, Howard KI, Grissom GR.Treatment Outcomes in Psychotherapy and Psychiatric Interventions. New York: Brunner/Mazel; 1996.

    Google Scholar 

  48. SAS Institute I.SAS/STAT(R) Software: Changes and Enhancements through Release 6.12. Cary, NC: SAS Institute, Inc; 1997.

    Google Scholar 

  49. Iezzoni LI, Dalay J. A description and clinical assessment of the computerized severity index.Quality Review Bulletin. February 1992:44–52.

  50. Iezzoni LI, Shwartz M, Ash AS, et al. Risk adjustment methods can affect perceptions of outcomes.Risk and Outcomes. 1994;9(2):43–48.

    Google Scholar 

  51. Romano PS. Can administrative data be used to compare the quality of health care?Medical Care Review. 1993;50(4):451–477.

    Google Scholar 

  52. Wainer H, Thissen D, Mislevy RJ.Computer Adaptive Testing. Hillsdale, NJ: Lawrence Erlbaum Associates; 2000.

    Google Scholar 

  53. Bickman L, Noser K, Summerfelt WT. Long-term effects of a system of care on children and adolescents.Journal of Behavioral Health Services & Research. 1999; 26(2): 185–202.

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Center for Mental Health Policy, Vanderbilt University, 1207 18th Avenue South, 37212, Nashville, TN

    E. Warren Lambert PhD (Senior Research Associate), Ann Doucette PhD (Senior Research Associate) & Leonard Bickman PhD (Director)

Authors
  1. E. Warren Lambert PhD
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ann Doucette PhD
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Leonard Bickman PhD
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to E. Warren Lambert PhD.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Lambert, E.W., Doucette, A. & Bickman, L. Measuring mental health outcomes with pre-post designs. The Journal of Behavioral Health Services & Research 28, 273–286 (2001). https://doi.org/10.1007/BF02287244

Download citation

  • Issue Date:

  • DOI: https://doi.org/10.1007/BF02287244

Keywords

Search

Navigation