Advertisement

European Child & Adolescent Psychiatry

, Volume 24, Issue 5, pp 553–563 | Cite as

Denoting treatment outcome in child and adolescent psychiatry: a comparison of continuous and categorical outcomes

  • Edwin de BeursEmail author
  • Marko Barendregt
  • Bente Rogmans
  • Sylvana Robbers
  • Marieke van Geffen
  • Marleen van Aggelen-Gerrits
  • Huub Houben
Original Contribution

Abstract

Various approaches have been proposed to denote treatment outcome, such as the effect size of the pre-to-posttest change, percentage improvement, statistically reliable change, and clinical significant change. The aim of the study is to compare these approaches and evaluate their aptitude to differentiate among child and adolescent mental healthcare providers regarding their treatment outcome. Comparing outcomes according to continuous and categorical outcome indicators using real-life data of seven mental healthcare providers, three using the Child Behavior Checklist and four using the Strengths and Difficulties Questionnaire as primary outcome measure. Within each dataset consistent differences were found between providers and the various methods led to comparable rankings of providers. Statistical considerations designate continuous outcomes as the optimal choice. Change scores have more statistical power and allow for a ranking of providers at first glance. Expressing providers’ performance in proportions of recovered, changed, unchanged, or deteriorated patients has supplementary value, as it denotes outcome in a manner more easily interpreted and appreciated by clinicians, managerial staff, and, last but not least, by patients or their parents.

Keywords

Treatment outcome research Effect size (ES) Reliable change index (RCI) Percentage improvement (PI) Benchmarking 

Notes

Acknowledgments

The authors are grateful to the following mental health providers for allowing us to use their outcome data: De Viersprong, Mentaal Beter, Mutsaersstichting, OCNR, Praktijk Buitenpost, Yorneo, and Yulius.

Conflict of interest

No conflicts declared.

References

  1. 1.
    Achenbach TM (1991) Manual for the child behavior checklist 4–18 and 1991 profiles. Department of Psychiatry, University of Vermont, BurlingtonGoogle Scholar
  2. 2.
    American Psychiatric Association (1994) Diagnostic and statistical manual of mental disorders IV. Author, Washington, DCGoogle Scholar
  3. 3.
    Bauer S, Lambert MJ, Nielsen SL (2004) Clinical significance methods: a comparison of statistical techniques. J Pers Assess 82:60–70CrossRefPubMedGoogle Scholar
  4. 4.
    Berwick DM, Nolan TW, Whittington J (2008) The triple aim: care, health, and cost. Health Aff 27:759–769CrossRefGoogle Scholar
  5. 5.
    Bickman L, Kelley SD, Breda C, de Andrade AR, Riemer M (2011) Effects of routine feedback to clinicians on mental health outcomes of youths: results of a randomized trial. Psychiatr Serv 62:1423–1429CrossRefPubMedGoogle Scholar
  6. 6.
    Carlier IV, Meuldijk D, van Vliet IM, van Fenema E, van der Wee NJ, Zitman FG (2012) Routine outcome monitoring and feedback on physical or mental health status: evidence and theory. J Eval Clin Pract 18:104–110CrossRefPubMedGoogle Scholar
  7. 7.
    Cohen J (1988) Statistical power analysis for the behavioral sciences. Lawrence Erlbaum Associates, HillsdaleGoogle Scholar
  8. 8.
    de Beurs E, den Hollander-Gijsman ME, van Rood YR, van der Wee NJ, Giltay EJ, van Noorden MS, van der Lem R, van Fenema E, Zitman FG (2011) Routine outcome monitoring in the Netherlands: practical experiences with a web-based strategy for the assessment of treatment outcome in clinical practice. Clin Psychol Psychother 18:1–12CrossRefPubMedGoogle Scholar
  9. 9.
    Fedorov V, Mannino F, Zhang R (2009) Consequences of dichotomization. Pharm Stat 8:50–61CrossRefPubMedGoogle Scholar
  10. 10.
    Fleiss JL (1973) Statistical methods for rates and proportions. Wiley, New YorkGoogle Scholar
  11. 11.
    Goodman R (2001) Psychometric properties of the strengths and difficulties questionnaire. J Am Acad Child Adolesc Psychiatr 40:1337–1345CrossRefGoogle Scholar
  12. 12.
    Goodman R, Scott S (1999) Comparing the strengths and difficulties questionnaire and the child behavior checklist: is small beautiful? J Abnorm Child Psychol 27:17–24CrossRefPubMedGoogle Scholar
  13. 13.
    Hall CL, Moldavsky M, Baldwin L, Marriot M, Newell K, Taylor J, Hollis C (2013) The use of routine outcome measures in two child and adolescent mental health services: a completed audit cycle. BMC Psychiatr 13:270CrossRefGoogle Scholar
  14. 14.
    Hermann RC, Rollins CK, Chan JA (2007) Risk-adjusting outcomes of mental health and substance-related care: a review of the literature. Harv Rev Psychiatr 15:52–69CrossRefGoogle Scholar
  15. 15.
    Hiller W, Schindler AC, Lambert MJ (2012) Defining response and remission in psychotherapy research: a comparison of the RCI and the method of percent improvement. Psychother Res 22:1–11CrossRefPubMedGoogle Scholar
  16. 16.
    Hudziak JJ, Achenbach TM, Althoff RR, Pine DS (2007) A dimensional approach to developmental psychopathology. Int J Methods Psychiatr Res 16:S16–S23CrossRefPubMedGoogle Scholar
  17. 17.
    Jacobson NS, Follette WC, Revenstorf D (1984) Psychotherapy outcome research: methods for reporting variability and evaluating clinical significance. Behav Ther 15:336–352CrossRefGoogle Scholar
  18. 18.
    Jacobson NS, Truax P (1991) Clinical significance: a statistical approach to defining meaningful change in psychotherapy research. J Consul Clin Psychol 59:12–19CrossRefGoogle Scholar
  19. 19.
    Jaeschke R, Singer J, Guyatt GH (1989) Measurement of health status: ascertaining the minimal clinically important difference. Control Clin Trials 10(4):407–415CrossRefPubMedGoogle Scholar
  20. 20.
    Kelley SD, Bickman L (2009) Beyond routine outcome monitoring: measurement feedback systems (MFS) in child and adolescent clinical practice. Cur Opin Psychiatr 22:363–368CrossRefGoogle Scholar
  21. 21.
    Klugh HE (2006) Normalized T Scores. In: Kotz S, Read CB, Balakrishnan N, Vidakovic B (eds) Encyclopedia of statistical sciences, 2nd edn. Wiley, New YorkGoogle Scholar
  22. 22.
    Knaup C, Koesters M, Schoefer D, Becker T, Puschner B (2009) Effect of feedback of treatment outcome in specialist mental healthcare: meta-analysis. Br J Psychiatry 195:15–22CrossRefPubMedGoogle Scholar
  23. 23.
    Kraemer HC, Morgan GA, Leech NL, Gliner JA, Vaske JJ, Harmon RJ (2003) Measures of clinical significance. J Am Ac Child Adolesc Psychiatr 42:1524–1529CrossRefGoogle Scholar
  24. 24.
    Lambert M (2007) Presidential address: what we have learned from a decade of research aimed at improving psychotherapy outcome in routine care. Psych Res 17:1–14Google Scholar
  25. 25.
    Markon KE, Chmielewski M, Miller CJ (2011) The reliability and validity of discrete and continuous measures of psychopathology: a quantitative review. Psych Bul 137:856–879CrossRefGoogle Scholar
  26. 26.
    McGlinchey JB, Atkins DC, Jacobson NS (2002) Clinical significance methods: which one to use and how useful are they? Behav Ther 33:529–550CrossRefGoogle Scholar
  27. 27.
    Muris P, Meesters C, van den Berg F (2003) The strengths and difficulties questionnaire (SDQ). Eur Child Adolesc Psychiatr 12:1–8CrossRefGoogle Scholar
  28. 28.
    Norman GR, Sloan JA, Wyrwich KW (2003) Interpretation of changes in health-related quality of life: the remarkable universality of half a standard deviation. Med Care 41:582–592PubMedGoogle Scholar
  29. 29.
    Porter ME, Teisberg EO (2006) Redefining healthcare: creating value-based competition on results. Harvard Business Press, CambridgeGoogle Scholar
  30. 30.
    Russel M (2000) Summarizing change in test scores: shortcomings of three common methods. Practical Assessment, Research & Evaluation 7Google Scholar
  31. 31.
    Sloan JA, Cella D, Hays RD (2005) Clinical significance of patient-reported questionnaire data: another step toward consensus. J Clin Epidem 58:1217–1219CrossRefGoogle Scholar
  32. 32.
    van Widenfelt B, Goedhart A, Treffers P, Goodman R (2003) Dutch version of the strengths and difficulties questionnaire (SDQ). Eur Child Adolesc Psychiatr 12:281–289CrossRefGoogle Scholar
  33. 33.
    Verhulst FC, Van der Ende J, Koot HM (1996) Handleiding voor de CBCL/4-18 [Dutch manual for the CBCL/4-18]. Department of Child and Adolescent Psychiatry, Erasmus University, Sophia Children’s Hospital, RotterdamGoogle Scholar
  34. 34.
    Walton K, Ormel J, Krueger R (2011) The dimensional nature of externalizing behaviors in adolescence: evidence from a direct comparison of categorical, dimensional, and hybrid models. J Abnorm Child Psychol 39:553–561CrossRefPubMedGoogle Scholar
  35. 35.
    Wilder J (1965) Pitfalls in the methodology of the law of initial value. Am J Psychother 19:577–584PubMedGoogle Scholar
  36. 36.
    Wise EA (2004) Methods for analyzing psychotherapy outcomes: a review of clinical significance, reliable change, and recommendations for future directions. J Pers Assess 82:50–59CrossRefPubMedGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2014

Authors and Affiliations

  • Edwin de Beurs
    • 1
    Email author
  • Marko Barendregt
    • 1
  • Bente Rogmans
    • 2
  • Sylvana Robbers
    • 3
  • Marieke van Geffen
    • 4
  • Marleen van Aggelen-Gerrits
    • 5
  • Huub Houben
    • 6
  1. 1.Stichting Benchmark GGZBilthovenThe Netherlands
  2. 2.Department of Clinical PsychologyLeiden UniversityLeidenThe Netherlands
  3. 3.Yulius AcademyRotterdamThe Netherlands
  4. 4.De ViersprongHalsterenThe Netherlands
  5. 5.PIONNPapenvoortThe Netherlands
  6. 6.Mentaal BeterDen DolderThe Netherlands

Personalised recommendations