Various approaches have been proposed to denote treatment outcome, such as the effect size of the pre-to-posttest change, percentage improvement, statistically reliable change, and clinical significant change. The aim of the study is to compare these approaches and evaluate their aptitude to differentiate among child and adolescent mental healthcare providers regarding their treatment outcome. Comparing outcomes according to continuous and categorical outcome indicators using real-life data of seven mental healthcare providers, three using the Child Behavior Checklist and four using the Strengths and Difficulties Questionnaire as primary outcome measure. Within each dataset consistent differences were found between providers and the various methods led to comparable rankings of providers. Statistical considerations designate continuous outcomes as the optimal choice. Change scores have more statistical power and allow for a ranking of providers at first glance. Expressing providers’ performance in proportions of recovered, changed, unchanged, or deteriorated patients has supplementary value, as it denotes outcome in a manner more easily interpreted and appreciated by clinicians, managerial staff, and, last but not least, by patients or their parents.
This is a preview of subscription content, log in to check access.
Buy single article
Instant access to the full article PDF.
Price includes VAT for USA
Subscribe to journal
Immediate online access to all issues from 2019. Subscription will auto renew annually.
This is the net price. Taxes to be calculated in checkout.
Achenbach TM (1991) Manual for the child behavior checklist 4–18 and 1991 profiles. Department of Psychiatry, University of Vermont, Burlington
American Psychiatric Association (1994) Diagnostic and statistical manual of mental disorders IV. Author, Washington, DC
Bauer S, Lambert MJ, Nielsen SL (2004) Clinical significance methods: a comparison of statistical techniques. J Pers Assess 82:60–70
Berwick DM, Nolan TW, Whittington J (2008) The triple aim: care, health, and cost. Health Aff 27:759–769
Bickman L, Kelley SD, Breda C, de Andrade AR, Riemer M (2011) Effects of routine feedback to clinicians on mental health outcomes of youths: results of a randomized trial. Psychiatr Serv 62:1423–1429
Carlier IV, Meuldijk D, van Vliet IM, van Fenema E, van der Wee NJ, Zitman FG (2012) Routine outcome monitoring and feedback on physical or mental health status: evidence and theory. J Eval Clin Pract 18:104–110
Cohen J (1988) Statistical power analysis for the behavioral sciences. Lawrence Erlbaum Associates, Hillsdale
de Beurs E, den Hollander-Gijsman ME, van Rood YR, van der Wee NJ, Giltay EJ, van Noorden MS, van der Lem R, van Fenema E, Zitman FG (2011) Routine outcome monitoring in the Netherlands: practical experiences with a web-based strategy for the assessment of treatment outcome in clinical practice. Clin Psychol Psychother 18:1–12
Fedorov V, Mannino F, Zhang R (2009) Consequences of dichotomization. Pharm Stat 8:50–61
Fleiss JL (1973) Statistical methods for rates and proportions. Wiley, New York
Goodman R (2001) Psychometric properties of the strengths and difficulties questionnaire. J Am Acad Child Adolesc Psychiatr 40:1337–1345
Goodman R, Scott S (1999) Comparing the strengths and difficulties questionnaire and the child behavior checklist: is small beautiful? J Abnorm Child Psychol 27:17–24
Hall CL, Moldavsky M, Baldwin L, Marriot M, Newell K, Taylor J, Hollis C (2013) The use of routine outcome measures in two child and adolescent mental health services: a completed audit cycle. BMC Psychiatr 13:270
Hermann RC, Rollins CK, Chan JA (2007) Risk-adjusting outcomes of mental health and substance-related care: a review of the literature. Harv Rev Psychiatr 15:52–69
Hiller W, Schindler AC, Lambert MJ (2012) Defining response and remission in psychotherapy research: a comparison of the RCI and the method of percent improvement. Psychother Res 22:1–11
Hudziak JJ, Achenbach TM, Althoff RR, Pine DS (2007) A dimensional approach to developmental psychopathology. Int J Methods Psychiatr Res 16:S16–S23
Jacobson NS, Follette WC, Revenstorf D (1984) Psychotherapy outcome research: methods for reporting variability and evaluating clinical significance. Behav Ther 15:336–352
Jacobson NS, Truax P (1991) Clinical significance: a statistical approach to defining meaningful change in psychotherapy research. J Consul Clin Psychol 59:12–19
Jaeschke R, Singer J, Guyatt GH (1989) Measurement of health status: ascertaining the minimal clinically important difference. Control Clin Trials 10(4):407–415
Kelley SD, Bickman L (2009) Beyond routine outcome monitoring: measurement feedback systems (MFS) in child and adolescent clinical practice. Cur Opin Psychiatr 22:363–368
Klugh HE (2006) Normalized T Scores. In: Kotz S, Read CB, Balakrishnan N, Vidakovic B (eds) Encyclopedia of statistical sciences, 2nd edn. Wiley, New York
Knaup C, Koesters M, Schoefer D, Becker T, Puschner B (2009) Effect of feedback of treatment outcome in specialist mental healthcare: meta-analysis. Br J Psychiatry 195:15–22
Kraemer HC, Morgan GA, Leech NL, Gliner JA, Vaske JJ, Harmon RJ (2003) Measures of clinical significance. J Am Ac Child Adolesc Psychiatr 42:1524–1529
Lambert M (2007) Presidential address: what we have learned from a decade of research aimed at improving psychotherapy outcome in routine care. Psych Res 17:1–14
Markon KE, Chmielewski M, Miller CJ (2011) The reliability and validity of discrete and continuous measures of psychopathology: a quantitative review. Psych Bul 137:856–879
McGlinchey JB, Atkins DC, Jacobson NS (2002) Clinical significance methods: which one to use and how useful are they? Behav Ther 33:529–550
Muris P, Meesters C, van den Berg F (2003) The strengths and difficulties questionnaire (SDQ). Eur Child Adolesc Psychiatr 12:1–8
Norman GR, Sloan JA, Wyrwich KW (2003) Interpretation of changes in health-related quality of life: the remarkable universality of half a standard deviation. Med Care 41:582–592
Porter ME, Teisberg EO (2006) Redefining healthcare: creating value-based competition on results. Harvard Business Press, Cambridge
Russel M (2000) Summarizing change in test scores: shortcomings of three common methods. Practical Assessment, Research & Evaluation 7
Sloan JA, Cella D, Hays RD (2005) Clinical significance of patient-reported questionnaire data: another step toward consensus. J Clin Epidem 58:1217–1219
van Widenfelt B, Goedhart A, Treffers P, Goodman R (2003) Dutch version of the strengths and difficulties questionnaire (SDQ). Eur Child Adolesc Psychiatr 12:281–289
Verhulst FC, Van der Ende J, Koot HM (1996) Handleiding voor de CBCL/4-18 [Dutch manual for the CBCL/4-18]. Department of Child and Adolescent Psychiatry, Erasmus University, Sophia Children’s Hospital, Rotterdam
Walton K, Ormel J, Krueger R (2011) The dimensional nature of externalizing behaviors in adolescence: evidence from a direct comparison of categorical, dimensional, and hybrid models. J Abnorm Child Psychol 39:553–561
Wilder J (1965) Pitfalls in the methodology of the law of initial value. Am J Psychother 19:577–584
Wise EA (2004) Methods for analyzing psychotherapy outcomes: a review of clinical significance, reliable change, and recommendations for future directions. J Pers Assess 82:50–59
The authors are grateful to the following mental health providers for allowing us to use their outcome data: De Viersprong, Mentaal Beter, Mutsaersstichting, OCNR, Praktijk Buitenpost, Yorneo, and Yulius.
Conflict of interest
No conflicts declared.
About this article
Cite this article
de Beurs, E., Barendregt, M., Rogmans, B. et al. Denoting treatment outcome in child and adolescent psychiatry: a comparison of continuous and categorical outcomes. Eur Child Adolesc Psychiatry 24, 553–563 (2015). https://doi.org/10.1007/s00787-014-0609-9
- Treatment outcome research
- Effect size (ES)
- Reliable change index (RCI)
- Percentage improvement (PI)