Journal of Public Health

, Volume 27, Issue 1, pp 29–35 | Cite as

Comparing single-level and multilevel regression analysis for risk adjustment of treatment outcomes in common mental health disorders

  • Lisanne WarmerdamEmail author
  • Edwin de Beurs
  • Marko Barendregt
  • Jos Twisk
Original Article



The aim of this paper is to compare single-level and multilevel regression analysis to obtain risk-adjusted outcomes from mental health care providers.

Subject and methods

The study population consisted of adult patients receiving treatment for common mental health disorders. The outcome was self-reported symptom level at post-test. Risk adjustment models were developed using single- and multilevel regression analysis. In the multilevel approach, a random intercept for each provider was included. The intraclass correlation coefficient was used to estimate the proportion of variability in treatment outcome between providers. Spearman correlation coefficient of ranks was used to compare results between the two approaches.


The effects of most casemix variables on outcomes were similar for the two models. The ranking of providers in both methods was also quite similar (ρ = .99). The multilevel model estimated that 5.4% of total variability in adjusted post-test scores was explained by the provider factor.


The findings of risk adjustment of mental health outcomes are quite robust for the use of single-level or multilevel regression analysis in the current study. However, given the small but significant amount of variation in outcomes that is attributable to providers, the multilevel approach is recommended for dealing with outcomes when patients are clustered within providers.


Risk adjustment Mental health care Multilevel analysis Outcomes 


Compliance with ethical standards

Ethical approval

All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki Declaration and its later amendments or comparable ethical standards.

Informed consent

The Dutch Central Medical Ethical Committee (CCMO) has ruled that Dutch Law regarding research with humans does not apply to the collection of anonymized information and, consequently, providing SBG with this data and analyzing anonymized data for the present study does not require additional informed consent from participants.

Conflicts of interest

The authors declare that they have no conflicts of interest.


  1. American Psychiatric Association (1994) Diagnostic and statistical manual of mental disorders, 4th edition, (DSM-IV). American Psychiatric Association, Washington DCGoogle Scholar
  2. Arling G, Lewis T, Kane RL, Mueller C, Flood S (2007) Improving quality assessment through multilevel modeling: the case of nursing home compare. Health Serv Res 43:1177–1199. CrossRefGoogle Scholar
  3. Arrindell WA, Ettema JHM (2003) SCL-90: Herziene handleiding bij een multidimensionale psychopathologie indicator. Swets & Zeitlinger, LisseGoogle Scholar
  4. Barkham M, Gilbert N, Connell J, Marshall C, Twigg E (2005) Suitability and utility of the CORE-OM and CORE-A for assessing severity of presenting problems in psychological therapy services based in primary and secondary care settings. Br J Psychiatry 186:239–246. CrossRefGoogle Scholar
  5. Burgess JF, Christiansen CL, Michalak SE, Morris CN (2000) Medical profiling: improving standards and risk adjustment using hierarchical models. J Health Econ 19:291–309. CrossRefGoogle Scholar
  6. de Beurs E (2008) Brief symptom inventory handleiding. PITS B.V, LeidenGoogle Scholar
  7. de Beurs E (2010) De genormaliseerde T-score, Een “euro” voor testuitslagen. MGv 65:685–696Google Scholar
  8. de Jong K, Nugter MA, Polak MG, Wagenborg JEA, Spinhoven P, Heiser WA (2007) The outcome questionnaire (OQ-45) in a Dutch population: a cross-cultural validation. Clin Psychol Psychother 14:288–301. CrossRefGoogle Scholar
  9. Derogatis LR (1975a) The symptom checklist-90-R. Clinical Psychometric Research, Inc, BaltimoreGoogle Scholar
  10. Derogatis LR (1975b) The brief symptom inventory. Clinical Psychometric Research, Inc, BaltimoreGoogle Scholar
  11. Goldstein H, Healy MJR (1995) The graphical presentation of a collection of means. J R Stat Soc Ser A Stat Soc 158:175–177. CrossRefGoogle Scholar
  12. Goldstein H, Spiegelhalter DJ (1996) League tables and their limitations: statistical issues in comparisons of institutional performance. J R Stat Soc Ser A Stat Soc 159:385–443. CrossRefGoogle Scholar
  13. Hall RC (1995) Global assessment of functioning: a modified scale. Psychosomatics 36:267–275. CrossRefGoogle Scholar
  14. Hermann RC, Rollins CK, Chan JA (2007) Risk-adjusting outcomes of mental health and substance-related care: a review of the literature. Harv Rev Psychiatry 15:52–69. CrossRefGoogle Scholar
  15. Iezzoni LE (2013) Risk adjustment for measuring healthcare outcomes, 4rd edn. Health Administration, ChicagoGoogle Scholar
  16. Israels A (2013) Methods of standardisation. Statistics Netherlands: The Hague/Heerlen.
  17. Lange A, Appelo M (2007) Korte Klachtenlijst, handleiding. Bohn Stafleu van Loghum, HoutenGoogle Scholar
  18. Lambert MJ, Gregersen AT, Burlingame GM (2004) The outcome questionnaire 45. In: Maruish M (ed) The use of psychological testing for treatment planning and outcomes assessment: instruments for adults. Lawrence Erlbaum Associates, Mahwah, pp 191–234Google Scholar
  19. Lovaglio PG (2012) Benchmarking strategies for measuring the quality of healthcare: problems and prospects. Sci World J 2012:1–13. CrossRefGoogle Scholar
  20. Lovibond SH, Lovibond PF (1995) Manual for the depression anxiety stress scales Second edition. Psychology Foundation, SydneyGoogle Scholar
  21. Moran V, Jacobs R (2015) Comparing the performance of English mental health providers in achieving patient outcomes. Soc Sci Med 140:127–135. CrossRefGoogle Scholar
  22. Nuttall D, Parkin D, Devlin N (2013) Inter-provider comparison of patient-reported outcomes: developing an adjustment to account for differences in patient case mix. Health Econ 24:41–54. CrossRefGoogle Scholar
  23. Rosen AK, Chatterjee S, Glickman ME, Spiro A, Seal P, Eisen SV (2009) Improving risk adjustment of self-reported mental health outcomes. J Behav Health Ser Res 37:292–306. Google Scholar
  24. Snijders TAB, Bosker RJ (1999) Multilevel analysis: an introduction to basic and advanced multilevel modeling. Sage Publishers, LondonGoogle Scholar
  25. Warmerdam L, Barendregt M, de Beurs E (2017) Risk adjustment of self-reported clinical outcomes in Dutch mental health care. J Public Health 25:311–319. CrossRefGoogle Scholar

Copyright information

© Springer-Verlag GmbH Germany, part of Springer Nature 2018

Authors and Affiliations

  • Lisanne Warmerdam
    • 1
    Email author
  • Edwin de Beurs
    • 1
  • Marko Barendregt
    • 1
  • Jos Twisk
    • 2
    • 3
  1. 1.Stichting Benchmark GGZ (the Dutch Benchmark Foundation in Mental Health Care)Bilthoventhe Netherlands
  2. 2.Department of Epidemiology & BiostatisticsVU University Medical CenterAmsterdamthe Netherlands
  3. 3.Department of Health SciencesVU UniversityAmsterdamthe Netherlands

Personalised recommendations