Skip to main content
Log in

Reliability of Therapist Effects in Practice-Based Psychotherapy Research: A Guide for the Planning of Future Studies

Administration and Policy in Mental Health and Mental Health Services Research Aims and scope Submit manuscript

Abstract

This paper aims to provide researchers with practical information on sample sizes for accurate estimations of therapist effects (TEs). The investigations are based on an integrated sample of 48,648 patients treated by 1800 therapists. Multilevel modeling and resampling were used to realize varying sample size conditions to generate empirical estimates of TEs. Sample size tables, including varying sample size conditions, were constructed and study examples given. This study gives an insight into the potential size of the TE and provides researchers with a practical guide to aid the planning of future studies in this field.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2

Notes

  1. To reduce the complexity of this paper, AIC values are not reported in detail but can be requested from the first author.

  2. Because of the nested data structure, three sample size parameters must be considered when weighting the arithmetic mean: (1) number of patients (2) number of therapists (3) number of patients per therapist. The mean TE weighted for the number of patients is 7.2 %, the mean TE weighted for the number of therapists is 7.1 and 5.75 % if the mean TE is weighted by the mean number of patients per therapist.

  3. Due to shortage of space and clarity, not all sample size conditions are displayed in Table 3.

  4. The explained variance (R2) was calculated in accordance with the recommendations of Hox (2010).

References

  • Adelson, L. J., & Owen, J. (2012). Bringing the psychotherapist back: Basic concepts for reading articles examining therapist effects using multilevel modeling. Psychotherapy, 49(2), 152–162. doi:10.1037/a0023990.

    Article  PubMed  Google Scholar 

  • Baldwin, S. A., & Imel, Z. E. (2013). Therapist effects: Findings and methods. In M. J. Lambert (Ed.), Bergin and Garfield’s handbook of psychotherapy and behavior change (6th ed., pp. 258–297). New York, NY: Wiley.

    Google Scholar 

  • Barkham, M., Margison, F., Leach, C., Lucock, M., Mellor-Clark, J., Evans, C.,…McGrath, G. (2001). Service profiling and outcomes benchmarking using the CORE-OM: Towards practice-based evidence in the psychological therapies. Journal of Consulting and Clinical Psychology, 69, 184–196. doi:10.1037/0022-006X.69.2.184.

  • Bates, D., Maechler, M., & Bolker, B. (2013). lme4: Linear mixed-effects models using s4classes [Software-Handbook]. (R package version 0.999999-2)

  • Bell, B. A., Morgan, G. B., Schoeneberger, J. A., Kromrey, J. D., & Ferron, J. M. (2014). How low can you go? An investigation of the influence of sample size and model complexity on point and interval estimates in two-level linear models. Methodology: European Journal of Research Methods for the Behavioral and Social Sciences, 10(1), 1–11. doi:10.1027/1614-2241/a000062.

    Article  Google Scholar 

  • Beutler, L. E., Malik, A., Alimohamed, S., Harwood, T. M., Talebi, H., Noble, W., & Wong, E. (2004). Therapist variables. In M. J. Lambert (Ed.), Bergin and Garfield’s handbook of psychotherapy and behavior change (5th ed., pp. 227–306). New York, NY: Wiley.

    Google Scholar 

  • Boswell, J. F., Castonguay, L. G., & Wasserman, R. H. (2010). Effects of psychotherapy training and intervention use on session outcome. Journal of Consulting and Clinical Psychology, 78, 717–723. doi:10.1037/a0020088.

    Article  PubMed  Google Scholar 

  • Burton, P., Gurrin, L., & Sly, P. (1998). Extending the simple linear regression model to account for correlated responses: An introduction to gear generalized estimating equations and multilevel mixed modelling. Statistics in Medicine, 17, 1261–1291.

    Article  PubMed  CAS  Google Scholar 

  • Crits-Christoph, P., Baranackie, K., Kurcias, J., Beck, A., Carroll, K., Perry, K.,… Zitrin, C. (1991). Meta-analysis of therapist effects in psychotherapy outcome studies. Psychotherapy Research, 1, 81–91. doi:10.1080/10503309112331335511.

  • Crits-Christoph, P., & Gallop, R. (2006). Therapist effects in the National Institute of Mental Health Treatment of Depression Collaborative Research Program and other psychotherapy studies. Psychotherapy Research, 16, 178–181. doi:10.1080/10503300500265025.

    Article  Google Scholar 

  • De Jong, K., Moerbeek, M., & Van der Leeden, R. (2010). A priori power analysis in longitudinal three-level multilevel models: an example with therapist effects. Psychotherapy Research, 20(3), 273–284. doi:10.1080/10503300903376320.

    Article  PubMed  Google Scholar 

  • Derogatis, L. R. (1975). Brief symptom inventory. Baltimore: Clinical Psychometric Research.

    Google Scholar 

  • Derogatis, L. R. (1977). SCL-90-R: Administration, Scoring and Procedures Manual I. Baltimore: Clinical Psychometric Research.

    Google Scholar 

  • Dinger, U., Strack, M., Leichsenring, F., Wilmers, F., & Schauenburg, H. (2008). Therapist effects on outcome and alliance in inpatient psychotherapy. Journal of Clinical Psychology, 64, 344–354. doi:10.1002/jclp.20443.

    Article  PubMed  Google Scholar 

  • Ehlers, A., Grey, N., Wild, J., Stott, R., Liness, S., Deale, A.,… & Clark, D. M. (2013). Implementation of cognitive therapy for PTSD in routine clinical care: effectiveness and moderators of outcome in a consecutive sample. Behaviour Research and Therapy, 51, 742–752. doi:10.1016/j.brat.2013.08.006.

  • Eldridge, S. M., Ashby, D., & Kerry, S. (2006). Sample size for cluster randomized trials: effect of coefficient of variation of cluster size and analysis method. International Journal of Epidemiology, 35(5), 1292–1300. doi:10.1093/ije/dyl129.

    Article  PubMed  Google Scholar 

  • Elkin, I., Falconnier, L., & Martinovich, Z. (2007). Misrepresentations in Wampold and Bolt’s critique of Elkin, Falconnier, Martinovich, and Mahoney’s study of therapist effects. Psychotherapy Research, 17, 253–256. doi:10.1080/10503300601039816.

    Article  Google Scholar 

  • Elkin, I., Falconnier, L., Martinovich, Z., & Mahoneya, C. (2006). Therapist effects in the National Institute of Mental Health Treatment of Depression Collaborative Research Program. Psychotherapy Research, 16, 144–160. doi:10.1080/10503300500268540.

    Article  Google Scholar 

  • Elkin, I., Shea, M. T., Watkins, J. T., Imber, S. D., Sotsky, S. M., Collins, J. F., & Parloff, M. B. (1989). National Institute of Mental Health Treatment of Depression Collaborative Research Program. General effectiveness of treatments. Archives of General Psychiatry, 46, 971–982. doi:10.1001/archpsyc.1989.01810110013002.

    Article  PubMed  CAS  Google Scholar 

  • Evans, C., Connell, J., Barkham, M., Margison, F., McGrath, G., Mellor-Clark, J., & Audin, K. (2002). Toward a standardized brief outcome measure: Psychometric properties and utility of the CORE-OM. The British Journal of Psychiatry, 180, 51–60. doi:10.1192/bjp.180.1.51.

    Article  PubMed  Google Scholar 

  • Franke, G. (2000). BSI: Brief Symptom Inventory von L.R. Derogatis (Kurzform der SCL-90-R) – Deutsche Version. Beltz Test GmbH.

  • Gao, F., Earnest, A., Matchar, D. B., Campbell, M. J., & Machin, D. (2015). Sample size calculations for the design of cluster randomized trials: A summary of methodology. Contemporary Clinical Trials, 42, 41–50.

    Article  PubMed  Google Scholar 

  • Gardiner, J. C., Luo, Z., & Roman, L. A. (2009). Fixed effects, random effects and GEE: What are the differences? Statistics in Medicine, 28, 221–239. doi:10.1002/sim.3478.

    Article  PubMed  Google Scholar 

  • Garfield, S. L. (1997). The therapist as a neglected variable in psychotherapy research. Clinical Psychology: Science and Practice, 4, 40–43. doi:10.1111/j.1468-2850.1997.tb00097.x.

    Article  Google Scholar 

  • Hofmann, S. G., & Barlow, D. H. (2014). Evidence-based psychological interventions and the common factors approach: the beginnings of a rapprochement? Psychotherapy, 5, 510–513. doi:10.1037/a0037045.

    Article  Google Scholar 

  • Howard, K., Brill, P., Lueger, R., O’Mahoney, M. & Grissom, G. (1993b). Compass outpatient tracking assessment: Psychometric properties. Integra.

  • Howard, K., Lueger, R., Maling, M., & Martinovich, Z. (1993a). A phase model of psychotherapy outcome: Causal mediation of change. Journal of Consulting and Clinical Psychology, 61, 678–685. doi:10.1037/0022-006X.61.4.678.

    Article  PubMed  CAS  Google Scholar 

  • Howard, K. I., Moras, K., Brill, P. L., Martinovich, Z., & Lutz, W. (1996). Evaluation of psychotherapy: Efficacy, effectiveness, and patient progress. American Psychologist, 51, 1059–1064. doi:10.1037/0003-066X.51.10.1059.

    Article  PubMed  CAS  Google Scholar 

  • Hox, J. (2010). Multilevel analysis: Techniques and applications (2nd ed.). England: Routledge.

    Book  Google Scholar 

  • Huppert, J. D., Bufka, L. F., Barlow, D. H., Gorman, J. M., Shear, M. K., & Woods, S. W. (2001). Therapists, therapist variables, and cognitive-behavioral therapy outcome in a multicenter trial for panic disorder. Journal of Consulting and Clinical Psychology, 69, 747–755. doi:10.1037/0022-006X.69.5.747.

    Article  PubMed  CAS  Google Scholar 

  • Kazdin, A. E., & Bass, D. (1989). Power to detect differences between alternative treatments in comparative psychotherapy outcome research. Journal of Consulting and Clinical Psychology, 57, 138–147. doi:10.1037/0022-006X.57.1.138.

    Article  PubMed  CAS  Google Scholar 

  • Kim, D., Wampold, B. E., & Bolt, D. M. (2006). Therapist effects in psychotherapy: A random-effects modeling of the National Institute of Mental Health Treatment of Depression Collaborative Research Program data. Psychotherapy Research, 16, 161–172. doi:10.1080/10503300500264911.

    Article  Google Scholar 

  • Kopta, S., & Lowry, J. (2002). Psychometric evaluation of the behavioral health questionnaire-20: A brief instrument for assessing global mental health and the three phases of psychotherapy outcome. Psychotherapy Research, 12, 413–426. doi:10.1093/ptr/12.4.413.

    Article  Google Scholar 

  • Kreft, I. G. G. (1996). Are multilevel techniques necessary? An overview, including imulation studies. Unpublished manuscript. Los Angeles: University of California, Department of Statistics.

  • Kroenke, K., Spitzer, R. L., & Williams, J. B. (2001). The PHQ-9: Validity of a brief depression severity measure. Journal of General Internal Medicine, 16, 606–613. doi:10.1046/j.1525-1497.2001.016009606.x.

    Article  PubMed  PubMed Central  CAS  Google Scholar 

  • Kuznetsova, A., Brockhoff, P. B., & Christensen, R. H. B. (2014). lmerTest: Tests for random and fixed effects for linear mixed effect models (lmer objects of lme4 package). (R package version 2.0-6)

  • Lambert, M. J. (1992). Psychotherapy outcome research: Implications for integrative and eclectic therapists. In J. C. Norcross & M. R. Goldfried (Eds.), Handbook of psychotherapy integration. New York: Basic Books.

    Google Scholar 

  • Lambert, M. J. (2004). Administration and scoring manual for the OQ-45.2 (outcome questionnaire). OQ Measures, LLC.

  • Lambert, M. J., Hansen, N. B., Umphress, V., Lunnen, K., Okiishi, J., Burlingame, G. M., et al. (1996). Administration and scoring manual for the Outcome Questionnaire (OQ-45.2). Stevenson MD: American Professional Credentialing Services.

  • Lueger, R. J., Howard, K. I., Martinovich, Z., Lutz, W., Anderson, E. E., & Grissom, G. (2001). Assessing treatment progress of individual patients using expected treatment response models. Journal of Consulting and Clinical Psychology, 69, 150–158. doi:10.1037/0022-006X.69.2.150.

    Article  PubMed  CAS  Google Scholar 

  • Lutz, W., & Barkham, M. (2015). Therapist effects. The encyclopedia of clinical psychology. Blackwell: Wiley.

    Google Scholar 

  • Lutz, W., Böhnke, J. R., & Köck, K. (2011). Lending an ear to feedback systems: Evaluation of recovery and non-response in psychotherapy in a German outpatient setting. Community Mental Health Journal, 47, 311–317. doi:10.1007/s10597-010-9307-3.

    Article  PubMed  Google Scholar 

  • Lutz, W., Leon, S. C., Martinovich, Z., Lyons, J. S., & Stiles, W. B. (2007). Therapist effects in outpatient psychotherapy: A three-level growth curve approach. Journal of Counseling Psychology, 54, 32–39. doi:10.1037/0022-0167.54.1.32.

    Article  Google Scholar 

  • Lyons, J. S., Howard, K. I., O’Mahoney, M. T., & Lish, J. (1997). The measurement and management of clinical outcomes in mental health services. New York, NY: Wiley.

    Google Scholar 

  • Maas, C. J. M., & Hox, J. J. (2005). Sufficient sample sizes for multilevel modeling. Methodology, 1, 86–92. doi:10.1027/1614-1881.1.3.86.

    Article  Google Scholar 

  • Manea, L., Gilbody, S., & McMillan, D. (2012). Optimal cut-off scores for diagnosing depression with the Patient Health Questionnaire (PHQ-9): A meta-analysis. Canadian Medical Association Journal, 184(3), E191–E196. doi:10.1503/cmaj.110829.

    Article  PubMed  Google Scholar 

  • Moerbeek, M. (2014). Multilevel modeling in the context of growth modeling. Annals of Nutrition & Metabolism, 65, 121–128. doi:10.1159/000360485.

    Article  CAS  Google Scholar 

  • Musca, S. C., Kamiejski, R., Nugier, A., Méot, A., Er-Rafiy, A., & Brauer, M. (2011). Data with hierarchical structure: impact of intraclass correlation and sample size on Type-I error. Frontiers in Psychology, 2, 1–6. doi:10.3389/fpsyg.2011.00074.

    Article  Google Scholar 

  • National Institute for Health and Care Excellence (2011). Common mental health disorders: Identification and pathways to care. [CG123]. London: National Institute for Health and Care Excellence. Retrieved on 21/04/2015 from http://www.nice.org.uk/guidance/CG123.

  • Okiishi, J. C., Lambert, M. J., Eggett, D., Nielsen, L., & Dayton, D. D. (2006). An analysis of therapist treatment effects: Toward providing feedback to individual therapists on their clients' psychotherapy outcome. Journal of Clinical Psychology, 62(9), 1157–1172. doi:10.1002/jclp.20272.

    Article  PubMed  Google Scholar 

  • Okiishi, J., Lambert, M. J., Nielsen, S. L., & Ogles, B. M. (2003). Waiting for supershrink: An empirical analysis of therapist effects. Clinical Psychology & Psychotherapy, 10, 361–373. doi:10.1002/cpp.383.

    Article  Google Scholar 

  • Owen, J., Tao, K., & Rodolfa, E. (2010). Microaggressions and women in short-term psychotherapy: Initial evidence. The Counseling Psychologist, 38, 923–946. doi:0.1177/0011000010376093.

    Article  Google Scholar 

  • R Development Core Team (2014). R [Computer software]. Retrieved from http://www.R-project.org/.

  • Raudenbush, S., & Bryk, A. (2002). Hierarchical linear models (2nd ed.). Newbury Park, CA: Sage.

    Google Scholar 

  • Ricks, D. F. (1974). Supershrink: Methods of a therapist judged successful on the basis of adult outcomes of adolescent patients. In D. F. Ricks, M. Roff, & A. Thomas (Eds.), Life history research in psychopathology (Vol. 3, pp. 275–297). Minneapolis: University of Minnesota Press.

    Google Scholar 

  • Saxon, D., & Barkham, M. (2012). Patterns of therapist variability: Therapist effects and the contribution of patient severity and risk. Journal of Consulting and Clinical Psychology, 80, 535–546. doi:10.1037/a0028898.

    Article  PubMed  Google Scholar 

  • Shoukri, M. M., Asyali, M. H., & Donner, A. (2004). Sample size requirements for the design of reliability study: Review and new results. Statistical Methods in Medical Research, 13, 251–271. doi:10.1191/0962280204sm365ra.

    Article  Google Scholar 

  • Wampold, B. E., & Bolt, D. M. (2006). Therapist effects: Clever ways to make them (and everything else) disappear. Psychotherapy Research, 16, 184–187. doi:10.1080/10503300500265181.

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Anne-Katharina Schiefele.

Appendix

Appendix

Two-Level Hierarchical Model

Level 1 (Patient Level): Outcomepost ij = π 0j + π 1j * initial impairment_ centered ij + e ij

$$\begin{array}{*{20}l} {{\text{Level }}2 \, \left( {\text{Therapist Level}} \right):\pi_{{0{\text{j}}}} = \beta_{00} + r_{{0{\text{j}}}} } \hfill \\ {\quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \pi_{{1{\text{j}}}} = \beta_{10} + r_{{1{\text{j}}}} } \hfill \\ \end{array}$$

Three-Level Hierarchical Model

Level 1 (Patient Level): Outcomepost ijk = π0jk + π1jk * initial impairment_ centered ijk + e ijk

$$\begin{array}{*{20}l} {{\text{Level }}2 \, \left( {\text{Therapist Level}} \right):\pi_{{0{\text{jk}}}} = \beta_{00k} + r_{{0{\text{jk}}}} } \hfill \\ {\quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \pi_{{1{\text{jk}}}} = \beta_{10k} + r_{{1{\text{jk}}}} } \hfill \\ \end{array}$$
$$\begin{array}{*{20}l} {{\text{Level 3 }}\left( {\text{Dataset Level}} \right):\beta_{00k} = \gamma_{000} + u_{00k} } \hfill \\ {\quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \,\beta_{10k} = \gamma_{100} + u_{10k} } \hfill \\ \end{array}$$

Note. MLM formulas for the hierarchical models predicting treatment outcome where patient i is nested within therapist j and therapist j is nested within dataset k. For each of the eight datasets, initial impairment was standardized on the mean and standard deviation of an appropriate country-specific outpatient reference sample (initial impairment_centered; see footnote 1) and included as a predictor on level 1 in order to capture the individual patient’s psychological distress at intake as a deviation from the relevant population mean. Considering the AIC a random intercept (r 0jk; u 00k) and random slope (r 1jk; u 10k) model consistently fit the data best.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Schiefele, AK., Lutz, W., Barkham, M. et al. Reliability of Therapist Effects in Practice-Based Psychotherapy Research: A Guide for the Planning of Future Studies. Adm Policy Ment Health 44, 598–613 (2017). https://doi.org/10.1007/s10488-016-0736-3

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10488-016-0736-3

Keywords

Navigation