Abstract
This paper aims to provide researchers with practical information on sample sizes for accurate estimations of therapist effects (TEs). The investigations are based on an integrated sample of 48,648 patients treated by 1800 therapists. Multilevel modeling and resampling were used to realize varying sample size conditions to generate empirical estimates of TEs. Sample size tables, including varying sample size conditions, were constructed and study examples given. This study gives an insight into the potential size of the TE and provides researchers with a practical guide to aid the planning of future studies in this field.
Notes
To reduce the complexity of this paper, AIC values are not reported in detail but can be requested from the first author.
Because of the nested data structure, three sample size parameters must be considered when weighting the arithmetic mean: (1) number of patients (2) number of therapists (3) number of patients per therapist. The mean TE weighted for the number of patients is 7.2 %, the mean TE weighted for the number of therapists is 7.1 and 5.75 % if the mean TE is weighted by the mean number of patients per therapist.
Due to shortage of space and clarity, not all sample size conditions are displayed in Table 3.
The explained variance (R2) was calculated in accordance with the recommendations of Hox (2010).
References
Adelson, L. J., & Owen, J. (2012). Bringing the psychotherapist back: Basic concepts for reading articles examining therapist effects using multilevel modeling. Psychotherapy, 49(2), 152–162. doi:10.1037/a0023990.
Baldwin, S. A., & Imel, Z. E. (2013). Therapist effects: Findings and methods. In M. J. Lambert (Ed.), Bergin and Garfield’s handbook of psychotherapy and behavior change (6th ed., pp. 258–297). New York, NY: Wiley.
Barkham, M., Margison, F., Leach, C., Lucock, M., Mellor-Clark, J., Evans, C.,…McGrath, G. (2001). Service profiling and outcomes benchmarking using the CORE-OM: Towards practice-based evidence in the psychological therapies. Journal of Consulting and Clinical Psychology, 69, 184–196. doi:10.1037/0022-006X.69.2.184.
Bates, D., Maechler, M., & Bolker, B. (2013). lme4: Linear mixed-effects models using s4classes [Software-Handbook]. (R package version 0.999999-2)
Bell, B. A., Morgan, G. B., Schoeneberger, J. A., Kromrey, J. D., & Ferron, J. M. (2014). How low can you go? An investigation of the influence of sample size and model complexity on point and interval estimates in two-level linear models. Methodology: European Journal of Research Methods for the Behavioral and Social Sciences, 10(1), 1–11. doi:10.1027/1614-2241/a000062.
Beutler, L. E., Malik, A., Alimohamed, S., Harwood, T. M., Talebi, H., Noble, W., & Wong, E. (2004). Therapist variables. In M. J. Lambert (Ed.), Bergin and Garfield’s handbook of psychotherapy and behavior change (5th ed., pp. 227–306). New York, NY: Wiley.
Boswell, J. F., Castonguay, L. G., & Wasserman, R. H. (2010). Effects of psychotherapy training and intervention use on session outcome. Journal of Consulting and Clinical Psychology, 78, 717–723. doi:10.1037/a0020088.
Burton, P., Gurrin, L., & Sly, P. (1998). Extending the simple linear regression model to account for correlated responses: An introduction to gear generalized estimating equations and multilevel mixed modelling. Statistics in Medicine, 17, 1261–1291.
Crits-Christoph, P., Baranackie, K., Kurcias, J., Beck, A., Carroll, K., Perry, K.,… Zitrin, C. (1991). Meta-analysis of therapist effects in psychotherapy outcome studies. Psychotherapy Research, 1, 81–91. doi:10.1080/10503309112331335511.
Crits-Christoph, P., & Gallop, R. (2006). Therapist effects in the National Institute of Mental Health Treatment of Depression Collaborative Research Program and other psychotherapy studies. Psychotherapy Research, 16, 178–181. doi:10.1080/10503300500265025.
De Jong, K., Moerbeek, M., & Van der Leeden, R. (2010). A priori power analysis in longitudinal three-level multilevel models: an example with therapist effects. Psychotherapy Research, 20(3), 273–284. doi:10.1080/10503300903376320.
Derogatis, L. R. (1975). Brief symptom inventory. Baltimore: Clinical Psychometric Research.
Derogatis, L. R. (1977). SCL-90-R: Administration, Scoring and Procedures Manual I. Baltimore: Clinical Psychometric Research.
Dinger, U., Strack, M., Leichsenring, F., Wilmers, F., & Schauenburg, H. (2008). Therapist effects on outcome and alliance in inpatient psychotherapy. Journal of Clinical Psychology, 64, 344–354. doi:10.1002/jclp.20443.
Ehlers, A., Grey, N., Wild, J., Stott, R., Liness, S., Deale, A.,… & Clark, D. M. (2013). Implementation of cognitive therapy for PTSD in routine clinical care: effectiveness and moderators of outcome in a consecutive sample. Behaviour Research and Therapy, 51, 742–752. doi:10.1016/j.brat.2013.08.006.
Eldridge, S. M., Ashby, D., & Kerry, S. (2006). Sample size for cluster randomized trials: effect of coefficient of variation of cluster size and analysis method. International Journal of Epidemiology, 35(5), 1292–1300. doi:10.1093/ije/dyl129.
Elkin, I., Falconnier, L., & Martinovich, Z. (2007). Misrepresentations in Wampold and Bolt’s critique of Elkin, Falconnier, Martinovich, and Mahoney’s study of therapist effects. Psychotherapy Research, 17, 253–256. doi:10.1080/10503300601039816.
Elkin, I., Falconnier, L., Martinovich, Z., & Mahoneya, C. (2006). Therapist effects in the National Institute of Mental Health Treatment of Depression Collaborative Research Program. Psychotherapy Research, 16, 144–160. doi:10.1080/10503300500268540.
Elkin, I., Shea, M. T., Watkins, J. T., Imber, S. D., Sotsky, S. M., Collins, J. F., & Parloff, M. B. (1989). National Institute of Mental Health Treatment of Depression Collaborative Research Program. General effectiveness of treatments. Archives of General Psychiatry, 46, 971–982. doi:10.1001/archpsyc.1989.01810110013002.
Evans, C., Connell, J., Barkham, M., Margison, F., McGrath, G., Mellor-Clark, J., & Audin, K. (2002). Toward a standardized brief outcome measure: Psychometric properties and utility of the CORE-OM. The British Journal of Psychiatry, 180, 51–60. doi:10.1192/bjp.180.1.51.
Franke, G. (2000). BSI: Brief Symptom Inventory von L.R. Derogatis (Kurzform der SCL-90-R) – Deutsche Version. Beltz Test GmbH.
Gao, F., Earnest, A., Matchar, D. B., Campbell, M. J., & Machin, D. (2015). Sample size calculations for the design of cluster randomized trials: A summary of methodology. Contemporary Clinical Trials, 42, 41–50.
Gardiner, J. C., Luo, Z., & Roman, L. A. (2009). Fixed effects, random effects and GEE: What are the differences? Statistics in Medicine, 28, 221–239. doi:10.1002/sim.3478.
Garfield, S. L. (1997). The therapist as a neglected variable in psychotherapy research. Clinical Psychology: Science and Practice, 4, 40–43. doi:10.1111/j.1468-2850.1997.tb00097.x.
Hofmann, S. G., & Barlow, D. H. (2014). Evidence-based psychological interventions and the common factors approach: the beginnings of a rapprochement? Psychotherapy, 5, 510–513. doi:10.1037/a0037045.
Howard, K., Brill, P., Lueger, R., O’Mahoney, M. & Grissom, G. (1993b). Compass outpatient tracking assessment: Psychometric properties. Integra.
Howard, K., Lueger, R., Maling, M., & Martinovich, Z. (1993a). A phase model of psychotherapy outcome: Causal mediation of change. Journal of Consulting and Clinical Psychology, 61, 678–685. doi:10.1037/0022-006X.61.4.678.
Howard, K. I., Moras, K., Brill, P. L., Martinovich, Z., & Lutz, W. (1996). Evaluation of psychotherapy: Efficacy, effectiveness, and patient progress. American Psychologist, 51, 1059–1064. doi:10.1037/0003-066X.51.10.1059.
Hox, J. (2010). Multilevel analysis: Techniques and applications (2nd ed.). England: Routledge.
Huppert, J. D., Bufka, L. F., Barlow, D. H., Gorman, J. M., Shear, M. K., & Woods, S. W. (2001). Therapists, therapist variables, and cognitive-behavioral therapy outcome in a multicenter trial for panic disorder. Journal of Consulting and Clinical Psychology, 69, 747–755. doi:10.1037/0022-006X.69.5.747.
Kazdin, A. E., & Bass, D. (1989). Power to detect differences between alternative treatments in comparative psychotherapy outcome research. Journal of Consulting and Clinical Psychology, 57, 138–147. doi:10.1037/0022-006X.57.1.138.
Kim, D., Wampold, B. E., & Bolt, D. M. (2006). Therapist effects in psychotherapy: A random-effects modeling of the National Institute of Mental Health Treatment of Depression Collaborative Research Program data. Psychotherapy Research, 16, 161–172. doi:10.1080/10503300500264911.
Kopta, S., & Lowry, J. (2002). Psychometric evaluation of the behavioral health questionnaire-20: A brief instrument for assessing global mental health and the three phases of psychotherapy outcome. Psychotherapy Research, 12, 413–426. doi:10.1093/ptr/12.4.413.
Kreft, I. G. G. (1996). Are multilevel techniques necessary? An overview, including imulation studies. Unpublished manuscript. Los Angeles: University of California, Department of Statistics.
Kroenke, K., Spitzer, R. L., & Williams, J. B. (2001). The PHQ-9: Validity of a brief depression severity measure. Journal of General Internal Medicine, 16, 606–613. doi:10.1046/j.1525-1497.2001.016009606.x.
Kuznetsova, A., Brockhoff, P. B., & Christensen, R. H. B. (2014). lmerTest: Tests for random and fixed effects for linear mixed effect models (lmer objects of lme4 package). (R package version 2.0-6)
Lambert, M. J. (1992). Psychotherapy outcome research: Implications for integrative and eclectic therapists. In J. C. Norcross & M. R. Goldfried (Eds.), Handbook of psychotherapy integration. New York: Basic Books.
Lambert, M. J. (2004). Administration and scoring manual for the OQ-45.2 (outcome questionnaire). OQ Measures, LLC.
Lambert, M. J., Hansen, N. B., Umphress, V., Lunnen, K., Okiishi, J., Burlingame, G. M., et al. (1996). Administration and scoring manual for the Outcome Questionnaire (OQ-45.2). Stevenson MD: American Professional Credentialing Services.
Lueger, R. J., Howard, K. I., Martinovich, Z., Lutz, W., Anderson, E. E., & Grissom, G. (2001). Assessing treatment progress of individual patients using expected treatment response models. Journal of Consulting and Clinical Psychology, 69, 150–158. doi:10.1037/0022-006X.69.2.150.
Lutz, W., & Barkham, M. (2015). Therapist effects. The encyclopedia of clinical psychology. Blackwell: Wiley.
Lutz, W., Böhnke, J. R., & Köck, K. (2011). Lending an ear to feedback systems: Evaluation of recovery and non-response in psychotherapy in a German outpatient setting. Community Mental Health Journal, 47, 311–317. doi:10.1007/s10597-010-9307-3.
Lutz, W., Leon, S. C., Martinovich, Z., Lyons, J. S., & Stiles, W. B. (2007). Therapist effects in outpatient psychotherapy: A three-level growth curve approach. Journal of Counseling Psychology, 54, 32–39. doi:10.1037/0022-0167.54.1.32.
Lyons, J. S., Howard, K. I., O’Mahoney, M. T., & Lish, J. (1997). The measurement and management of clinical outcomes in mental health services. New York, NY: Wiley.
Maas, C. J. M., & Hox, J. J. (2005). Sufficient sample sizes for multilevel modeling. Methodology, 1, 86–92. doi:10.1027/1614-1881.1.3.86.
Manea, L., Gilbody, S., & McMillan, D. (2012). Optimal cut-off scores for diagnosing depression with the Patient Health Questionnaire (PHQ-9): A meta-analysis. Canadian Medical Association Journal, 184(3), E191–E196. doi:10.1503/cmaj.110829.
Moerbeek, M. (2014). Multilevel modeling in the context of growth modeling. Annals of Nutrition & Metabolism, 65, 121–128. doi:10.1159/000360485.
Musca, S. C., Kamiejski, R., Nugier, A., Méot, A., Er-Rafiy, A., & Brauer, M. (2011). Data with hierarchical structure: impact of intraclass correlation and sample size on Type-I error. Frontiers in Psychology, 2, 1–6. doi:10.3389/fpsyg.2011.00074.
National Institute for Health and Care Excellence (2011). Common mental health disorders: Identification and pathways to care. [CG123]. London: National Institute for Health and Care Excellence. Retrieved on 21/04/2015 from http://www.nice.org.uk/guidance/CG123.
Okiishi, J. C., Lambert, M. J., Eggett, D., Nielsen, L., & Dayton, D. D. (2006). An analysis of therapist treatment effects: Toward providing feedback to individual therapists on their clients' psychotherapy outcome. Journal of Clinical Psychology, 62(9), 1157–1172. doi:10.1002/jclp.20272.
Okiishi, J., Lambert, M. J., Nielsen, S. L., & Ogles, B. M. (2003). Waiting for supershrink: An empirical analysis of therapist effects. Clinical Psychology & Psychotherapy, 10, 361–373. doi:10.1002/cpp.383.
Owen, J., Tao, K., & Rodolfa, E. (2010). Microaggressions and women in short-term psychotherapy: Initial evidence. The Counseling Psychologist, 38, 923–946. doi:0.1177/0011000010376093.
R Development Core Team (2014). R [Computer software]. Retrieved from http://www.R-project.org/.
Raudenbush, S., & Bryk, A. (2002). Hierarchical linear models (2nd ed.). Newbury Park, CA: Sage.
Ricks, D. F. (1974). Supershrink: Methods of a therapist judged successful on the basis of adult outcomes of adolescent patients. In D. F. Ricks, M. Roff, & A. Thomas (Eds.), Life history research in psychopathology (Vol. 3, pp. 275–297). Minneapolis: University of Minnesota Press.
Saxon, D., & Barkham, M. (2012). Patterns of therapist variability: Therapist effects and the contribution of patient severity and risk. Journal of Consulting and Clinical Psychology, 80, 535–546. doi:10.1037/a0028898.
Shoukri, M. M., Asyali, M. H., & Donner, A. (2004). Sample size requirements for the design of reliability study: Review and new results. Statistical Methods in Medical Research, 13, 251–271. doi:10.1191/0962280204sm365ra.
Wampold, B. E., & Bolt, D. M. (2006). Therapist effects: Clever ways to make them (and everything else) disappear. Psychotherapy Research, 16, 184–187. doi:10.1080/10503300500265181.
Author information
Authors and Affiliations
Corresponding author
Appendix
Appendix
Two-Level Hierarchical Model
Level 1 (Patient Level): Outcomepost ij = π 0j + π 1j * initial impairment_ centered ij + e ij
Three-Level Hierarchical Model
Level 1 (Patient Level): Outcomepost ijk = π0jk + π1jk * initial impairment_ centered ijk + e ijk
Note. MLM formulas for the hierarchical models predicting treatment outcome where patient i is nested within therapist j and therapist j is nested within dataset k. For each of the eight datasets, initial impairment was standardized on the mean and standard deviation of an appropriate country-specific outpatient reference sample (initial impairment_centered; see footnote 1) and included as a predictor on level 1 in order to capture the individual patient’s psychological distress at intake as a deviation from the relevant population mean. Considering the AIC a random intercept (r 0jk; u 00k) and random slope (r 1jk; u 10k) model consistently fit the data best.
Rights and permissions
About this article
Cite this article
Schiefele, AK., Lutz, W., Barkham, M. et al. Reliability of Therapist Effects in Practice-Based Psychotherapy Research: A Guide for the Planning of Future Studies. Adm Policy Ment Health 44, 598–613 (2017). https://doi.org/10.1007/s10488-016-0736-3
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10488-016-0736-3