Cognitive Therapy and Research

, Volume 42, Issue 4, pp 369–384 | Cite as

Revisiting How We Assess Therapist Competence in Cognitive Therapy

  • Iony D. Schmidt
  • Daniel R. Strunk
  • Robert J. DeRubeis
  • Laren R. Conklin
  • Justin D. Braun
Original Article


We report on two approaches to addressing the problem of low reliability in the assessment of therapist competence in cognitive therapy (CT). In Study 1, we utilized ratings of a session from each of six therapists and investigated how CT experts evaluate specific therapist behaviors in making ratings on the Cognitive Therapy Scale (CTS). Departures from the consensus on judgments of these therapist behaviors were associated with more discrepant competence ratings. We take these results to suggest that, at least among experts, providing greater clarity about which therapist behaviors are relevant to each CTS item may enhance reliability. In Study 2, we utilize a sample of 14 therapists and pilot test a standardized assessment of therapist competence. The standardized assessment exhibited an intra-class correlation coefficient of 0.89 for a single rater. Standardized assessments of competence were related to four measures of therapist experience and qualifications, whereas the CTS was only related to two of these measures. With these promising initial results, we encourage future research to examine the predictive validity of standardized competence assessments in CT.


Therapist competence Assessment Cognitive therapy Depression 



This study was funded in part by the Aaron T. Beck Foundation.

Compliance with Ethical Standards

Conflict of Interest

Iony D. Schmidt, Daniel R. Strunk, Robert J. DeRubeis, Laren R. Conklin, and Justin D. Braun declare that they have no conflict of interest.

Ethical Approval

All procedures performed in studies involving human participants were in accordance with the ethical standards of the Institutional and/or National Research Committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards.

Informed Consent

Informed consent was obtained from all individual participants included in the study.

Animal Rights

This article does not contain any studies with animals performed by any of the authors.

Supplementary material

10608_2018_9908_MOESM1_ESM.docx (20 kb)
Supplementary material 1 (DOCX 20 KB)


  1. Academy of Cognitive Therapy. (2018). Certification: Certification process. Retrieved from
  2. Baldwin, S. A., & Imel, Z. E. (2013). Therapist effects: Findings and methods. In M. E. Lambert (Ed.), Bergin and Garfield’s handbook of psychotherapy and behavior change (6th ed., pp. 258–297). New York: Wiley.Google Scholar
  3. Beck, A. T., Rush, A. J., Shaw, B. F., & Emery, G. (1979). Cognitive therapy of depression. New York: Guilford.Google Scholar
  4. Beck Institute for Cognitive Behavior Therapy. (2018). Cognitive Therapy Scale rating manual. Retrieved from
  5. Beck Institute for Cognitive Behavior Therapy. (2018). Cognitive Therapy Rating Scale. Retrieved from
  6. Blackburn, I., Milne, D. L., Baker, C., Standart, S., Garland, A., & Reichelt, F. K. (2001). The Revised Cognitive Therapy Scale (CTS-R): Psychometric properties. Behavioural and Cognitive Psychotherapy, 29, 431–446. Scholar
  7. Branson, A., Myles, P., Mahdi, M., & Shafran, R. (2018). The relationship between competence and patient outcome with low-intensity cognitive behavioural interventions. Behavioural and Cognitive Psychotherapy, 46, 101–114. Scholar
  8. Branson, A., Shafran, R., & Myles, P. (2015). Investigating the relationship between competence and patient outcome with CBT. Behaviour Research and Therapy, 68, 19–26. Scholar
  9. Cohen, J., Cohen, P., West, S. G., & Aiken, L. S. (2003). Applied multiple correlation/regression analysis for the behavioral sciences. Mahwah, NJ: Lawrence Erlbaum.Google Scholar
  10. Decker, S. E., Carroll, K. M., Nich, C., Canning-Ball, M., & Martino, S. (2013). Correspondence of motivational interviewing adherence and competence ratings in real and role-played client sessions. Psychological Assessment, 25, 306–312. Scholar
  11. Dennhag, I., Gibbons, M. B. C., Barber, J. P., Gallop, R., & Crits-Christoph, P. (2012). How many treatment sessions and patients are needed to create a stable score of adherence and competence in the treatment of cocaine dependence? Psychotherapy Research, 22, 475–488.CrossRefPubMedPubMedCentralGoogle Scholar
  12. DeRubeis, R. J., Brotman, M. A., & Gibbons, C. J. (2005). A conceptual and methodological analysis of the nonspecifics argument. Clinical Psychology: Science and Practice, 12, 174–183. Scholar
  13. DeRubeis, R. J., Gelfand, L. A., German, R. E., Fournier, J. C., & Forand, N. R. (2013). Understanding processes of change: How some patients reveal more than others—And some groups of therapists less—About what matters in psychotherapy. Psychotherapy Research, 24, 419–428. Scholar
  14. DeRubeis, R. J., Hollon, S. D., Amsterdam, J. D., Shelton, R. C., Young, P. R., Salomon, R. M., … Gallop, R. (2005). Cognitive therapy vs medications in the treatment of moderate to severe depression. Archives of General Psychiatry, 62, 409–416. Scholar
  15. Dobson, K. S., Shaw, B. F., & Vallis, T. M. (1985). Reliability of a measure of quality of cognitive therapy. British Journal of Clinical Psychology, 24, 295–300. Scholar
  16. Fairburn, C. G., & Cooper, Z. (2011). Therapist competence, therapy quality, and therapist training. Behaviour Research and Therapy, 49, 373–378. Scholar
  17. Hollon, S. D., Evans, M. D., Auerbach, A., DeRubeis, R. J., Elkin, I., Lowery, A., … Piasecki, J. (1988). Development of a system for rating therapies for depression: Differentiating cognitive therapy, interpersonal psychotherapy and clinical management pharmacotherapy. Unpublished manuscript. Nashville: Vanderbilt University.Google Scholar
  18. Imel, Z. E., Baldwin, S. A., Baer, J. S., Hartzler, B., Dunn, C., Rosengren, D. B., & Atkins, D. C. (2014). Evaluating therapist adherence in motivational interviewing by comparing performance with standardized and real patients. Journal of Consulting and Clinical Psychology, 82, 472–481. Scholar
  19. Jacobson, N. S., & Gortner, E. T. (2000). Can depression be de-medicalized in the 21st century: Scientific revolutions, counter-revolutions and the magnetic field of normal science. Behaviour Research and Therapy, 38, 103–117. Scholar
  20. Keen, A. J., & Freeston, M. H. (2008). Assessing competence in cognitive–behavioural therapy. The British Journal of Psychiatry, 193, 60–64. Scholar
  21. Kuyken, W., & Tsivrikos, D. (2009). Therapist competence, comorbidity and cognitive-behavioral therapy for depression. Psychotherapy and Psychosomatics, 78, 42–48. Scholar
  22. Lee, I. A., & Preacher, K. J. (2013). Calculation for the test of the difference between two dependent correlations with one variable in common [Computer software]. Available from
  23. Lilienfeld, S. O., Ammirati, R., & Landfield, K. (2009). Giving debiasing away: Can psychological research on correcting cognitive errors promote human welfare. A Journal of the Association for Psychological Science, 4, 390–398. Scholar
  24. Littell, R. C., Milliken, G. A., Stroup, W. W., Wolfinger, R. D., & Schabenberger, O. (2006). SAS system for mixed models (2nd ed.). Cary, NC: SAS Institute.Google Scholar
  25. McManus, F., Rakovshik, S., Kennerley, H., Fennell, M., & Westbrook, D. (2012). An investigation of the accuracy of therapists’ self-assessment of cognitive-behaviour therapy skills. British Journal of Clinical Psychology, 51, 292–306. Scholar
  26. Muse, K., & McManus, F. (2013). A systematic review of methods for assessing competence in cognitive-behavioural therapy. Clinical Psychology Review, 33, 484–499.CrossRefPubMedGoogle Scholar
  27. Muse, K., & McManus, F. (2016). Expert insight into the assessment of competence in cognitive–behavioural therapy: A qualitative exploration of experts’ experiences, opinions and recommendations. Clinical Psychology & Psychotherapy, 23, 246–259.CrossRefGoogle Scholar
  28. Newble, D. (2004). Techniques for measuring clinical competence: objective structured clinical examinations. Medical Education, 38, 199e203. Scholar
  29. Shaw, B. F., Elkin, I., Yamaguchi, J., Olmstead, M., Vallis, T. M., Dobson, K. S., Lowery, A., … Imber, S. D. (1999). Therapist competence ratings in relation to clinical outcome in cognitive therapy of depression. Journal of Consulting and Clinical Psychology, 67, 837–846. Scholar
  30. Sholomskas, D. E., Syracuse-Siewert, G., Rounsaville, B. J., Ball, S. A., Nuro, K. F., & Carroll, K. M. (2005). We don’t train in vain: A dissemination trial of three strategies of training clinicians in cognitive-behavioral therapy. Journal of Consulting and Clinical Psychology, 73, 106–115. Scholar
  31. Shrout, P. E., & Fleiss, J. L. (1979). Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin, 86, 420–428. Scholar
  32. Strunk, D. R., Brotman, M. A., DeRubeis, R. J., & Hollon, S. D. (2010). Therapist competence in cognitive therapy for depression: predicting subsequent symptom change. Journal of Consulting and Clinical Psychology, 78, 429–437. Scholar
  33. Trepka, C., Rees, A., Shapiro, D. A., Hardy, G. E., & Barkham, M. (2004). Therapist competence and outcome of cognitive therapy for depression. Cognitive Therapy & Research, 28, 143–157. Scholar
  34. Vallis, T. M., Shaw, B. F., & Dobson, K. S. (1986). The Cognitive Therapy Scale: Psychometric properties. Journal of Consulting and Clinical Psychology, 54, 381–385. Scholar
  35. Waltz, J., Addis, M. E., Koerner, K., & Jacobson, N. S. (1993). Testing the integrity of a psychotherapy protocol: assessment of adherence and competence. Journal of Consulting and Clinical Psychology, 61, 620–630. Scholar
  36. Webb, C. A., DeRubeis, R. J., & Barber, J. P. (2010). Therapist adherence/competence and treatment outcome: A meta-analytic review. Journal of Consulting and Clinical Psychology, 78, 200–211. Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2018

Authors and Affiliations

  1. 1.Department of PsychologyThe Ohio State UniversityColumbusUSA
  2. 2.Department of PsychologyUniversity of PennsylvaniaPhiladelphiaUSA
  3. 3.Chalmers P. Wylie VA Ambulatory Care CenterColumbusUSA

Personalised recommendations