Skip to main content
SpringerLink
Log in

How Reliable Are Therapeutic Competence Ratings? Results of a Systematic Review and Meta-Analysis

  • Original Article
  • Published:
Cognitive Therapy and Research Aims and scope Submit manuscript

Abstract

Assessments of psychotherapeutic competencies play a crucial role in research and training. However, research on the reliability and validity of such assessments is sparse. This study aimed to provide an overview of the current evidence and to provide an average interrater reliability (IRR) of psychotherapeutic competence ratings. A systematic review was conducted, and 20 studies reported in 32 publications were collected. These 20 studies were included in a narrative synthesis, and 20 coefficients were entered into the meta-analysis. Most primary studies referred to cognitive-behavioral therapies and the treatment of depression, used the Cognitive Therapy Scale, based ratings on videos, and trained the raters. Our meta-analysis revealed a pooled ICC of 0.82, but at the same time severe heterogeneity. The evidence map highlighted a variety of variables related to competence assessments. Further aspects influencing the reliability of competence ratings and regarding the considerable heterogeneity are discussed in detail throughout the manuscript.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

References

References marked with an asterisk indicate studies included in the qualitative summary

  • American Psychiatric Association (APA). (2013). Diagnostic and statistical manual of mental disorders (5th ed.). Arlington VA: American Psychiatric Association.

    Book  Google Scholar 

  • American Psychiatric Association (APA). (2017). What is psychotherapy? Retrieved from https://www.apa.org/ptsd-guideline/patients-and-families/psychotherapy.

  • * Barber, J. P., & Crits-Christoph, P. (1996). Development of a therapist adherence/competence rating scale for supportive-expressive dynamic psychotherapy: A preliminary report. Psychotherapy Research, 6(2), 81–94. https://doi.org/10.1080/10503309612331331608.

  • * Barber, J. P., Crits-Christoph, P., & Luborsky, L. (1996). Effects of therapist adherence and competence on patient outcome in brief dynamic therapy. Journal of Consulting and Clinical Psychology, 64(3), 619–622. https://doi.org/10.1037/0022-006x.64.3.619.

  • * Barber, J. P., Foltz, C., Crits-Christoph, P., & Chittams, J. (2004). Therapists’ adherence and competence and treatment discrimination in the NIDA Collaborative Cocaine Treatment Study. Journal of Clinical Psychology, 60(1), 29–41. https://doi.org/10.1002/jclp.10186

  • * Barber, J. P., Krakauer, I., Calvo, N., & Badgio, P. C. (1997). Measuring adherence and competence of dynamic therapists in the treatment of cocaine dependence. Journal of Psychotherapy Practice & Research, 6(1), 12–24.

  • * Barber, J. P., Liese, B. S., & Abrams, M. J. (2003). Development of the cognitive therapy adherence and competence scale. Psychotherapy Research, 13(2), 205–221.

  • Barber, J. P., Sharpless, B. A., Klostermann, S., & McCarthy, K. S. (2007). Assessing intervention competence and its relation to therapy outcome: A selected review derived from the outcome literature. Professional Psychology: Research and Practice,38, 493–500. https://doi.org/10.1037/0735-7028.38.5.493.

    Article  Google Scholar 

  • Baujat, B., Mahé, C., Pignon, J. P., & Hill, C. (2002). A graphical method for exploring heterogeneity in meta-analyses: Application to a meta-analysis of 65 trials. Statistics in Medicine,21(18), 2641–2652.

    Article  Google Scholar 

  • Beck Institute for Cognitive Behavior Therapy. (2019, October 10). Cognitive Therapy Rating Scale (CTRS). Retrieved from https://beckinstitute.org/wp-content/uploads/2017/03/CTRS-Scale-and-Score-Report-2016.pdf.

  • * Blackburn, I. M., James, I. A., Milne, D. L., Baker, C., Standart, S., Garland, A., & Reichelt, F. K. (2001). The revised cognitive therapy scale (CTS-R): Psychometric properties. Behavioural and cognitive psychotherapy, 29(4), 431–446.

  • Borenstein, M., Hedges, L. V., Higgins, J. P., & Rothstein, H. R. (2009). Introduction to meta-analysis. Hoboken: Wiley.

    Book  Google Scholar 

  • * Brueck, R. K., Frick, K., Loessl, B., Kriston, L., Schondelmaier, S., Go, C.,…, Berner, M. (2009). Psychometric properties of the German version of the motivational interviewing treatment integrity code. Journal of Substance Abuse Treatment, 36(1), 44–48. https://doi.org/10.1016/j.jsat.2008.04.004.

  • * Chevron, E. S., & Rounsaville, B. J. (1983). Evaluating the clinical skills of psychotherapists: A comparison of techniques. Archives of General Psychiatry, 40(10), 1129–1132. https://doi.org/10.1001/archpsyc.1983.01790090091014.

  • * Dennhag, I., Gibbons, M. B. C., Barber, J. P., Gallop, R., & Crits-Christoph, P. (2012a). Do supervisors and independent judges agree on evaluations of therapist adherence and competence in the treatment of cocaine dependence? Psychotherapy Research, 22(6), 720–730. https://doi.org/10.1080/10503307.2012.716528.

  • * Dennhag, I., Gibbons, M. B. C., Barber, J. P., Gallop, R., & Crits-Christoph, P. (2012b). How many treatment sessions and patients are needed to create a stable score of adherence and competence in the treatment of cocaine dependence? Psychotherapy Research, 22(4), 475–488. https://doi.org/10.1080/10503307.2012.674790.

  • * Dittmann, C., Müller‐Engelmann, M., Stangier, U., Priebe, K., Fydrich, T., Görg, N., …, Steil, R. (2017). Disorder‐ and treatment‐specific therapeutic competence scales for posttraumatic stress disorder intervention: Development and psychometric properties. Journal of Traumatic Stress. https://doi.org/10.1002/jts.22236.

  • * Dobson, K. S., Shaw, B. F., & Vallis, T. M. (1985). Reliability of a measure of the quality of cognitive therapy. British Journal of Clinical Psychology, 24(4), 295–300. https://doi.org/10.1111/j.2044-8260.1985.tb00662.x.

  • Duffy, L., Gajree, S., Langhorne, P., Stott, D. J., & Quinn, T. J. (2013). Reliability (inter-rater agreement) of the Barthel Index for assessment of stroke survivors: Systematic review and meta-analysis. Stroke,44(2), 462–468. https://doi.org/10.1161/strokeaha.112.678615.

    Article  PubMed  Google Scholar 

  • Egger, M., Davey Smith, G., Schneider, M., & Minder, C. (1997). Bias in meta-analysis detected by a simple, graphical test. British Medical Journal Open,315, 629–634.

    Article  CAS  Google Scholar 

  • Fairburn, C. G., & Cooper, Z. (2011). Therapist competence, therapy quality, and therapist training. Behaviour Research and Therapy,49, 373–378. https://doi.org/10.1016/j.brat.2011.03.005.

    Article  PubMed  PubMed Central  Google Scholar 

  • Fisher, Z., & Tipton, E. (2015). robumeta: An R-package for robust variance estimation in meta-analysis. http://arxiv.org/abs/1503.02220.

  • Hempel, S., Miles, J. N., Booth, M. J., Wang, Z., Morton, S. C., & Shekelle, P. G. (2013). Risk of bias: A simulation study of power to detect study-level moderator effects in meta-analysis. Systematic Reviews,2, 107. https://doi.org/10.1186/2046-4053-2-107.

    Article  PubMed  PubMed Central  Google Scholar 

  • Higgins, J., & Green, S. (2011). Cochrane Handbook for Systematic Reviews of Interventions Version 5.1. 0 [updated March 2011]. The Cochrane Collaboration, 2011. Retrieved August, 29 from www.cochrane-handbook.org.

  • * Hoffart, A., Sexton, H., Nordahl, H. M., & Stiles, T. C. (2005). Connection between patient and therapist and therapist’s competence in schema-focused therapy of personality problems. Psychotherapy Research, 15(4), 409–419. https://doi.org/10.1080/10503300500091702.

  • * Karterud, S., Pedersen, G., Engen, M., Johansen, M. S., Johansson, P. N., Schlüter, C., …, Bateman, A. W. (2013). The MBT Adherence and Competence Scale (MBT-ACS): Development, structure and reliability. Psychotherapy Research, 23(6), 705–717. https://doi.org/10.1080/10503307.2012.708795.

  • Kazantzis, N. (2003). Therapist competence in cognitive-behavioural therapies: Review of the contemporary empirical evidence. Behaviour Change,20(1), 1–12. https://doi.org/10.1375/bech.20.1.1.24845.

    Article  Google Scholar 

  • * Kazantzis, N., Clayton, X., Cronin, T. J., Farchione, D., Limburg, K., & Dobson, K. S. (2018). The Cognitive Therapy Scale and Cognitive Therapy Scale-Revised as measures of therapist competence in cognitive behavior therapy for depression: Relations with short and long term outcome. Cognitive Therapy and Research, 42(4), 385–397. https://doi.org/10.1007/s10608-018-9919-4.

  • Koo, T. K., & Li, M. Y. (2016). A guideline of selecting and reporting intraclass correlation coefficients for reliability research. Journal of Chiropractic Medicine,15(2), 155–163. https://doi.org/10.1016/j.jcm.2016.02.012.

    Article  PubMed  PubMed Central  Google Scholar 

  • Kottner, J., Audigé, L., Brorson, S., Donner, A., Gajeweski, B. J., Hróbjartsson, A., et al. (2011). Guidelines for reporting reliability and agreement studies (GRRAS) were proposed. International Journal of Nursing Studies,48, 661–671. https://doi.org/10.1016/j.ijnurstu.2011.01.016.

    Article  PubMed  Google Scholar 

  • Kuyken, W., & Tsivrikos, D. (2009). Therapist competence, comorbidity and cognitive-behavioral therapy for depression. Psychotherapy and Psychosomatics,78(1), 42–48. https://doi.org/10.1159/000172619.

    Article  PubMed  Google Scholar 

  • Machmutow, K., Holtforth, M. G., Krieger, T., & Watzke, B. (2018). Identifying relapse prevention elements during psychological treatment of depression: Development of an observer-based rating instrument. Journal of Affective Disorders,227, 358–365. https://doi.org/10.1016/j.jad.2017.11.009.

    Article  PubMed  Google Scholar 

  • * McGrath, K. B. (2013). Validation of the Drexel University ACT/tCBT Adherence and Competence Rating Scale: Revised for use in a clinical population. Doctoral dissertation. Retrieved from https://idea.library.drexel.edu/islandora/object/idea%3A3803.

  • Meister, R., Jansen, A., Härter, M., Nestoriuc, Y., & Kriston, L. (2017). Placebo and nocebo reactions in randomized trials of pharmacological treatments for persistent depressive disorder. A meta-regression analysis. Journal of Affective Disorders,215, 288–298.

    Article  CAS  Google Scholar 

  • Moher, D., Liberati, A., Tetzlaff, J., & Altman, D. G. (2009). Preferred reporting items for systematic reviews and meta-analyses: The PRISMA statement. PLoS Med,6(7), e1000097. https://doi.org/10.1371/journal.pmed.1000097.

    Article  PubMed  PubMed Central  Google Scholar 

  • Muse, K., & McManus, F. (2013). A systematic review of methods for assessing competence in cognitive–behavioural therapy. Clinical Psychology Review,33(3), 484–499. https://doi.org/10.1016/j.cpr.2013.01.010.

    Article  PubMed  Google Scholar 

  • Muse, K., & McManus, F. (2016). Expert insight into the assessment of competence in cognitive-behavioural therapy: A qualitative exploration of experts’ experiences, opinions and recommendations. Clinical Psychology and Psychotherapy,23, 246–259. https://doi.org/10.1002/cpp.1952.

    Article  PubMed  Google Scholar 

  • Muse, K., McManus, F., Rakovshik, S., & Thwaites, R. (2017). Development and psychometric evaluation of the Assessment of Core CBT Skills (ACCS): An observation-based tool for assessing cognitive behavioral therapy competence. Psychological Assessment,29(5), 542–555. https://doi.org/10.1037/pas0000372.

    Article  PubMed  Google Scholar 

  • Portney, L. G., & Watkins, M. P. (2009). Foundations of clinical research: Applications to practice. Upper Saddle River, NJ: Pearson/Prentice Hall.

    Google Scholar 

  • Quintana, D. S. (2015). From pre-registration to publication: A non-technical primer for conducting a meta-analysis to synthesize correlational data. Frontiers in Psychology,6, 1549. https://doi.org/10.3389/fpsyg.2015.01549.

    Article  PubMed  PubMed Central  Google Scholar 

  • R-Core-Team. (2018). R: A language and environment for statistical computing. R Foundation for Statistical Computing. Vienna, Austria. Retrieved from https://www.R-project.org/.

  • * Reichelt, F. K., James, I. A., & Blackburn, I. M. (2003). Impact of training on rating competence in cognitive therapy. Journal of Behavior Therapy and Experimental Psychiatry, 34(2), 87–99.

  • Roth, A. D., & Pilling, S. (2007). The competences required to deliver effective cognitive and behavioural therapy for people with depression and with anxiety disorders. Retrieved from https://www.ucl.ac.uk/pals/research/clinical-ed.

  • Santelmann, H., Franklin, J., Bußhoff, J., & Baethge, C. (2016). Interrater reliability of schizoaffective disorder compared with schizophrenia, bipolar disorder, and unipolar depression—A systematic review and meta-analysis. Schizophrenia Research,176(2), 357–363. https://doi.org/10.1016/j.schres.2016.07.012.

    Article  PubMed  Google Scholar 

  • * Schmidt, I. D., Strunk, D. R., DeRubeis, R. J., Conklin, L. R., & Braun, J. D. (2018). Revisiting how we assess therapist competence in cognitive therapy. Cognitive Therapy and Research, 42(4), 369–384. https://doi.org/10.1007/s10608-018-9908-7.

  • * Strunk, D. R., Brotman, M. A., DeRubeis, R. J., & Hollon, S. D. (2010). Therapist competence in cognitive therapy for depression: Predicting subsequent symptom change. J Consult Clin Psychol, 78(3), 429–437. https://doi.org/10.1037/a0019631.

  • * Soygüt, G., Uluç, S., & Tüzün, Z. (2008). A pilot study of the reliability and validity of the turkish cognitive therapy adherence and competence scale. Turkish Journal of Psychiatry, 19(2).

  • * Svartberg, M. (1989). Manualization and competence monitoring of short-term anxiety-provoking psychotherapy. Psychotherapy: Theory, Research, Practice, Training, 26(4), 564–571. https://doi.org/10.1037/h0085477.

  • * Tadic, M., Drapeau, M., Solai, S., de Roten, Y., & Despland, J. N. (2003). Development of a competence scale for brief psychodynamic investigation: A pilot study. Schweizer Archiv für Neurologie und Psychiatrie, 154(1), 28–35. https://doi.org/10.4414/sanp.2003.01333.

  • Trajković, G., Starčević, V., Latas, M., Leštarević, M., Ille, T., Bukumirić, Z., et al. (2011). Reliability of the Hamilton Rating Scale for depression: A meta-analysis over a period of 49years. Psychiatry Research,189(1), 1–9. https://doi.org/10.1016/j.psychres.2010.12.007.

    Article  PubMed  Google Scholar 

  • * Vallis, T. M., Shaw, B. F., & Dobson, K. S. (1986). The Cognitive Therapy Scale: Psychometric properties. Journal of Consulting and Clinical Psychology, 54(3), 381–385. https://doi.org/10.1037/0022-006x.54.3.381.

  • * Vallis, T. M., Shaw, B. F., & McCabe, S. B. (1988). The relationship between therapist competency and cognitive therapy and general therapy skill. Journal of Cognitive Psychotherapy: An International Quarterly, 2(4), 237–249.

  • Viechtbauer, W. (2010). Conducting meta-analyses in R with the metafor package. Journal of Statistical Software,36, 1–48.

    Article  Google Scholar 

  • * von Consbruch, K., Clark, D. M., & Stangier, U. (2012). Assessing therapeutic competence in cognitive therapy for social phobia: Psychometric properties of the Cognitive Therapy Competence Scale for Social Phobia (CTCS-SP). Behavioural and Cognitive Psychotherapy, 40(2), 149–161. https://doi.org/10.1017/s1352465811000622.

  • Waltz, J., Addis, M. E., Koerner, K., & Jacobson, N. S. (1993). Testing the integrity of a psychotherapy protocol: Assessment of adherence and competence. Journal of Consulting and Clinical Psychology,61(4), 620–630.

    Article  CAS  Google Scholar 

  • Warshaw, M. G., Dyck, I., Allsworth, J., Stout, R. L., & Keller, M. B. (2001). Maintaining reliability in a long-term psychiatric study: An ongoing inter-rater reliability monitoring program using the longitudinal interval follow-up evaluation. Journal of Psychiatric Research,35(5), 297–305.

    Article  CAS  Google Scholar 

  • Webb, C. A., DeRubeis, R. J., & Barber, J. P. (2010). Therapist adherence/competence and treatment outcome: A meta-analytic review. Journal of Consulting and Clinical Psychology,78, 200–211. https://doi.org/10.1037/a0018912.

    Article  PubMed  PubMed Central  Google Scholar 

  • * Weck, F., Bohn, C., Ginzburg, D. M., & Stangier, U. (2011). Assessment of adherence and competence in cognitive therapy: Comparing session segments with entire sessions. Psychotherapy Research, 21(6), 658–669. https://doi.org/10.1080/10503307.2011.602751.

  • * Weck, F., Grikscheit, F., Höfling, V., & Stangier, U. (2014). Assessing treatment integrity in cognitive-behavioral therapy: Comparing session segments with entire sessions. Behavior Therapy, 45(4), 541–552. https://doi.org/10.1016/j.beth.2014.03.003.

  • * Weck, F., Hautzinger, M., Heidenreich, T., & Stangier, U. (2011). Psychoedukation bei depressiven störungen -erfassung von interventionsmerkmalen und behandlungskompetenzen = Psychoeducation for depression - features of interventions and therapeutic competencies. PPmP: Psychotherapie Psychosomatik Medizinische Psychologie, 61(3-4), 148–153. https://doi.org/10.1055/s-0030-1269902.

  • * Weck, F., Hilling, C., Schermelleh-Engel, K., Rudari, V., & Stangier, U. (2011). Reliability of adherence and competence assessment in cognitive behavioral therapy: Influence of clinical experience. Journal of Nervous and Mental Disease, 199(4), 276–279. https://doi.org/10.1097/nmd.0b013e318212461.

  • * Weck, F., Weigel, M., Richtberg, S., & Stangier, U. (2011). Reliability of adherence and competence assessment in psychoeducational treatment: Influence of clinical experience. Journal of Nervous and Mental Disease, 199(12), 983–986. https://doi.org/10.1097/nmd.0b013e3182392da1.

  • WHO. (1992). The ICD-10 classification of mental and behavioural disorders: Clinical descriptions and diagnostic guidelines (6th ed.). Geneva: World Health Organization.

    Google Scholar 

  • Wickham, H., Francois, R., Henry, L., & Müller, K. (2015). dplyr: A grammar of data manipulation. R package version 0.4, 3.

  • Wirtz, M. A. (2017). Interrater Reliability. In V. Zeigler-Hill & T. K. Shackelford (Eds.), Encyclopedia of personality and individual differences (pp. 1–4). New York: Springer.

    Google Scholar 

  • Wirtz, M., & Caspar, F. (2002). Interrater agreement and interrater reliability. Göttingen: Hogrefe.

    Google Scholar 

  • * Wittorf, A., Jakobi-Malterre, U. E., Beulen, S., Bechdolf, A., Müller, B. W., Sartory, G., …, Klingberg, S. (2013). Associations between therapy skills and patient experiences of change processes in cognitive behavioral therapy for psychosis. Psychiatry Research, 210(3), 702–709. https://doi.org/10.1016/j.psychres.2013.08.01.

  • Young, J. E., & Beck, A. T. (1980). Cognitive Therapy Scale: Rating manual. Unpublished Manuscript, University of Pennsylvania, Philadelphia, PA.

    Google Scholar 

  • Zarafonitis-Müller, S., Kuhr, K., & Bechdorf, A. (2014). Der Zusammenhang der Therapeutenkompetenz und Adhärenz zum Therapieerfolg in der Kognitiven Verhaltenstherapie - metaanalytische Ergebnisse. Fortschritte der Neurologie-Psychiatrie,82, 502–510.

    Article  Google Scholar 

Download references

Acknowledgements

We would like to thank, Ricarda Löscher B.Sc. psych., for her assistance with study screening and data extraction.

Funding

No external funding.

Author information

Authors and Affiliations

  1. Department of Psychology, Clinical Psychology and Psychotherapy, University of Potsdam, Karl-Liebknecht-Str. 24-25, 14476, Potsdam, Germany

    Franziska Kühne, Ulrike Maaß, Tatjana Paunov & Florian Weck

  2. Department of Medical Psychology, University Medical Center Hamburg Eppendorf, Hamburg, Germany

    Ramona Meister

Authors
  1. Franziska Kühne
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ramona Meister
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ulrike Maaß
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Tatjana Paunov
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Florian Weck
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Franziska Kühne.

Ethics declarations

Conflict of interest

Florian Weck is an author of six of the publications included in the review. Franziska Kühne, Ramona Meister, Ulrike Maaß and Tatjana Paunov declare that they have no conflict of interest.

Ethical Approval

This article does not contain any studies involving human participants.

Research Involving Animal Rights

This article does not contain any studies with animals.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (PDF 30 kb)

Supplementary material 2 (PDF 4 kb)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Kühne, F., Meister, R., Maaß, U. et al. How Reliable Are Therapeutic Competence Ratings? Results of a Systematic Review and Meta-Analysis. Cogn Ther Res 44, 241–257 (2020). https://doi.org/10.1007/s10608-019-10056-5

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10608-019-10056-5

Keywords

Search

Navigation