Observational Coding Training Methods for CBT Treatment Fidelity: A Systematic Review

Original Article


Observational coding is the gold standard for coding treatment fidelity in Cognitive Behavioral Therapy (CBT). This method allows researchers to objectively assess therapist behaviors to determine if CBT was delivered as intended. However, there is no standard method or protocol that ensures reliability within and between research groups, compromising comparability between studies. The aim of this study was to evaluate procedures used to train raters to quantify CBT fidelity. A systematic review was conducted. PsycINFO and Web of Science databases were reviewed for articles up until July 2015; the parameters of the search included CBT, depression, treatment fidelity, and coding/rating. A total of 521 articles were screened for inclusion, which yielded 81 articles. Data was extracted across four categories (study characteristics, training process, inter-rater reliability, and rating), and 62 codes. The results revealed variability in training approaches (e.g., 25.9% of articles reported 10–50 h of training). Very little information was provided about the training process (i.e. only one study specified training duration). Training materials were: manuals (21.0%), videos (18.5%), audiotapes (2.4%), readings (1.2%), and lectures (1.2%). Almost all (90.1%) studies did not describe the trainer, and 28.4% did not describe raters. The relation between CBT fidelity and clinical outcomes is equivocal, perhaps because intra-team and inter-team reliability is difficult to achieve. This systematic review revealed variability in training others to rate CBT fidelity, of which some approaches might inform the field for future use.


Systematic review Observational coding Training Fidelity CBT 



We would like to acknowledge the following research assistants for their hard work in coding articles for this study: Paige Schultz, Madison Walker, Kayne Mettert, and Jasmine Shaffer.

Compliance with Ethical Standards

Conflict of Interest

Natalie Rodriguez-Quintana and Cara C. Lewis declare that they have no conflict of interest.

Human and Animal Rights

This article does not contain any studies with human participants or animals performed by any of the authors.

Informed Consent

All procedures followed were in accordance with the ethical standards of the responsible committee on human experimentation (national and institutional). Informed consent was obtained from all individual subjects participating in the study.

Supplementary material

10608_2018_9898_MOESM1_ESM.docx (61 kb)
Supplementary material 1 (DOCX 60 KB)
10608_2018_9898_MOESM2_ESM.docx (102 kb)
Supplementary material 2 (DOCX 102 KB)
10608_2018_9898_MOESM3_ESM.docx (22 kb)
Supplementary material 3 (DOCX 22 KB)


  1. Bakeman, R., & Gottman, J. M. (1987). Applying observational methods: A systematic view. In J. D. Osofsky (Ed.), Handbook of infant development (2nd edn., pp. 818–854). New York: Wiley.Google Scholar
  2. Baldwin, S. A., & Imel, Z. E. (2013). Therapist Effects: Findings and Methods. In B. A. Garfield (Ed.), Handbook of psychotherapy and behavior change (6th edn., pp. 258–297). New York: Wiley.Google Scholar
  3. Beck, J. S. (2011). Cognitive behavior therapy: Basics and beyond. New York: Guilford Press.Google Scholar
  4. Blackburn, I. -M., James, I. A., Milne, D. L., Baker, C., Standart, S., Garland, A., & Reichelt, F. K. (2001). The revised cognitive therapy scale (CTS-R): Psychometric properties. Behavioural and Cognitive Psychotherapy, 29, 431–446.  https://doi.org/10.1017/S1352465801004040.CrossRefGoogle Scholar
  5. Borrelli, B. (2011). The assessment, monitoring, and enhancement of treatment fidelity in public health clinical trials. Journal of Public Health Dentistry, 71, S52-S63.  https://doi.org/10.1111/j.1752-7325.2011.00233.x.CrossRefPubMedPubMedCentralGoogle Scholar
  6. Brauhardt, A., de Zwaan, M., Herpertz, S., Zipfel, S., Svaldi, J., Friederich, H. -C., & Hilbert, A. (2014). Therapist adherence in individual cognitive-behavioral therapy for binge-eating disorder: Assessment, course, and predictors. Behaviour Research and Therapy, 61, 55–60.  https://doi.org/10.1016/j.brat.2014.07.014.CrossRefPubMedGoogle Scholar
  7. Castorr, A. H., Thompson, K. O., Ryan, J. W., Phillips, C. Y., Prescott, P. A., & Soeken, K. L. (1990). The process of rater training for observational instruments: Implications for interrater reliability. Research in Nursing & Health, 13, 311–318.  https://doi.org/10.1002/nur.4770130507.CrossRefGoogle Scholar
  8. Cissna, K. N., Garvin, B. J., & Kennedy, C. W. (1990). Reliability in coding social interaction: A study of confirmation. Communication Reports, 3, 58–69.  https://doi.org/10.1080/08934219009367505.CrossRefGoogle Scholar
  9. Cohen, L. L., Manimala, R., & Blount, R. L. (2000). Easier said than done: What parents say they do and what they do during children’s immunizations. Children’s Health Care, 29, 79–86.  https://doi.org/10.1207/S15326888CHC2902_1.CrossRefGoogle Scholar
  10. Corace, K. M., Srigley, J. A., Hargadon, D. P., Yu, D., MacDonald, T. K., Fabrigar, L. R., & Garber, G. E. (2016). Using behavior change frameworks to improve healthcare worker influenza vaccination rates: A systematic review. Vaccine, 34, 3235–3242.  https://doi.org/10.1016/j.vaccine.2016.04.071.CrossRefPubMedGoogle Scholar
  11. Crits-Christoph, P., Siqueland, L., Blaine, J., Frank, A., Luborsky, L., Onken, L. S., … Gastfriend, D. R. (1999). Psychosocial treatments for cocaine dependence: National Institute on Drug Abuse collaborative cocaine treatment study. Archives of General Psychiatry, 56, 493–502.  https://doi.org/10.1001/pubs.ArchGenPsychiatry-ISSN-0003-990x-56-6-yoa8244.CrossRefPubMedGoogle Scholar
  12. Gaudiano, B. A. (2008). Cognitive-behavioral therapies: Achievements and challenges. Evidence-Based Mental Health, 11, 5–7.  https://doi.org/10.1136/ebmh.11.1.5.CrossRefPubMedPubMedCentralGoogle Scholar
  13. Gortner, E. T., Gollan, J. K., Dobson, K. S., & Jacobson, N. S. (1998). Cognitive–behavioral treatment for depression: Relapse prevention. Journal of Consulting and Clinical Psychology, 66, 377.CrossRefPubMedGoogle Scholar
  14. Guttman, H. A., Spector, R. M., Sigal, J. J., Rakoff, V., & Epstein, N. B. (1971). Reliability of coding affective communication in family therapy sessions: Problems of measurement and interpretation. Journal of Consulting and Clinical Psychology, 37, 397–402.  https://doi.org/10.1037/h0032003.CrossRefPubMedGoogle Scholar
  15. Hallgren, K. A. (2012). Computing inter-rater reliability for observational data: An overview and tutorial. Tutorials in Quantitative Methods for Psychology, 8, 23–34.  https://doi.org/10.20982/tqmp.08.1.p023.CrossRefPubMedPubMedCentralGoogle Scholar
  16. Horn, S. D., & Horn, R. A. (1986). Reliability and validity of the severity of illness index. Medical Care, 24, 159–178.CrossRefPubMedGoogle Scholar
  17. Hruschka, D. J., Schwartz, D., John, D. C. S., Picone-Decaro, E., Jenkins, R. A., & Carey, J. W. (2004). Reliability in coding open-ended data: Lessons learned from HIV behavioral research. Field Methods, 16, 307–331.  https://doi.org/10.1177/1525822X04266540.CrossRefGoogle Scholar
  18. Jacobson, N. S. (1998, July 21–26). How important are cognitive interventions in cognitive therapy? Paper presented at the World Congress of Behavioral and Cognitive Therapies, Acapulco, Mexico.Google Scholar
  19. Jacobson, N. S., Dobson, K. S., Truax, P. A., Addis, M. E., Koerner, K., Gollan, J. K., … Prince, S. E. (1996). A component analysis of cognitive-behavioral treatment for depression. Journal of Consulting and Clinical Psychology, 64, 295–304.  https://doi.org/10.1037/0022-006X.70.2.288.CrossRefPubMedGoogle Scholar
  20. Kazantzis, N. (2003). Therapist competence in cognitive-behavioural therapies: Review of the contemporary empirical evidence. Behaviour Change, 20, 1–12.  https://doi.org/10.1375/bech. Scholar
  21. Lewis, C. C., Scott, K. E., & Hendricks, K. E. (2014). A model and guide for evaluating supervision outcomes in cognitive–behavioral therapy-focused training programs. Training and Education in Professional Psychology, 8, 165–173.  https://doi.org/10.1037/tep0000029.CrossRefGoogle Scholar
  22. Maissi, E., Ridge, K., Treasure, J., Chalder, T., Roche, S., Bartlett, J., … Ismail, K. (2011). Nurse-led psychological interventions to improve diabetes control: Assessing competencies. Patient Education and Counseling, 84, e37-43.  https://doi.org/10.1016/j.pec.2010.07.036.CrossRefPubMedGoogle Scholar
  23. Markowitz, J. C., Kocsis, J. H., Fishman, B., Spielman, L. A., Jacobsberg, L. B., Frances, A. J., … Perry, S. W. (1998). Treatment of depressive symptoms in human immunodeficiency virus-positive patients. Archives of General Psychiatry, 55, 452–457.  https://doi.org/10.1001/archpsyc.55.5.452.CrossRefPubMedGoogle Scholar
  24. McLeod, B. D., & Weisz, J. R. (2005). The therapy process observational coding system-alliance scale: Measure characteristics and prediction of outcome in usual clinical practice. Journal of Consulting and Clinical Psychology, 73, 323–333.  https://doi.org/10.1037/0022-006X.73.2.323.CrossRefPubMedGoogle Scholar
  25. McManus, F., Rakovshik, S., Kennerley, H., Fennell, M., & Westbrook, D. (2012). An investigation of the accuracy of therapists’ self-assessment of cognitive-behaviour therapy skills. British Journal of Clinical Psychology, 51, 292–306.  https://doi.org/10.1111/j.2044-8260.2011.02028.x.CrossRefPubMedGoogle Scholar
  26. Muse, K., & McManus, F. (2013). A systematic review of methods for assessing competence in cognitive–behavioural therapy. Clinical Psychology Review, 33, 484–499.  https://doi.org/10.1016/j.cpr.2013.01.010.CrossRefPubMedGoogle Scholar
  27. Perepletchikova, F., Treat, T. A., & Kazdin, A. E. (2007). Treatment integrity in psychotherapy research: Analysis of the studies and examination of the associated factors. Journal of Consulting and Clinical Psychology, 75, 829–841.  https://doi.org/10.1037/0022-006X.75.6.829.CrossRefPubMedGoogle Scholar
  28. Plumb, J. C., & Vilardaga, R. (2010). Assessing treatment integrity in acceptance and commitment therapy: Strategies and suggestions. International Journal of Behavioral Consultation and Therapy, 6, 263–295.  https://doi.org/10.1037/h0100912.CrossRefGoogle Scholar
  29. Rozek, D. C., Serrano, J. L., Marriott, B., Hickman, L. B., Lewis, C. C., Brothers, B. M., & Simons, A. D. (2017). Cognitive Behavioral Therapy competency: Pilot data from a comparison of multiple perspectives. Behavioural and Cognitive Psychotherapy.  https://doi.org/10.1017/S1352465817000662 PubMedGoogle Scholar
  30. Samoilov, A., Goldfried, M. R., & Shapiro, D. A. (2000). Coding system of therapeutic focus on action and insight. Journal of Consulting and Clinical Psychology, 68, 513–514.  https://doi.org/10.1037/0022-006X.68.3.513.CrossRefPubMedGoogle Scholar
  31. Sheidow, A. J., Donohue, B. C., Hill, H. H., Henggeler, S. W., & Ford, J. D. (2008). Development of an audiotape review system for supporting adherence to an evidence-based treatment. Professional Psychology: Research and Practice, 39, 553–560.  https://doi.org/10.1037/0735-7028.39.5.553.CrossRefGoogle Scholar
  32. Simons, A. D., Padesky, C. A., Montemarano, J., Lewis, C. C., Murakami, J., Lamb, K., … Beck, A. T. (2010). Training and dissemination of cognitive behavior therapy for depression in adults: A preliminary examination of therapist competence and client outcomes. Journal of Consulting and Clinical Psychology, 78, 751–756.  https://doi.org/10.1037/a0020569.CrossRefPubMedGoogle Scholar
  33. Simons, A. D., Rozek, D. C., & Serrano, J. L. (2013). Wanted: Reliable and valid measures for the science of cognitive behavioral therapy dissemination and implementation. Clinical Psychology: Science and Practice, 20, 181–194.  https://doi.org/10.1111/cpsp.12033.Google Scholar
  34. Strunk, D. R., Brotman, M. A., & DeRubeis, R. J. (2010). The process of change in cognitive therapy for depression: Predictors of early inter-session symptom gains. Behaviour Research and Therapy, 48, 599–606.  https://doi.org/10.1016/j.brat.2010.03.011.CrossRefPubMedPubMedCentralGoogle Scholar
  35. Vallis, T. M., Shaw, B. F., & Dobson, K. S. (1986). The Cognitive Therapy Scale: Psychometric properties. Journal of Consulting and Clinical Psychology, 54, 381–385.  https://doi.org/10.1037/0022-006X.54.3.381.CrossRefPubMedGoogle Scholar
  36. Waltz, J., Addis, M. E., Koerner, K., & Jacobson, N. S. (1993). Testing the integrity of a psychotherapy protocol: Assessment of adherence and competence. Journal of Consulting and Clinical Psychology, 61, 620–630.  https://doi.org/10.1037/0022-006X.61.4.620.CrossRefPubMedGoogle Scholar
  37. Webb, C. A., DeRubeis, R. J., & Barber, J. P. (2010). Therapist adherence/competence and treatment outcome: A meta-analytic review. Journal of Consulting and Clinical Psychology, 78, 200–211.  https://doi.org/10.1037/a0018912.CrossRefPubMedPubMedCentralGoogle Scholar
  38. Weck, F., Bohn, C., Ginzburg, D. M., & Stangier, U. (2011a). Assessment of adherence and competence in cognitive therapy: Comparing session segments with entire sessions. Psychotherapy Research, 21, 658–669.  https://doi.org/10.1080/10503307.2011.602751.CrossRefPubMedGoogle Scholar
  39. Weck, F., Hilling, C., Schermelleh-Engel, K., Rudari, V., & Stangier, U. (2011b). Reliability of adherence and competence assessment in cognitive behavioral therapy: Influence of clinical experience. The Journal of Nervous and Mental Disease, 199, 276–279.  https://doi.org/10.1037/adb0000216.CrossRefPubMedGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2018

Authors and Affiliations

  1. 1.Department of Psychological and Brain SciencesIndiana UniversityBloomingtonUSA
  2. 2.Kaiser Permanente Washington Health Research InstituteMacColl Center for Health Care InnovationSeattleUSA
  3. 3.Department of Psychiatry & Behavioral SciencesUniversity of WashingtonSeattleUSA

Personalised recommendations