Abstract
Observational coding is the gold standard for coding treatment fidelity in Cognitive Behavioral Therapy (CBT). This method allows researchers to objectively assess therapist behaviors to determine if CBT was delivered as intended. However, there is no standard method or protocol that ensures reliability within and between research groups, compromising comparability between studies. The aim of this study was to evaluate procedures used to train raters to quantify CBT fidelity. A systematic review was conducted. PsycINFO and Web of Science databases were reviewed for articles up until July 2015; the parameters of the search included CBT, depression, treatment fidelity, and coding/rating. A total of 521 articles were screened for inclusion, which yielded 81 articles. Data was extracted across four categories (study characteristics, training process, inter-rater reliability, and rating), and 62 codes. The results revealed variability in training approaches (e.g., 25.9% of articles reported 10–50 h of training). Very little information was provided about the training process (i.e. only one study specified training duration). Training materials were: manuals (21.0%), videos (18.5%), audiotapes (2.4%), readings (1.2%), and lectures (1.2%). Almost all (90.1%) studies did not describe the trainer, and 28.4% did not describe raters. The relation between CBT fidelity and clinical outcomes is equivocal, perhaps because intra-team and inter-team reliability is difficult to achieve. This systematic review revealed variability in training others to rate CBT fidelity, of which some approaches might inform the field for future use.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Bakeman, R., & Gottman, J. M. (1987). Applying observational methods: A systematic view. In J. D. Osofsky (Ed.), Handbook of infant development (2nd edn., pp. 818–854). New York: Wiley.
Baldwin, S. A., & Imel, Z. E. (2013). Therapist Effects: Findings and Methods. In B. A. Garfield (Ed.), Handbook of psychotherapy and behavior change (6th edn., pp. 258–297). New York: Wiley.
Beck, J. S. (2011). Cognitive behavior therapy: Basics and beyond. New York: Guilford Press.
Blackburn, I. -M., James, I. A., Milne, D. L., Baker, C., Standart, S., Garland, A., & Reichelt, F. K. (2001). The revised cognitive therapy scale (CTS-R): Psychometric properties. Behavioural and Cognitive Psychotherapy, 29, 431–446. https://doi.org/10.1017/S1352465801004040.
Borrelli, B. (2011). The assessment, monitoring, and enhancement of treatment fidelity in public health clinical trials. Journal of Public Health Dentistry, 71, S52-S63. https://doi.org/10.1111/j.1752-7325.2011.00233.x.
Brauhardt, A., de Zwaan, M., Herpertz, S., Zipfel, S., Svaldi, J., Friederich, H. -C., & Hilbert, A. (2014). Therapist adherence in individual cognitive-behavioral therapy for binge-eating disorder: Assessment, course, and predictors. Behaviour Research and Therapy, 61, 55–60. https://doi.org/10.1016/j.brat.2014.07.014.
Castorr, A. H., Thompson, K. O., Ryan, J. W., Phillips, C. Y., Prescott, P. A., & Soeken, K. L. (1990). The process of rater training for observational instruments: Implications for interrater reliability. Research in Nursing & Health, 13, 311–318. https://doi.org/10.1002/nur.4770130507.
Cissna, K. N., Garvin, B. J., & Kennedy, C. W. (1990). Reliability in coding social interaction: A study of confirmation. Communication Reports, 3, 58–69. https://doi.org/10.1080/08934219009367505.
Cohen, L. L., Manimala, R., & Blount, R. L. (2000). Easier said than done: What parents say they do and what they do during children’s immunizations. Children’s Health Care, 29, 79–86. https://doi.org/10.1207/S15326888CHC2902_1.
Corace, K. M., Srigley, J. A., Hargadon, D. P., Yu, D., MacDonald, T. K., Fabrigar, L. R., & Garber, G. E. (2016). Using behavior change frameworks to improve healthcare worker influenza vaccination rates: A systematic review. Vaccine, 34, 3235–3242. https://doi.org/10.1016/j.vaccine.2016.04.071.
Crits-Christoph, P., Siqueland, L., Blaine, J., Frank, A., Luborsky, L., Onken, L. S., … Gastfriend, D. R. (1999). Psychosocial treatments for cocaine dependence: National Institute on Drug Abuse collaborative cocaine treatment study. Archives of General Psychiatry, 56, 493–502. https://doi.org/10.1001/pubs.ArchGenPsychiatry-ISSN-0003-990x-56-6-yoa8244.
Gaudiano, B. A. (2008). Cognitive-behavioral therapies: Achievements and challenges. Evidence-Based Mental Health, 11, 5–7. https://doi.org/10.1136/ebmh.11.1.5.
Gortner, E. T., Gollan, J. K., Dobson, K. S., & Jacobson, N. S. (1998). Cognitive–behavioral treatment for depression: Relapse prevention. Journal of Consulting and Clinical Psychology, 66, 377.
Guttman, H. A., Spector, R. M., Sigal, J. J., Rakoff, V., & Epstein, N. B. (1971). Reliability of coding affective communication in family therapy sessions: Problems of measurement and interpretation. Journal of Consulting and Clinical Psychology, 37, 397–402. https://doi.org/10.1037/h0032003.
Hallgren, K. A. (2012). Computing inter-rater reliability for observational data: An overview and tutorial. Tutorials in Quantitative Methods for Psychology, 8, 23–34. https://doi.org/10.20982/tqmp.08.1.p023.
Horn, S. D., & Horn, R. A. (1986). Reliability and validity of the severity of illness index. Medical Care, 24, 159–178.
Hruschka, D. J., Schwartz, D., John, D. C. S., Picone-Decaro, E., Jenkins, R. A., & Carey, J. W. (2004). Reliability in coding open-ended data: Lessons learned from HIV behavioral research. Field Methods, 16, 307–331. https://doi.org/10.1177/1525822X04266540.
Jacobson, N. S. (1998, July 21–26). How important are cognitive interventions in cognitive therapy? Paper presented at the World Congress of Behavioral and Cognitive Therapies, Acapulco, Mexico.
Jacobson, N. S., Dobson, K. S., Truax, P. A., Addis, M. E., Koerner, K., Gollan, J. K., … Prince, S. E. (1996). A component analysis of cognitive-behavioral treatment for depression. Journal of Consulting and Clinical Psychology, 64, 295–304. https://doi.org/10.1037/0022-006X.70.2.288.
Kazantzis, N. (2003). Therapist competence in cognitive-behavioural therapies: Review of the contemporary empirical evidence. Behaviour Change, 20, 1–12. https://doi.org/10.1375/bech.20.1.1.24845.
Lewis, C. C., Scott, K. E., & Hendricks, K. E. (2014). A model and guide for evaluating supervision outcomes in cognitive–behavioral therapy-focused training programs. Training and Education in Professional Psychology, 8, 165–173. https://doi.org/10.1037/tep0000029.
Maissi, E., Ridge, K., Treasure, J., Chalder, T., Roche, S., Bartlett, J., … Ismail, K. (2011). Nurse-led psychological interventions to improve diabetes control: Assessing competencies. Patient Education and Counseling, 84, e37-43. https://doi.org/10.1016/j.pec.2010.07.036.
Markowitz, J. C., Kocsis, J. H., Fishman, B., Spielman, L. A., Jacobsberg, L. B., Frances, A. J., … Perry, S. W. (1998). Treatment of depressive symptoms in human immunodeficiency virus-positive patients. Archives of General Psychiatry, 55, 452–457. https://doi.org/10.1001/archpsyc.55.5.452.
McLeod, B. D., & Weisz, J. R. (2005). The therapy process observational coding system-alliance scale: Measure characteristics and prediction of outcome in usual clinical practice. Journal of Consulting and Clinical Psychology, 73, 323–333. https://doi.org/10.1037/0022-006X.73.2.323.
McManus, F., Rakovshik, S., Kennerley, H., Fennell, M., & Westbrook, D. (2012). An investigation of the accuracy of therapists’ self-assessment of cognitive-behaviour therapy skills. British Journal of Clinical Psychology, 51, 292–306. https://doi.org/10.1111/j.2044-8260.2011.02028.x.
Muse, K., & McManus, F. (2013). A systematic review of methods for assessing competence in cognitive–behavioural therapy. Clinical Psychology Review, 33, 484–499. https://doi.org/10.1016/j.cpr.2013.01.010.
Perepletchikova, F., Treat, T. A., & Kazdin, A. E. (2007). Treatment integrity in psychotherapy research: Analysis of the studies and examination of the associated factors. Journal of Consulting and Clinical Psychology, 75, 829–841. https://doi.org/10.1037/0022-006X.75.6.829.
Plumb, J. C., & Vilardaga, R. (2010). Assessing treatment integrity in acceptance and commitment therapy: Strategies and suggestions. International Journal of Behavioral Consultation and Therapy, 6, 263–295. https://doi.org/10.1037/h0100912.
Rozek, D. C., Serrano, J. L., Marriott, B., Hickman, L. B., Lewis, C. C., Brothers, B. M., & Simons, A. D. (2017). Cognitive Behavioral Therapy competency: Pilot data from a comparison of multiple perspectives. Behavioural and Cognitive Psychotherapy. https://doi.org/10.1017/S1352465817000662
Samoilov, A., Goldfried, M. R., & Shapiro, D. A. (2000). Coding system of therapeutic focus on action and insight. Journal of Consulting and Clinical Psychology, 68, 513–514. https://doi.org/10.1037/0022-006X.68.3.513.
Sheidow, A. J., Donohue, B. C., Hill, H. H., Henggeler, S. W., & Ford, J. D. (2008). Development of an audiotape review system for supporting adherence to an evidence-based treatment. Professional Psychology: Research and Practice, 39, 553–560. https://doi.org/10.1037/0735-7028.39.5.553.
Simons, A. D., Padesky, C. A., Montemarano, J., Lewis, C. C., Murakami, J., Lamb, K., … Beck, A. T. (2010). Training and dissemination of cognitive behavior therapy for depression in adults: A preliminary examination of therapist competence and client outcomes. Journal of Consulting and Clinical Psychology, 78, 751–756. https://doi.org/10.1037/a0020569.
Simons, A. D., Rozek, D. C., & Serrano, J. L. (2013). Wanted: Reliable and valid measures for the science of cognitive behavioral therapy dissemination and implementation. Clinical Psychology: Science and Practice, 20, 181–194. https://doi.org/10.1111/cpsp.12033.
Strunk, D. R., Brotman, M. A., & DeRubeis, R. J. (2010). The process of change in cognitive therapy for depression: Predictors of early inter-session symptom gains. Behaviour Research and Therapy, 48, 599–606. https://doi.org/10.1016/j.brat.2010.03.011.
Vallis, T. M., Shaw, B. F., & Dobson, K. S. (1986). The Cognitive Therapy Scale: Psychometric properties. Journal of Consulting and Clinical Psychology, 54, 381–385. https://doi.org/10.1037/0022-006X.54.3.381.
Waltz, J., Addis, M. E., Koerner, K., & Jacobson, N. S. (1993). Testing the integrity of a psychotherapy protocol: Assessment of adherence and competence. Journal of Consulting and Clinical Psychology, 61, 620–630. https://doi.org/10.1037/0022-006X.61.4.620.
Webb, C. A., DeRubeis, R. J., & Barber, J. P. (2010). Therapist adherence/competence and treatment outcome: A meta-analytic review. Journal of Consulting and Clinical Psychology, 78, 200–211. https://doi.org/10.1037/a0018912.
Weck, F., Bohn, C., Ginzburg, D. M., & Stangier, U. (2011a). Assessment of adherence and competence in cognitive therapy: Comparing session segments with entire sessions. Psychotherapy Research, 21, 658–669. https://doi.org/10.1080/10503307.2011.602751.
Weck, F., Hilling, C., Schermelleh-Engel, K., Rudari, V., & Stangier, U. (2011b). Reliability of adherence and competence assessment in cognitive behavioral therapy: Influence of clinical experience. The Journal of Nervous and Mental Disease, 199, 276–279. https://doi.org/10.1037/adb0000216.
Acknowledgements
We would like to acknowledge the following research assistants for their hard work in coding articles for this study: Paige Schultz, Madison Walker, Kayne Mettert, and Jasmine Shaffer.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of Interest
Natalie Rodriguez-Quintana and Cara C. Lewis declare that they have no conflict of interest.
Human and Animal Rights
This article does not contain any studies with human participants or animals performed by any of the authors.
Informed Consent
All procedures followed were in accordance with the ethical standards of the responsible committee on human experimentation (national and institutional). Informed consent was obtained from all individual subjects participating in the study.
Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
About this article
Cite this article
Rodriguez-Quintana, N., Lewis, C.C. Observational Coding Training Methods for CBT Treatment Fidelity: A Systematic Review. Cogn Ther Res 42, 358–368 (2018). https://doi.org/10.1007/s10608-018-9898-5
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10608-018-9898-5