Abstract
Treatment fidelity is an essential outcome of implementation research. The gold standard measure for Cognitive-Behavioral Therapy (CBT) fidelity is the Cognitive Therapy Rating Scale (CTRS). Despite its widespread use in research and training programs, the structure of the CTRS has not been examined in a sample of community mental health clinicians with adult and child clients. The current study addressed this gap. The sample consisted of 355 clinicians and 1298 CBT sessions scored using the CTRS. High interrater reliability was observed and factor analysis yielded separate structures for child and adult treatment sessions. These structures were not consistent with previous factor analyses conducted on the scale. Findings suggest that the CTRS is a reliable measure of CBT in community mental health settings but that its structure may depend on the clinical population measured. Additionally, the factor structure can provide guidance for delivering feedback in training and supervision settings.
Similar content being viewed by others
References
Aarons, G. A., Miller, E. A., Green, A. E., Perrott, J. A., & Bradway, R. (2012). Adaptation happens: A qualitative case study of implementation of the incredible years evidence-based parent training programme in a residential substance abuse treatment programme. Journal of Children’s Services, 7, 233–245.
Arbuckle, J. L. (2012). Amos (Version 21.0) [Computer Program]. Chicago: IBM SPSS.
Beavers, A. S., Lounsbury, J. W., Richards, J. K., Huck, S. W., Skolits, G. J., & Esquivel, S. L. (2013). Practical considerations for using exploratory factor analysis in educational research. Practical Assessment, Research & Evaluation, 18, 1–13.
Beck, A. T., & Haigh, E. A. (2014). Advances in cognitive theory and therapy: The generic cognitive model. Annual Review of Clinical Psychology, 10, 1–24.
Borkovec, T. D., Newman, M. G., Pincus, A. L., & Lytle, R. (2002). A component analysis of cognitive-behavioral therapy for generalized anxiety disorder and the role of interpersonal problems. Journal of Consulting and Clinical Psychology, 70, 288–298. https://doi.org/10.1037/0022-006X.70.2.288.
Branson, A., Shafran, R., & Myles, P. (2015). Investigating the relationship between competence and patient outcome with CBT. Behaviour research and therapy, 68, 19–26.
Byrne, B. M. (2004). Testing for multigroup invariance using AMOS graphics: A road less traveled. Structural Equation Modeling, 11, 272–300.
Chavira, D. A., Golinelli, D., Sherbourne, C., Stein, M. B., Sullivan, G., Bystritsky, A., … Bumgardner, K. (2014). Treatment engagement and response to CBT among Latinos with anxiety disorders in primary care. Journal of Consulting and Clinical Psychology, 82, 392–403.
Chu, B. C., & Kendall, P. C. (2009). Therapist responsiveness to child engagement: flexibility within manual-based CBT for anxious youth. Journal of Clinical Psychology, 65, 736–754.
Cook, J. M., Dinnen, S., Thompson, R., Simiola, V., & Schnurr, P. P. (2014). Changes in implementation of two evidence-based psychotherapies for PTSD in VA residential treatment programs: A national investigation. Journal of Traumatic Stress, 27, 137–143.
Costello, A. B., & Osborne, J. W. (2005). Best practices in exploratory factor analysis: Four recommendations for getting the most from your analysis. Practical Assessment, Research & Evaluation, 10, 1–9.
Creed, T. A., Frankel, S. A., German, R. E., Green, K. L., Jager-Hyman, S., Taylor, K. P., … Williston, M. A. (2016a). Implementation of transdiagnostic cognitive therapy in community behavioral health: The Beck Community Initiative. Journal of Consulting and Clinical Psychology, 84, 1116–1126.
Creed, T. A., Jager-Hyman, S., Pontoski, K., Feinberg, B., Rosenberg, Z., Evans, A. C., Hurford, M. O., & Beck, A. T. (2013). The Beck Initiative: A strength-based approach to training school-based mental health staff in cognitive therapy. International Journal of Emotional Education, 5, 49–66.
Creed, T. A., Stirman, S. W., Evans, A. C., & Beck, A. T. (2014). A model for implementation of cognitive therapy in community mental health: The Beck Initiative. Behavior Therapist, 37, 56–64.
Creed, T. A., Wolk, C. B., Feinberg, B., Evans, A. C., & Beck, A. T. (2016b). Beyond the label: Relationship between community therapists’ self-report of a cognitive behavioral therapy orientation and observed skills. Administration and Policy in Mental Health and Mental Health Services Research, 43, 36–43. https://doi.org/10.1007/s10488-014-0618-5.
Dimidjian, S., Hollon, S. D., Dobson, K. S., Schmaling, K. B., Kohlenberg, R. J., Addis, M. E., … Atkins, D. C. (2006). Randomized trial of behavioral activation, cognitive therapy, and antidepressant medication in the acute treatment of adults with major depression. Journal of Consulting and Clinical Psychology, 74, 658–670.
Dobson, K. S., Shaw, B. F., & Vallis, T. M. (1985). Reliability of a measure of the quality of cognitive therapy. British Journal of Clinical Psychology, 24, 295–300.
Essock, S. M., Nossel, I. R., McNamara, K., Bennett, M. E., Buchanan, R. W., Kreyenbuhl, J. A., … Dixon, L. B. (2015). Practical monitoring of treatment fidelity: examples from a team-based intervention for people with early psychosis. Psychiatric Services, 66, 674–676.
Fixsen, D. L., Naoom, S. F., Blase, K. A., Friedman, R. M., & Wallace, F. (2005). Implementation research: A synthesis of the literature. Tampa: University of South Florida, Louis de la Parte Florida Mental Health Institute, The National Implementation Research Network.
Forand, N. R., Evans, S., Haglin, D., & Fishman, B. (2011). Cognitive behavioral therapy in practice: Treatment delivered by trainees at an outpatient clinic is clinically effective. Behavior Therapy, 42, 612–623.
Fuggle, P., Dunsmuir, S., & Curry, V. (2012). CBT with children, young people and families. London: Sage.
Garety, P. A., Craig, T. K., Iredale, C. H., Basit, N., Fornells-Ambrojo, M., Halkoree, R., … Zala, D. (2017). Training the frontline workforce to deliver evidence-based therapy to people with psychosis: Challenges in the GOALS study. Psychiatric Services, 69, 9–11.
Gaudiano, B. A. (2008). Cognitive-behavioral therapies: Achievements and challenges. Evidence Based Mental Health, 11, 5–7. https://doi.org/10.1136/ebmh.11.1.5.
German, R. E., Adler, A., Frankel, S. A., Stirman, S. W., Pinedo, P., Evans, A. C., … Creed, T. A. (2017). Testing a web-based, trained-peer model to build capacity for evidence-based practices in community mental health systems. Psychiatric Services, 69, 286–292.
Ginsburg, G. S., Becker, K. D., Drazdowski, T. K., & Tein, J. Y. (2012). Treating anxiety disorders in inner city schools: Results from a pilot randomized controlled trial comparing CBT and usual care. Child & Youth Care Forum, 41, 1–19.
Hamilton, J. D., Kendall, P. C., Gosch, E., Furr, J. M., & Sood, E. (2008). Flexibility within fidelity. Journal of the American Academy of Child & Adolescent Psychiatry, 47, 987–993.
Herschell, A. D., Kolko, D. J., Baumann, B. L., & Davis, A. C. (2010). The role of therapist training in the implementation of psychosocial treatments: A review and critique with recommendations. Clinical Psychology Review, 30, 448–466.
Hofmann, S. G., Asmundson, G. J., & Beck, A. T. (2013). The science of cognitive therapy. Behavior Therapy, 44, 199–212.
Hofmann, S. G., Asnaani, A., Vonk, I. J., Sawyer, A. T., & Fang, A. (2012). The efficacy of cognitive behavioral therapy: A review of meta-analyses. Cognitive Therapy and Research, 36, 427–440.
Hogue, A., Henderson, C. E., Dauber, S., Barajas, P. C., Fried, A., & Liddle, H. A. (2008). Treatment adherence, competence, and outcome in individual and family therapy for adolescent behavior problems. Journal of Consulting and Clinical Psychology, 76, 544–555.
Hu, L. T., & Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling: A Multidisciplinary Journal, 6, 1–55.
Huang, F. L. (2016). Alternatives to multilevel modeling for the analysis of clustered data. The Journal of Experimental Education, 84, 175–196.
Jacobson, N. S., & Gortner, E. T. (2000). Can depression be de-medicalized in the 21st century: Scientific revolutions, counter-revolutions and the magnetic field of normal science. Behaviour Research and Therapy, 38, 103–117.
James, A. C., James, G., Cowdrey, F. A., Soler, A., & Choke, A. (2013). Cognitive behavioural therapy for anxiety disorders in children and adolescents. Cochrane Database of Systematic Reviews, 2013(6), CD004790. https://doi.org/10.1002/14651858.CD004690.pub4.
Kazantzis, N., Clayton, X., Cronin, T. J., Farchione, D., Limburg, K., & Dobson, K. S. (2018). The Cognitive Therapy Scale and Cognitive Therapy Scale-Revised as measures of therapist competence in cognitive behavior therapy for depression: Relations with short and long term outcome. Cognitive Therapy and Research, 42, 385–397.
Keen, A. J., & Freeston, M. H. (2008). Assessing competence in cognitive-behavioural therapy. The British Journal of Psychiatry, 193, 60–64.
Landis, R., Edwards, B. D., & Cortina, J. (2009). Correlated residuals among items in the estimation of measurement models. In C. E. Lance & R. J. Vandenberg (Eds.), Statistical and methodological myths and urban legends: Doctrine, verity, and fable in the organizational and social sciences (pp. 195–214). New York, NY: Routledge.
Lewis, C. C., Scott, K. E., & Hendricks, K. E. (2014). A model and guide for evaluating supervision outcomes in cognitive–behavioral therapy-focused training programs. Training and Education in Professional Psychology, 8, 165–173.
Matsunaga, M. (2010). How to factor-analyze your data right: Do’s, don’ts, and how-to’s. International Journal of Psychological Research, 3, 97–110.
McHugh, R. K., & Barlow, D. H. (2010). The dissemination and implementation of evidence-based psychological treatments: A review of current efforts. American Psychologist, 65, 73–84.
McHugh, R. K., Murray, H. W., & Barlow, D. H. (2009). Balancing fidelity and adaptation in the dissemination of empirically-supported treatments: The promise of transdiagnostic interventions. Behaviour Research and Therapy, 47, 946–953.
McManus, F., Westbrook, D., Vazquez-Montes, M., Fennell, M., & Kennerley, H. (2010). An evaluation of the effectiveness of diploma-level training in cognitive behaviour therapy. Behaviour Research and Therapy, 48, 1123–1132.
Miller, W. R., Yahne, C. E., Moyers, T. B., Martinez, J., & Pirritano, M. (2004). A randomized trial of methods to help clinicians learn motivational interviewing. Journal of consulting and Clinical Psychology, 72, 1050–1062.
Morse, G., Salyers, M. P., Rollins, A. L., Monroe-DeVita, M., & Pfahler, C. (2012). Burnout in mental health services: A review of the problem and its remediation. Administration and Policy in Mental Health and Mental Health Services Research, 39, 341–352.
Muse, K., & McManus, F. (2013). A systematic review of methods for assessing competence in cognitive–behavioural therapy. Clinical Psychology Review, 33, 484–499.
Patil, V. H., Singh, S. N., Mishra, S., & Donavan, D. T. (2008). Efficient theory development and factor retention criteria: Abandon the ‘eigenvalue greater than one’ criterion. Journal of Business Research, 61, 162–170.
Pontoski, K., Jager-Hyman, S., Cunningham, A., Sposato, R., Schultz, L., Evans, A. C., Beck, A. T., & Creed, T. A. (2016). Using a Cognitive Behavioral framework to train staff serving individuals who experience chronic homelessness. Journal of Community Psychology, 44, 674–680.
Proctor, E., Silmere, H., Raghavan, R., Hovmand, P., Aarons, G., Bunger, A., … Hensley, M. (2011). Outcomes for implementation research: Conceptual distinctions, measurement challenges, and research agenda. Administration and Policy in Mental Health and Mental Health Services Research, 38, 65–76.
Proctor, E. K., Landsverk, J., Aarons, G., Chambers, D., Glisson, C., & Mittman, B. (2009). Implementation research in mental health services: An emerging science with conceptual, methodological, and training challenges. Administration and Policy in Mental Health and Mental Health Services Research, 36, 24–34.
Riggs, S. E., & Creed, T. A. (2017). A model to transform psychosis milieu treatment using CBT-informed interventions. Cognitive and Behavioral Practice, 24, 353–362.
Rollins, A. L., McGrew, J. H., Kukla, M., McGuire, A. B., Flanagan, M. E., Hunt, M. G., … Salyers, M. P. (2016). Comparison of assertive community treatment fidelity assessment methods: Reliability and validity. Administration and Policy in Mental Health and Mental Health Services Research, 43, 157–167.
Roth, A. D., Calder, F., & Pilling, S. (2011). A competence framework for child and adolescent mental health services. Edinburgh: NHS Education for Scotland.
Rozek, D. C., Serrano, J. L., Marriott, B. R., Scott, K. S., Hickman, L. B., Brothers, B. M., … Simons, A. D. (2018). Cognitive behavioural therapy competency: Pilot data from a comparison of multiple perspectives. Behavioural and Cognitive Psychotherapy, 46, 244–250.
Schoenwald, S. K., Carter, R. E., Chapman, J. E., & Sheidow, A. J. (2008). Therapist adherence and organizational effects on change in youth behavior problems one year after multisystemic therapy. Administration and Policy in Mental Health and Mental Health Services Research, 35, 379–394.
Shaw, B. F., Elkin, I., Yamaguchi, J., Olmsted, M., Vallis, T. M., Dobson, K. S., … Imber, S. D. (1999). Therapist competence ratings in relation to clinical outcome in cognitive therapy of depression. Journal of Consulting and Clinical Psychology, 67, 837–846.
Sholomskas, D. E., Syracuse-Siewert, G., Rounsaville, B. J., Ball, S. A., Nuro, K. F., & Carroll, K. M. (2005). We don’t train in vain: a dissemination trial of three strategies of training clinicians in cognitive-behavioral therapy. Journal of Consulting and Clinical Psychology, 73, 106–115.
Simons, A. D., Padesky, C. A., Montemarano, J., Lewis, C. C., Murakami, J., Lamb, K., … Beck, A. T. (2010). Training and dissemination of cognitive behavior therapy for depression in adults: A preliminary examination of therapist competence and client outcomes. Journal of Consulting and Clinical Psychology, 78, 751–756.
Stallard, P. (2005). A clinicians guide to think good feel good: Using CBT with children and young people. Winchester: Wiley.
Stallard, P., Myles, P., & Branson, A. (2014). The cognitive behaviour therapy scale for children and young people (CBTS-CYP): Development and psychometric properties. Behavioural and Cognitive Psychotherapy, 42, 269–282.
Stirman, S. W., Calloway, A., Toder, K., Miller, C. J., DeVito, A. K., Meisel, S. N., … Crits-Christoph, P. (2013a). Community mental health provider modifications to cognitive therapy: Implications for sustainability. Psychiatric Services, 64, 1056–1059.
Stirman, S. W., Kimberly, J., Cook, N., Calloway, A., Castro, F., & Charns, M. (2012). The sustainability of new programs and innovations: A review of the empirical literature and recommendations for future research. Implementation Science, 7, 1–19.
Stirman, S. W., Marques, L., Creed, T. A., Gutner, C. A., DeRubeis, R., Barnett, P. G., … Jo, B. (2018). Leveraging routine clinical materials and mobile technology to assess CBT fidelity: The Innovative Methods to Assess Psychotherapy Practices (imAPP) study. Implementation Science, 13, 1–11.
Stirman, S. W., Miller, C. J., Toder, K., & Calloway, A. (2013b). Development of a framework and coding system for modifications and adaptations of evidence-based interventions. Implementation Science, 8, 1–12.
Strunk, D. R., Brotman, M. A., & DeRubeis, R. J. (2010). The process of change in cognitive therapy for depression: Predictors of early inter-session symptom gains. Behaviour Research and Therapy, 48, 599–606.
Thomas, S. L., & Heck, R. H. (2001). Analysis of large-scale secondary data in higher education research: Potential perils associated with complex sampling designs. Research in Higher Education, 42, 517–540.
Trepka, C., Rees, A., Shapiro, D. A., Hardy, G. E., & Barkham, M. (2004). Therapist competence and outcome of cognitive therapy for depression. Cognitive Therapy and Research, 28, 143–157.
Vallis, T. M., Shaw, B. F., & Dobson, K. S. (1986). The cognitive therapy scale: Psychometric properties. Journal of Consulting and Clinical Psychology, 54, 381–385.
Van Prooijen, J. W., & Van Der Kloot, W. A. (2001). Confirmatory analysis of exploratively obtained factor structures. Educational and Psychological Measurement, 61(5), 777–792.
Waltman, S. H., Hall, B., McFarr, L., Beck, A. T., & Creed, T. A. (2017). In-session stuck points and pitfalls of community clinicians learning CBT: A qualitative investigation. Cognitive and Behavioral Practice, 24, 256–267. https://doi.org/10.1016/j.cbpra.2016.04.002.
Waltman, S. H., Hall, B. C., McFarr, L. M., & Creed, T. A. (2018). Clinical case consultation and experiential learning in cognitive behavioral therapy implementation: Brief qualitative investigation. Journal of Cognitive Psychotherapy, 32, 112–127.
Westbrook, D., Sedgwick-Taylor, A., Bennett-Levy, J., Butler, G., & McManus, F. (2008). A pilot evaluation of a brief CBT training course: Impact on trainees’ satisfaction, clinical skills and patient outcomes. Behavioural and Cognitive Psychotherapy, 36, 569–579.
Westra, H. A., Arkowitz, H., & Dozois, D. J. (2009). Adding a motivational interviewing pretreatment to cognitive behavioral therapy for generalized anxiety disorder: A preliminary randomized controlled trial. Journal of Anxiety Disorders, 23, 1106–1117.
Young, J., & Beck, A. (1980). Cognitive therapy scale: Rating manual. Unpublished manuscript. Philadelphia: Center for Cognitive Therapy.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
The authors declare that they have no conflict of interest.
Ethical Approval
All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards.
Research Involving Animal Rights
This article does not contain any studies with animals performed by any of the authors.
Informed Consent
Informed consent was obtained from all individual participants included in the study.
Additional information
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Affrunti, N.W., Creed, T.A. The Factor Structure of the Cognitive Therapy Rating Scale (CTRS) in a Sample of Community Mental Health Clinicians. Cogn Ther Res 43, 642–655 (2019). https://doi.org/10.1007/s10608-019-09998-7
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10608-019-09998-7