Skip to main content

Advertisement

Log in

The Factor Structure of the Cognitive Therapy Rating Scale (CTRS) in a Sample of Community Mental Health Clinicians

  • Original Article
  • Published:
Cognitive Therapy and Research Aims and scope Submit manuscript

Abstract

Treatment fidelity is an essential outcome of implementation research. The gold standard measure for Cognitive-Behavioral Therapy (CBT) fidelity is the Cognitive Therapy Rating Scale (CTRS). Despite its widespread use in research and training programs, the structure of the CTRS has not been examined in a sample of community mental health clinicians with adult and child clients. The current study addressed this gap. The sample consisted of 355 clinicians and 1298 CBT sessions scored using the CTRS. High interrater reliability was observed and factor analysis yielded separate structures for child and adult treatment sessions. These structures were not consistent with previous factor analyses conducted on the scale. Findings suggest that the CTRS is a reliable measure of CBT in community mental health settings but that its structure may depend on the clinical population measured. Additionally, the factor structure can provide guidance for delivering feedback in training and supervision settings.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2

Similar content being viewed by others

References

  • Aarons, G. A., Miller, E. A., Green, A. E., Perrott, J. A., & Bradway, R. (2012). Adaptation happens: A qualitative case study of implementation of the incredible years evidence-based parent training programme in a residential substance abuse treatment programme. Journal of Children’s Services, 7, 233–245.

    Article  Google Scholar 

  • Arbuckle, J. L. (2012). Amos (Version 21.0) [Computer Program]. Chicago: IBM SPSS.

    Google Scholar 

  • Beavers, A. S., Lounsbury, J. W., Richards, J. K., Huck, S. W., Skolits, G. J., & Esquivel, S. L. (2013). Practical considerations for using exploratory factor analysis in educational research. Practical Assessment, Research & Evaluation, 18, 1–13.

    Google Scholar 

  • Beck, A. T., & Haigh, E. A. (2014). Advances in cognitive theory and therapy: The generic cognitive model. Annual Review of Clinical Psychology, 10, 1–24.

    Article  Google Scholar 

  • Borkovec, T. D., Newman, M. G., Pincus, A. L., & Lytle, R. (2002). A component analysis of cognitive-behavioral therapy for generalized anxiety disorder and the role of interpersonal problems. Journal of Consulting and Clinical Psychology, 70, 288–298. https://doi.org/10.1037/0022-006X.70.2.288.

    Article  CAS  PubMed  Google Scholar 

  • Branson, A., Shafran, R., & Myles, P. (2015). Investigating the relationship between competence and patient outcome with CBT. Behaviour research and therapy, 68, 19–26.

    Article  Google Scholar 

  • Byrne, B. M. (2004). Testing for multigroup invariance using AMOS graphics: A road less traveled. Structural Equation Modeling, 11, 272–300.

    Article  Google Scholar 

  • Chavira, D. A., Golinelli, D., Sherbourne, C., Stein, M. B., Sullivan, G., Bystritsky, A., … Bumgardner, K. (2014). Treatment engagement and response to CBT among Latinos with anxiety disorders in primary care. Journal of Consulting and Clinical Psychology, 82, 392–403.

    Article  Google Scholar 

  • Chu, B. C., & Kendall, P. C. (2009). Therapist responsiveness to child engagement: flexibility within manual-based CBT for anxious youth. Journal of Clinical Psychology, 65, 736–754.

    Article  Google Scholar 

  • Cook, J. M., Dinnen, S., Thompson, R., Simiola, V., & Schnurr, P. P. (2014). Changes in implementation of two evidence-based psychotherapies for PTSD in VA residential treatment programs: A national investigation. Journal of Traumatic Stress, 27, 137–143.

    Article  Google Scholar 

  • Costello, A. B., & Osborne, J. W. (2005). Best practices in exploratory factor analysis: Four recommendations for getting the most from your analysis. Practical Assessment, Research & Evaluation, 10, 1–9.

    Google Scholar 

  • Creed, T. A., Frankel, S. A., German, R. E., Green, K. L., Jager-Hyman, S., Taylor, K. P., … Williston, M. A. (2016a). Implementation of transdiagnostic cognitive therapy in community behavioral health: The Beck Community Initiative. Journal of Consulting and Clinical Psychology, 84, 1116–1126.

    Article  Google Scholar 

  • Creed, T. A., Jager-Hyman, S., Pontoski, K., Feinberg, B., Rosenberg, Z., Evans, A. C., Hurford, M. O., & Beck, A. T. (2013). The Beck Initiative: A strength-based approach to training school-based mental health staff in cognitive therapy. International Journal of Emotional Education, 5, 49–66.

    Google Scholar 

  • Creed, T. A., Stirman, S. W., Evans, A. C., & Beck, A. T. (2014). A model for implementation of cognitive therapy in community mental health: The Beck Initiative. Behavior Therapist, 37, 56–64.

    Google Scholar 

  • Creed, T. A., Wolk, C. B., Feinberg, B., Evans, A. C., & Beck, A. T. (2016b). Beyond the label: Relationship between community therapists’ self-report of a cognitive behavioral therapy orientation and observed skills. Administration and Policy in Mental Health and Mental Health Services Research, 43, 36–43. https://doi.org/10.1007/s10488-014-0618-5.

    Article  Google Scholar 

  • Dimidjian, S., Hollon, S. D., Dobson, K. S., Schmaling, K. B., Kohlenberg, R. J., Addis, M. E., … Atkins, D. C. (2006). Randomized trial of behavioral activation, cognitive therapy, and antidepressant medication in the acute treatment of adults with major depression. Journal of Consulting and Clinical Psychology, 74, 658–670.

    Article  Google Scholar 

  • Dobson, K. S., Shaw, B. F., & Vallis, T. M. (1985). Reliability of a measure of the quality of cognitive therapy. British Journal of Clinical Psychology, 24, 295–300.

    Article  Google Scholar 

  • Essock, S. M., Nossel, I. R., McNamara, K., Bennett, M. E., Buchanan, R. W., Kreyenbuhl, J. A., … Dixon, L. B. (2015). Practical monitoring of treatment fidelity: examples from a team-based intervention for people with early psychosis. Psychiatric Services, 66, 674–676.

    Article  Google Scholar 

  • Fixsen, D. L., Naoom, S. F., Blase, K. A., Friedman, R. M., & Wallace, F. (2005). Implementation research: A synthesis of the literature. Tampa: University of South Florida, Louis de la Parte Florida Mental Health Institute, The National Implementation Research Network.

    Google Scholar 

  • Forand, N. R., Evans, S., Haglin, D., & Fishman, B. (2011). Cognitive behavioral therapy in practice: Treatment delivered by trainees at an outpatient clinic is clinically effective. Behavior Therapy, 42, 612–623.

    Article  Google Scholar 

  • Fuggle, P., Dunsmuir, S., & Curry, V. (2012). CBT with children, young people and families. London: Sage.

    Google Scholar 

  • Garety, P. A., Craig, T. K., Iredale, C. H., Basit, N., Fornells-Ambrojo, M., Halkoree, R., … Zala, D. (2017). Training the frontline workforce to deliver evidence-based therapy to people with psychosis: Challenges in the GOALS study. Psychiatric Services, 69, 9–11.

    Article  Google Scholar 

  • Gaudiano, B. A. (2008). Cognitive-behavioral therapies: Achievements and challenges. Evidence Based Mental Health, 11, 5–7. https://doi.org/10.1136/ebmh.11.1.5.

    Article  PubMed  PubMed Central  Google Scholar 

  • German, R. E., Adler, A., Frankel, S. A., Stirman, S. W., Pinedo, P., Evans, A. C., … Creed, T. A. (2017). Testing a web-based, trained-peer model to build capacity for evidence-based practices in community mental health systems. Psychiatric Services, 69, 286–292.

    Article  Google Scholar 

  • Ginsburg, G. S., Becker, K. D., Drazdowski, T. K., & Tein, J. Y. (2012). Treating anxiety disorders in inner city schools: Results from a pilot randomized controlled trial comparing CBT and usual care. Child & Youth Care Forum, 41, 1–19.

    Article  Google Scholar 

  • Hamilton, J. D., Kendall, P. C., Gosch, E., Furr, J. M., & Sood, E. (2008). Flexibility within fidelity. Journal of the American Academy of Child & Adolescent Psychiatry, 47, 987–993.

    Article  Google Scholar 

  • Herschell, A. D., Kolko, D. J., Baumann, B. L., & Davis, A. C. (2010). The role of therapist training in the implementation of psychosocial treatments: A review and critique with recommendations. Clinical Psychology Review, 30, 448–466.

    Article  Google Scholar 

  • Hofmann, S. G., Asmundson, G. J., & Beck, A. T. (2013). The science of cognitive therapy. Behavior Therapy, 44, 199–212.

    Article  Google Scholar 

  • Hofmann, S. G., Asnaani, A., Vonk, I. J., Sawyer, A. T., & Fang, A. (2012). The efficacy of cognitive behavioral therapy: A review of meta-analyses. Cognitive Therapy and Research, 36, 427–440.

    Article  Google Scholar 

  • Hogue, A., Henderson, C. E., Dauber, S., Barajas, P. C., Fried, A., & Liddle, H. A. (2008). Treatment adherence, competence, and outcome in individual and family therapy for adolescent behavior problems. Journal of Consulting and Clinical Psychology, 76, 544–555.

    Article  Google Scholar 

  • Hu, L. T., & Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling: A Multidisciplinary Journal, 6, 1–55.

    Article  Google Scholar 

  • Huang, F. L. (2016). Alternatives to multilevel modeling for the analysis of clustered data. The Journal of Experimental Education, 84, 175–196.

    Article  Google Scholar 

  • Jacobson, N. S., & Gortner, E. T. (2000). Can depression be de-medicalized in the 21st century: Scientific revolutions, counter-revolutions and the magnetic field of normal science. Behaviour Research and Therapy, 38, 103–117.

    Article  CAS  Google Scholar 

  • James, A. C., James, G., Cowdrey, F. A., Soler, A., & Choke, A. (2013). Cognitive behavioural therapy for anxiety disorders in children and adolescents. Cochrane Database of Systematic Reviews, 2013(6), CD004790. https://doi.org/10.1002/14651858.CD004690.pub4.

    Article  Google Scholar 

  • Kazantzis, N., Clayton, X., Cronin, T. J., Farchione, D., Limburg, K., & Dobson, K. S. (2018). The Cognitive Therapy Scale and Cognitive Therapy Scale-Revised as measures of therapist competence in cognitive behavior therapy for depression: Relations with short and long term outcome. Cognitive Therapy and Research, 42, 385–397.

    Article  Google Scholar 

  • Keen, A. J., & Freeston, M. H. (2008). Assessing competence in cognitive-behavioural therapy. The British Journal of Psychiatry, 193, 60–64.

    Article  Google Scholar 

  • Landis, R., Edwards, B. D., & Cortina, J. (2009). Correlated residuals among items in the estimation of measurement models. In C. E. Lance & R. J. Vandenberg (Eds.), Statistical and methodological myths and urban legends: Doctrine, verity, and fable in the organizational and social sciences (pp. 195–214). New York, NY: Routledge.

  • Lewis, C. C., Scott, K. E., & Hendricks, K. E. (2014). A model and guide for evaluating supervision outcomes in cognitive–behavioral therapy-focused training programs. Training and Education in Professional Psychology, 8, 165–173.

    Article  Google Scholar 

  • Matsunaga, M. (2010). How to factor-analyze your data right: Do’s, don’ts, and how-to’s. International Journal of Psychological Research, 3, 97–110.

    Article  Google Scholar 

  • McHugh, R. K., & Barlow, D. H. (2010). The dissemination and implementation of evidence-based psychological treatments: A review of current efforts. American Psychologist, 65, 73–84.

    Article  Google Scholar 

  • McHugh, R. K., Murray, H. W., & Barlow, D. H. (2009). Balancing fidelity and adaptation in the dissemination of empirically-supported treatments: The promise of transdiagnostic interventions. Behaviour Research and Therapy, 47, 946–953.

    Article  Google Scholar 

  • McManus, F., Westbrook, D., Vazquez-Montes, M., Fennell, M., & Kennerley, H. (2010). An evaluation of the effectiveness of diploma-level training in cognitive behaviour therapy. Behaviour Research and Therapy, 48, 1123–1132.

    Article  Google Scholar 

  • Miller, W. R., Yahne, C. E., Moyers, T. B., Martinez, J., & Pirritano, M. (2004). A randomized trial of methods to help clinicians learn motivational interviewing. Journal of consulting and Clinical Psychology, 72, 1050–1062.

    Article  Google Scholar 

  • Morse, G., Salyers, M. P., Rollins, A. L., Monroe-DeVita, M., & Pfahler, C. (2012). Burnout in mental health services: A review of the problem and its remediation. Administration and Policy in Mental Health and Mental Health Services Research, 39, 341–352.

    Article  Google Scholar 

  • Muse, K., & McManus, F. (2013). A systematic review of methods for assessing competence in cognitive–behavioural therapy. Clinical Psychology Review, 33, 484–499.

    Article  Google Scholar 

  • Patil, V. H., Singh, S. N., Mishra, S., & Donavan, D. T. (2008). Efficient theory development and factor retention criteria: Abandon the ‘eigenvalue greater than one’ criterion. Journal of Business Research, 61, 162–170.

    Article  Google Scholar 

  • Pontoski, K., Jager-Hyman, S., Cunningham, A., Sposato, R., Schultz, L., Evans, A. C., Beck, A. T., & Creed, T. A. (2016). Using a Cognitive Behavioral framework to train staff serving individuals who experience chronic homelessness. Journal of Community Psychology, 44, 674–680.

    Article  Google Scholar 

  • Proctor, E., Silmere, H., Raghavan, R., Hovmand, P., Aarons, G., Bunger, A., … Hensley, M. (2011). Outcomes for implementation research: Conceptual distinctions, measurement challenges, and research agenda. Administration and Policy in Mental Health and Mental Health Services Research, 38, 65–76.

    Article  Google Scholar 

  • Proctor, E. K., Landsverk, J., Aarons, G., Chambers, D., Glisson, C., & Mittman, B. (2009). Implementation research in mental health services: An emerging science with conceptual, methodological, and training challenges. Administration and Policy in Mental Health and Mental Health Services Research, 36, 24–34.

    Article  Google Scholar 

  • Riggs, S. E., & Creed, T. A. (2017). A model to transform psychosis milieu treatment using CBT-informed interventions. Cognitive and Behavioral Practice, 24, 353–362.

    Article  Google Scholar 

  • Rollins, A. L., McGrew, J. H., Kukla, M., McGuire, A. B., Flanagan, M. E., Hunt, M. G., … Salyers, M. P. (2016). Comparison of assertive community treatment fidelity assessment methods: Reliability and validity. Administration and Policy in Mental Health and Mental Health Services Research, 43, 157–167.

    Article  Google Scholar 

  • Roth, A. D., Calder, F., & Pilling, S. (2011). A competence framework for child and adolescent mental health services. Edinburgh: NHS Education for Scotland.

    Google Scholar 

  • Rozek, D. C., Serrano, J. L., Marriott, B. R., Scott, K. S., Hickman, L. B., Brothers, B. M., … Simons, A. D. (2018). Cognitive behavioural therapy competency: Pilot data from a comparison of multiple perspectives. Behavioural and Cognitive Psychotherapy, 46, 244–250.

    Article  Google Scholar 

  • Schoenwald, S. K., Carter, R. E., Chapman, J. E., & Sheidow, A. J. (2008). Therapist adherence and organizational effects on change in youth behavior problems one year after multisystemic therapy. Administration and Policy in Mental Health and Mental Health Services Research, 35, 379–394.

    Article  Google Scholar 

  • Shaw, B. F., Elkin, I., Yamaguchi, J., Olmsted, M., Vallis, T. M., Dobson, K. S., … Imber, S. D. (1999). Therapist competence ratings in relation to clinical outcome in cognitive therapy of depression. Journal of Consulting and Clinical Psychology, 67, 837–846.

    Article  CAS  Google Scholar 

  • Sholomskas, D. E., Syracuse-Siewert, G., Rounsaville, B. J., Ball, S. A., Nuro, K. F., & Carroll, K. M. (2005). We don’t train in vain: a dissemination trial of three strategies of training clinicians in cognitive-behavioral therapy. Journal of Consulting and Clinical Psychology, 73, 106–115.

    Article  Google Scholar 

  • Simons, A. D., Padesky, C. A., Montemarano, J., Lewis, C. C., Murakami, J., Lamb, K., … Beck, A. T. (2010). Training and dissemination of cognitive behavior therapy for depression in adults: A preliminary examination of therapist competence and client outcomes. Journal of Consulting and Clinical Psychology, 78, 751–756.

    Article  Google Scholar 

  • Stallard, P. (2005). A clinicians guide to think good feel good: Using CBT with children and young people. Winchester: Wiley.

    Google Scholar 

  • Stallard, P., Myles, P., & Branson, A. (2014). The cognitive behaviour therapy scale for children and young people (CBTS-CYP): Development and psychometric properties. Behavioural and Cognitive Psychotherapy, 42, 269–282.

    Article  Google Scholar 

  • Stirman, S. W., Calloway, A., Toder, K., Miller, C. J., DeVito, A. K., Meisel, S. N., … Crits-Christoph, P. (2013a). Community mental health provider modifications to cognitive therapy: Implications for sustainability. Psychiatric Services, 64, 1056–1059.

    Article  Google Scholar 

  • Stirman, S. W., Kimberly, J., Cook, N., Calloway, A., Castro, F., & Charns, M. (2012). The sustainability of new programs and innovations: A review of the empirical literature and recommendations for future research. Implementation Science, 7, 1–19.

    Article  Google Scholar 

  • Stirman, S. W., Marques, L., Creed, T. A., Gutner, C. A., DeRubeis, R., Barnett, P. G., … Jo, B. (2018). Leveraging routine clinical materials and mobile technology to assess CBT fidelity: The Innovative Methods to Assess Psychotherapy Practices (imAPP) study. Implementation Science, 13, 1–11.

    Google Scholar 

  • Stirman, S. W., Miller, C. J., Toder, K., & Calloway, A. (2013b). Development of a framework and coding system for modifications and adaptations of evidence-based interventions. Implementation Science, 8, 1–12.

    Article  Google Scholar 

  • Strunk, D. R., Brotman, M. A., & DeRubeis, R. J. (2010). The process of change in cognitive therapy for depression: Predictors of early inter-session symptom gains. Behaviour Research and Therapy, 48, 599–606.

    Article  Google Scholar 

  • Thomas, S. L., & Heck, R. H. (2001). Analysis of large-scale secondary data in higher education research: Potential perils associated with complex sampling designs. Research in Higher Education, 42, 517–540.

    Article  Google Scholar 

  • Trepka, C., Rees, A., Shapiro, D. A., Hardy, G. E., & Barkham, M. (2004). Therapist competence and outcome of cognitive therapy for depression. Cognitive Therapy and Research, 28, 143–157.

    Article  Google Scholar 

  • Vallis, T. M., Shaw, B. F., & Dobson, K. S. (1986). The cognitive therapy scale: Psychometric properties. Journal of Consulting and Clinical Psychology, 54, 381–385.

    Article  CAS  Google Scholar 

  • Van Prooijen, J. W., & Van Der Kloot, W. A. (2001). Confirmatory analysis of exploratively obtained factor structures. Educational and Psychological Measurement, 61(5), 777–792.

    Article  Google Scholar 

  • Waltman, S. H., Hall, B., McFarr, L., Beck, A. T., & Creed, T. A. (2017). In-session stuck points and pitfalls of community clinicians learning CBT: A qualitative investigation. Cognitive and Behavioral Practice, 24, 256–267. https://doi.org/10.1016/j.cbpra.2016.04.002.

    Article  Google Scholar 

  • Waltman, S. H., Hall, B. C., McFarr, L. M., & Creed, T. A. (2018). Clinical case consultation and experiential learning in cognitive behavioral therapy implementation: Brief qualitative investigation. Journal of Cognitive Psychotherapy, 32, 112–127.

    Article  Google Scholar 

  • Westbrook, D., Sedgwick-Taylor, A., Bennett-Levy, J., Butler, G., & McManus, F. (2008). A pilot evaluation of a brief CBT training course: Impact on trainees’ satisfaction, clinical skills and patient outcomes. Behavioural and Cognitive Psychotherapy, 36, 569–579.

    Article  Google Scholar 

  • Westra, H. A., Arkowitz, H., & Dozois, D. J. (2009). Adding a motivational interviewing pretreatment to cognitive behavioral therapy for generalized anxiety disorder: A preliminary randomized controlled trial. Journal of Anxiety Disorders, 23, 1106–1117.

    Article  Google Scholar 

  • Young, J., & Beck, A. (1980). Cognitive therapy scale: Rating manual. Unpublished manuscript. Philadelphia: Center for Cognitive Therapy.

    Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Torrey A. Creed.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Ethical Approval

All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards.

Research Involving Animal Rights

This article does not contain any studies with animals performed by any of the authors.

Informed Consent

Informed consent was obtained from all individual participants included in the study.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Affrunti, N.W., Creed, T.A. The Factor Structure of the Cognitive Therapy Rating Scale (CTRS) in a Sample of Community Mental Health Clinicians. Cogn Ther Res 43, 642–655 (2019). https://doi.org/10.1007/s10608-019-09998-7

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10608-019-09998-7

Keywords

Navigation