The Sustained Effects of CBT Training on Therapist Competence and Patient Outcomes

The continued effectiveness of in-service psychological therapy training requires evaluation. This study assessed therapist competence and patient clinical outcome during cognitive behaviour therapy training and 12+ months post-training. Trainee competence was assessed using audio-recorded sessions rated on the Cognitive Therapy Scale Revised at the beginning (n = 33) and end of training (n = 45), and at least 12 months post-training (n = 45). Pre-to-posttreatment clinical outcome for trainees’ patients during the course (n = 360) and post-training (n = 360) was evaluated using standardised self-report measures. The relationship between therapist competence and patient outcomes was explored. Trainees achieved competence during training (100%, n = 45) and largely maintained competence post-training (84%, n = 38). Patients demonstrated pre-to-posttreatment effect sizes between 1.38 and 1.89 and reliable improvement exceeding 80% during and after training. Competence was not significantly associated with patient outcome. Trainees predominantly maintained competence and achieved good clinical outcomes post-training. Structured training and continued use of regular supervision possibly supported retention of competence.


Introduction
The delivery of psychological therapies at scale is an important international issue, with many countries developing programmes for the dissemination of evidence-based practice across common mental health problems. If these are to be successful, systems for training large numbers of therapists to competence (benchmarked to a required standard) must be developed and evaluated. Whilst evaluating the acquisition of therapy competence and patient outcomes during training is important, monitoring the maintenance of competence and patient outcome post-training in the practice setting is essential to realise the public benefit.
Over the last decade, two national training programmes with different training models have substantially increased the availability of evidence-based treatment with promising results. In the UK, the Improving Access to Psychological Therapies (IAPT) initiative, which focuses on evidence-based treatments for depression and anxiety disorders, has reported large pre-post treatment effect sizes for depression and anxiety cases (Clark, 2011;Clark et al., 2018a) with large numbers of therapists trained to an agreed level of competence (Branson, Shafran, & Myles, 2015;Liness et al., under review). In the USA, the Department of Veteran Affairs (VA) focuses on evidence-based treatments for posttraumatic stress disorder and related conditions, with over 10,000 practitioners trained (Rosen, Ruzek, & Karlin, 2017) and patient clinical improvement with medium to large effect sizes across disorders (Eftekhari et al., 2013;Karlin et al., 2012). Availability of treatment for common mental health disorders is also being scaled up in low and middle-income countries (Lund, Thomlinson, & Patel, 2016;Singla et al., 2017). Promising evidence indicates that nonspecialist practitioners can be trained to deliver effective treatment in these settings Singla et al., 2017). Whilst a positive picture is emerging for the dissemination of evidence-based therapies, success must be measured in the longer-term by assessing therapists' clinical competence once they are fully embedded within the clinical services.
Two inter-related aspects of therapist skill are involved in assessing the standard of therapy: (1) therapy adherencewhether the therapist implemented the relevant procedures (2) therapy competencewhether procedures were implemented in a competent manner (Blackburn et al., 2001). These are commonly seen as requisite for the effective delivery of therapy in routine practice (Barber, Liese, & Abrams, 2003;Fairburn & Cooper, 2011;Muse & McManus 2013). If training is to demonstrate sustained effects, it is important that therapy adherence and competence, and good clinical outcomes are maintained. This study focuses on therapy competence and clinical outcomes.
Information on the sustained use of evidence-based interventions in the workplace after training has been captured in surveys and questionnaires (Karlin et al., 2012;Liness, Lea, Nestler, Parker, & Clark, 2017;Rosen et al., 2016), with trainees self-reporting continued use. However, self-report may be an inaccurate reflection of therapeutic practice (Walfish, McAlister O'Donnell, & Lambert, 2012). Psychological therapists may also drift from evidence-based interventions following qualification (Waller & Turner, 2016), thus demonstrating the need for on-going evaluation.
The few studies employing independent observer ratings on clinical assessment to investigate long-term maintenance of trainees' competence and patient outcomes have generated mixed results. CBT competence and patient outcome improved pre-to 6 months post-training and were maintained at 12-months follow-up for 12 therapists who received brief training in the treatment of depression in the community (Simons et al., 2010). A comparison of three CBT training methods similarly found that adherence and competenceassessed via structured role playimproved pre-to-post-training, and that therapists in a seminar plus supervision and web condition retained gains following training (Sholomskas et al., 2005). Additionally, a study of low-intensity practitioners delivering self-help CBT in the IAPT programme reported significantly better clinical outcome 12 months after training (Branson, Myles, Mahdi, & Shafran, 2018). However, novice therapists followed-up at 6-, 12-, and 18-months post-training in computer-assisted CBT for anxiety (Brown et al., 2013) demonstrated significant reductions in adherence and a non-significant trend towards reduction in competence over time; booster-training was proposed for novice therapists to limit potential decline in protocol adherence.
The influence of therapist competence on patient outcome presents as a mixed picture (Webb, DeRubeis, & Barber, 2010) with some evidence for a relationship to outcome in CBT for depression (Zarafonitis-Müller, Kuhr, & Bechdolf, 2014) and social anxiety disorder (Ginzburg et al., 2012). Higher CBT competence, but not adherence, was associated with better clinical outcomes in novice therapists (Brown et al., 2013). No general relationship was found on UK IAPT-based training courses (Branson et al., 2015;Jolley et al., 2015) apart from a small relationship in depression cases (Liness et al., under review) and some evidence that patients treated by the most competent therapists are more likely to achieve reliable improvement in symptoms (Branson et al., 2015(Branson et al., , 2018. Given the mixed findings, small sample sizes, and varied methods, measures, and focus of these studies, further follow-up studies are essential to assess the long-term benefits of large-scale training initiatives, the role of therapist competence, and the transfer of skills from training to the workplace.

Training, Supervision and Service Integration.
A work environment and structure that supports training initiatives and encourages therapist development may enhance the transfer of skills learned and maintenance of competence (Beidas & Kendall, 2010;Stirman, Gutner, Langdon, & Graham, 2016). Within that structure, the importance of on-going consultation and supervision post-training has been highlighted (Edmunds, Beidas, & Kendall, 2013;Herschell, Kolko, Baumann & Davis., 2010;Miller, Yahne, Moyers, Martinez, & Pirritano, 2004); however, empirical support for the role of supervision in CBT training is limited. A systematic review examining the effects of supervision on CBT therapist competence across five studies reported potential effects on novice therapists' competence (Alfonsson, Parling, Spännargård, Andersson, & Lundgren 2018). Palliative care practitioners randomised to supervision versus no supervision postbrief CBT training differed significantly in skills 6-months post-training, with the no supervision group decreasing and the supervision group increasing (Mannix et al., 2006). by their trainees' university to enhance collaboration and to encourage comparable supervision methods during and after training. IAPT graduates who remain in IAPT services continue to receive regular supervision from a BABCP-accredited CBT therapist. Given the specific design of the IAPT initiative, and emerging evidence for the importance of on-going supervision and consultation, exploration of this topic at follow-up is warranted.

IAPT CBT Training
IAPT CBT trainees work in IAPT services whilst being trained in evidence-based CBT for adults with depression and anxiety disorders. They attend university two days per week and work in IAPT clinical-services three days per week, where they maintain a caseload of twelve cases. Training follows a national curriculum (DoH, 2011) covering behavioural activation and cognitive therapy models for depression, and evidence-based protocols for anxiety disorders (Roth & Pilling, 2008). Trainees are required to treat a range of disorders and are formally assessed on both depression and a range of anxiety cases. Therapists in-training are closely monitored, with regular skills assessments, on-going informal and formal skills feedback, weekly clinical supervision involving live review of therapy clips and whole sessions using the Cognitive Therapy Scale -Revised (CTS-R; Blackburn et al., 2001), and patient outcome measure monitoring. They are encouraged to see training cases, defined as straightforward anxiety or depression presentations with clear main problem descriptors, to enable the trainee to learn National Institute for Health and Care Excellence (NICE) guidance evidence-based protocols for specific difficulties (Roth & Pilling, 2008). CBT trainees submit eight cases for formal assessment. IAPT trainees are employed on one-year training contracts, with an expectation that trainees will continue to work in IAPT services post-training, usually with an increase to full-time work with larger and more diverse caseloads. The intensive training, time, and resources invested in IAPT and other large-scale workforce initiatives, and the lack of studies evaluating the long-term benefits of training, makes follow-up assessment important. This study was set up to investigate therapist competence and patient outcomes during training and post-training when working as qualified CBT therapists.

Aims and Hypotheses
This study aimed to assess if IAPT trainees' CBT competence gained during training was maintained after training. It also set out to evaluate trainees' patient outcomes during and after training. It was predicted that (1) the level of CBT therapy competence attained during training would be maintained post-training (2) trainees would achieve good patient outcome in training and at follow-up.
The relationship between therapist competence to patient outcome was also explored, and descriptive information on patient demographics and supervision provision at follow-up was gathered to evaluate continuity of practice in the workplace.
Former trainees were contacted at least one-year following course completion regarding study participation. Due to timing of ethics approval, trainees clustered into two groups based on time to follow-up: prospective follow-up after 12 months (42%, n=19) and retrospective follow-up after 12+ months to 6 years (58%, n=26). Of 119 contacted trainees, 45% (n=54) agreed to participate. Of those who declined, 43% (n=28) declined due to workload, 34% (n=22) due to life-events, and 23% (n=15) did not respond. Of those who agreed to participate, 83% (n=45) completed the study. All nine non-completers withdrew due to workload. There were no significant differences in age, gender, profession, pretraining CTS-R scores, mean end-of-training CTS-R scores, or final course result between trainees who completed the study and those who declined to participate or those who withdrew, p>.05.

Measures
Therapist competence was evaluated via assessor ratings of audio-recorded therapy sessions with the 12-item Cognitive Therapy Scale -Revised (CTS-R; Blackburn et al., 2001): score range = 0-72; competence threshold set at ≥36. Items are rated from 0-6 (0=noncompetent, 6= expert). Items can be separated into two sub-scales: general therapy skills (Items 1-5) and CBT-specific skills (Items 6-12). The CTS-R has been criticized for inconsistent inter-rater reliability (Gordon, 2006;Reichelt, James, & Blackburn, 2003;Rozek et al., 2018), a lack of focus on specific mental health problems, and the lack of an empirically validated competence threshold (Fairburn & Cooper 2011;Muse & McManus, 2013), although a score of ≥36 (item mean ≥ 3) is recommended in the manual. However, the CTS-R is the most widely-used measure of CBT competence on UK training courses and incorporates assessment of adherence as well as competence within item descriptors (James, Blackburn, & Reichelt, 2001). A more recent study (Kazantzis et al., 2018) reported acceptable internal consistency and inter-rater reliability. In the present study, internal interrater reliability was high between pairs of the twenty course markers for a random sample of 25% (n=31) of the 123 submitted tapes: one-way random single measures ICC (31,30) =.85.
Lack of resources mean that in most studies course tutors or supervisors are used to rate tapes; however, in the present study a randomly-selected 18% (n=22) of tapes were secondrated by an external expert who was blind to time (pre-training, post-training, or follow-up).
Clinical outcomes were evaluated using patient self-report questionnaires routinely collected at the start and end of therapy. Depression symptoms for all cases were assessed with the Patient Health Questionnaire (PHQ-9; Kroenke, Spitzer, & Williams, 2001). Anxiety symptoms for all cases were assessed with the Generalized Anxiety Disorder-7 (GAD-7; Spitzer et al., 2006). If provided, anxiety disorder-specific symptoms were assessed using one Case complexity was benchmarked against course criteria for complexity, namely primary or comorbid psychosis, personality disorder, autism spectrum disorder, substance dependence, severe and/or treatment-relevant physical health conditions, and severe psychosocial difficulties. Two researchers from the present study rated cases as complex or non-complex based on these criteria extracted from case information. Inter-rater reliability was acceptable, Kappa = .67, p<.001 (McHugh, 2012). All disagreed cases were discussed and an agreed rating was reached.

Design and Procedure
This study employed an observational longitudinal design, assessing trainees' therapy competence at the start and end of training and at follow-up, and assessing patient outcomes on the course and at follow-up. Therapists and patients provided informed consent for their audio-recordings and clinical data to be used anonymously and confidentially for research purposes.
Therapy Recordings. Trainees submitted therapy audio-recordings of a recent treatment session at the beginning of the course (Baseline), towards the end of training (Endof-Training), and at follow-up (Follow-Up). At all time points, trainees were requested to submit for assessment a representative session rather than their strongest. Only 33 Baseline recordings were available as the baseline evaluation was introduced in 2009; 45 recordings of sessions were available for end-of-training and at follow-up. All course and follow-up therapy recordings were rated by course staff who were accredited CBT therapists and supervisors with formal training and extensive experience rating the CTS-R; assessors were blind to clinical outcome, subject, and session number. Additional CTS-R rater training was not provided for the study as all markers were course staff who had already been trained to use the CTS-R for assessment. All staff who join the course are provided with induction training in using the CTS-R, which includes blind marking multiple recordings with an experienced marker until high inter-rater agreement is achieved. Following the initial training, on-going reliability monitoring of CTS-R ratings comprises second-ratings on all tapes below the competence threshold (CTS-R < 36) and on a random selection of approximately 20% for each tape submission to check rater drift. As reported above, both internal and external interrater reliability was high for randomly-selected tapes in the present study.   Figure 1 reports the mean total and subscale CTS-R scores across time; both total and subscale CTS-R scores increased significantly across training and were maintained at follow-up.

Therapist Competence at Follow-Up.
At follow-up, 85% (n = 38) of therapists remained competent and 15% (n=7) were non-competent. At follow-up, no significant (p>.05) differences were found between competent and non-competent therapists using chi-square tests for gender, highest educational qualification, and profession, and Mann-Whitney test for baseline CTS-R total or subscale scores, and end-of-training total or subscale CTS-R scores. Follow-up total and subscale CTS-R scores were significantly greater for competent than non-competent therapists, reported in Table 1.  Abbreviations: PHQ-9 = Patient Health Questionnaire (Kroenke et al., 2001). GAD-7 = Generalized Anxiety Disorder-7 (Spitzer et al., 2006). RI = Reliable Improvement. Table 3 reports Spearman correlations between total and subscale CTS-R scores and clinical outcomes for end-of-training and follow-up. There were no significant correlations.

Relationship between CTS-R Scores and Clinical Outcomes
There was a positive trend (p=.06) between the end-of-training total CTS-R and PHQ-9 effect size. At follow-up, the total CTS-R exhibited positive trends (p=.05-.09) with RI rate and GAD-7 effect size. There were also trends at follow-up (p=.07-.08) between the generic and specific subscales with RI rate, and the specific subscale with GAD-7 effect size.  (Kroenke et al., 2011); GAD-7 = Generalised Anxiety Disorder 7 (Spitzer et al., 2006) Mann-Whitney tests found no significant differences between competent and noncompetent therapists for clinical outcomes, reported in Table 4. There was a trend (p=.06) for greater PHQ-9 effect sizes for competent therapists but overall clinical outcomes were stable from training to follow up in both groups. Abbreviations: RI = reliable improvement; PHQ-9 = Patient Health Questionnaire (Kroenke et al., 2011); GAD-7 = Generalised Anxiety Disorder 7 (Spitzer et al., 2006)

Clinical Role and Supervision
During training, therapists received weekly 90-minute group supervision from a BABCP-accredited CBT therapist for 30 weeks as part of the course. Supervision content included review of live therapy clips every session and discussion of on-going training cases.
Trainees also received weekly individual supervision from an accredited CBT therapist at their work placement site, and were required to evidence 70 hours of such supervision by the end of training.
At follow-up, 89% (n=40) of therapists were still employed in IAPT services, and 64% (n=29) of those were employed within the NHS trust in which they had trained, while 11% (n=5) were working in CBT specialist services. Supervision was provided by accredited CBT therapists in line with IAPT procedure. At follow-up, 49% (n=22) of therapists continued to receive weekly and 40% (n=18) fortnightly clinical supervision, with a median duration of 1.5 hours (IQR = 1 hour). Competent therapists did not differ significantly from non-competent therapists in the frequency of supervision received: U=124.50, p=.77. Unlike course supervision where audio tape therapy review is an integral part of supervision, only 27% (n=12) of therapists reported continued use of live supervision methods, i.e. review of actual therapy sessions at follow-up. While 29% (n =11/38) of the competent therapists at follow-up still reviewed tapes in supervision compared to 14% (n =1/7) of the non-competent therapists, this difference was not significant: exact X 2 =.65, p=.39.
Therapists reported offering an adequate course of therapy with a mean of 11.82 (SD=3.20, range = 8-19) treatment sessions on the course, and 12.40 (SD=5.02, range 7-23) at follow-up.

Discussion
This study aimed to assess if trainee therapists' gains in competence and clinical outcomes attained during IAPT CBT training were maintained post-qualification in the workplace, and to investigate the relationship between competence and clinical outcomes at both time points. CBT competence was evaluated at the beginning and end of training, and at least twelve months following qualification using the CTS-R. All trainees achieved a recognised minimum level of competence by the end of training, and the majority maintained that level of competence at follow-up. Trainee clinical outcome was also examined, with course and follow-up cases demonstrating significant improvement across all measures despite an increase in complex presentations at follow-up.

CBT Competence
IAPT trainees achieved therapy competence by the end of the course, and the majority maintained competence when working as qualified therapists, notably in line with the expected competence threshold (Blackburn et al., 2001;Branson et al., 2015;Keen & Freeston, 2008). While baseline CTS-R scores were lower in the present study (M=30.45) than in some other CBT training studies (35.72-38.94; see Barnfield, Mathieson, & Beaumont, 2007;Branson et al., 2015;Keen & Freeston, 2008), trainees obtained competence in line with previous literature by the end of training. The CTS-R subscales for generic and CBT-specific items also improved and were maintained at follow-up, adding to the evidence for post-training maintenance of both generic and CBT-specific competence (Sholomskas et al., 2005;Simons et al., 2010). The subset who did not maintain competence post-training was small, which makes conclusions difficult. The seven therapists who were non-competent at follow-up nearly reached CTS-R competence (36) with a median of 33.50, perhaps indicating drift rather than major relapse and that additional input may have enabled them to maintain competence.

Therapist Competence and Patient Outcome
No significant relationship between therapist competence and patient outcome was found at the end of training or at follow up, although positive trends emerged. Previous training studies reported similar results (Branson et al., 2015;Jolley et al., 2015;Liness et al., under review). There was also no significant difference in patient outcome between competent and non-competent therapists at follow up. The absence of significant findings was likely influenced by the small number of non-competent therapists (n=7) at follow-up, a somewhat ironic consequence of so many therapists retaining competence. The lack of an association of therapist competence to patient outcome may be due to the restriction of range in therapist competence (Öst et al., 2012). Other factors that may influence the competenceoutcome relationship and patient outcome at any given time including therapeutic alliance (Webb et al., 2010) and patient characteristics such as symptom severity and chronicity,

Clinical Role and Supervision
The majority of therapists in this study continued to work in IAPT services posttraining and a substantial number remained in the same NHS trust. Patient characteristics at follow-up were similar to training cases, with complexity increased slightly as expected.
Trainees continuing to work in similar workplace settings and see similar patient presentations may assist the consolidation of therapist competence, and contrasts with previous large-scale training initiatives where skill maintenance difficulties were partially attributed to a lack of continuity from training to the work environment (Brooker, Saul, Robinson, King and Dudley, 2003). Therapists' continued use of regular expert supervision may also have supported maintenance of competence (Mannix et al., 2006;Rakovshik et al., 2016). Therapists however reported reduced use of 'live' supervision (i.e. review of therapy tapes). This is a key characteristic of training supervision, and its quick reduction postqualification is noteworthy as therapists have higher and more complex caseloads. Only 1/7 non-competent therapists reported that supervision included feedback on audio recordings of

Limitations
Present findings have the constraints of a naturalistic study with no control condition.
Therapists self-selected audiotaped sessions that were submitted for assessment and also patients that they brought to weekly supervision and subsequently included in their case reports. The representativeness of both the audio sessions and patients is unknown as submitted cases only partially reflect overall caseloads. However, trainees were instructed to select representative audio tapes, and patient selection occurred before clinical outcome was known. Clinical outcome at follow-up was requested for the most recent eight cases irrespective of outcome. Several therapists were unable to complete the study due to workload, consequently the sample size was smaller than intended. Delay in ethics approval also affected participation and led to varied timings of follow up submissions for the earlier cohorts. Recruitment, ethics approval and attrition are common difficulties in workplace research (Brosan, Reynolds, & Moore, 2007;Stirman et al., 2015). Further evaluation of training effects and maintenance of competence is crucial for the sustainability of workforce training such as the IAPT programme, and perhaps merits increased resources and funding for services to enable timely research and busy therapists' participation.

Conclusions and Future Directions
This is the first study to our knowledge that evaluates CBT therapist competence and patient outcomes from the IAPT workforce training programme during training and at follow-up. The maintenance of therapists' gains post-training is promising for large-scale evidence-based training worldwide. The highly-structured IAPT set-up that encourages training and service collaboration, and emphasizes the importance of regular clinical supervision both during and after training, possibly has a role to play in the sustainability of competence. The generalizability of this structure to less resource-rich environments is unknown and requires further long-term evaluation and access to the necessary resources to increase participation by busy practitioners. The focus on alternative clinical assessment and supervision methods, the training of non-mental health professionals, and the creation of online resources are all encouraging developments for embedding evidence-based practices worldwide (Fairburn & Patel, 2014;Kobak, Wolitzky-Taylor, Craske, & Rose, 2017).
Overall, this study demonstrates that a large-scale training programme can equip therapists with sustainable competence to address the challenge of mental health in the population.