Administration and Policy in Mental Health and Mental Health Services Research

, Volume 42, Issue 1, pp 87–98

Effectiveness of Cognitive Behavioral Therapy in Public Mental Health: Comparison to Treatment as Usual for Treatment-Resistant Depression

Authors

    • School of Social WorkThe University of Texas at Austin
  • Monica A. Basco
    • Center for Scientific ReviewNational Institute of Health
Original Article

DOI: 10.1007/s10488-014-0546-4

Cite this article as:
Lopez, M.A. & Basco, M.A. Adm Policy Ment Health (2015) 42: 87. doi:10.1007/s10488-014-0546-4

Abstract

State mental health systems have been leaders in the implementation of evidence-based approaches to care for individuals with severe mental illness. Numerous case studies of the wide-scale implementation of research-supported models such as integrated dual diagnosis treatment and assertive community treatment are documented. However, relatively few dissemination efforts have focused on cognitive behavioral therapy (CBT) for individuals with major depression despite evidence indicating its efficacy with this population. A multi-site effectiveness trial of CBT was conducted within the Texas public mental health system. Eighty-three adults with major depression received CBT from community clinicians trained through a workshop and regular consultation with a master clinician. Outcomes were compared to a matched sample of individuals receiving pharmacotherapy. Outcome measures used included the quick inventory of depressive symptomatology and beck depression inventory. Individuals receiving CBT showed greater improvements in depression symptoms than those in the comparison group. Greater pre-treatment symptom severity predicted better treatment response, while the presence of comorbid personality disorders was associated with poorer outcomes.

Keywords

Cognitive behavioral therapyDepressionEvidence-based practicesEffectiveness

Introduction

Cognitive Therapy (CT; Beck et al. 1979) is an empirically supported treatment for major depressive disorder (MDD). It has generally been found equivalent in efficacy to pharmacological treatments during the acute phase of treatment and superior to medication in the prevention of relapse (Butler et al. 2006; Cuijpers et al. 2013; David et al. 2008). Although cognitive behavioral therapies (CBT), such as CT, have been recommended as a first line treatment for MDD [American Psychiatric Association (APA) 2010], the national movement to encourage the dissemination of evidence-based practices (EBPs) within state mental health (MH) systems has not generally included CBT for adults with MDD. CBT is not included in the EBP toolkits funded by the Substance Abuse and Mental Health Services Administration (SAMHSA) nor is it among the seven adult EBPs on which SAMHSA requires state mental health authorities to report annually.

There are a number of possible reasons why state MH systems have not fully embraced CBT including (1) concerns about its suitability for severe depression, (2) questions about its additive benefit given the availability of pharmacological treatments, (3) possible misconceptions that manualized treatments such as CBT are commonly provided, (4) challenges to implementation given the diagnostic and psychosocial complexity of the patient population and limited treatment resources, and (5) a limited number of effectiveness trials of CBT for MDD in public settings. Without real-world effectiveness trials, policy makers lack information to determine if the research outcomes are generalizable to public mental health systems and to determine whether wide-scale implementation of this treatment modality is feasible. While recent studies might address these concerns, limited dissemination of relevant findings to decision-makers may present a barrier.

Recent Support for the Usefulness of CBT in Public MH

Severity of Depression

Although current guidelines (APA 2010) identify CBT as a first line treatment only for individuals with mild or moderate forms of depression, there are data to support its efficacy with more severe depression. Several clinical trials have recently demonstrated cognitive and behavioral therapies to be as effective as medication for adults with moderate to severe depression (DeRubeis et al. 2005; Dimidjian et al. 2006). In a meta-analysis of 67 clinical trials with nearly 6,000 patients, Cuijpers et al. (2013) found that while number of weekly sessions predicted the efficacy of psychotherapy for depression, baseline severity of symptoms had no significant effect.

Pharmacotherapy

Given the accessibility of pharmacotherapy in community MH settings, it is not unreasonable that questions have been raised regarding the added effectiveness of CBT compared to medication alone (Cuijpers et al. 2009). Some reviews of the literature have suggested that combination treatment may have some advantage over medication (or psychotherapy) alone during the acute phase of treatment, but the additional benefit is generally small (Butler et al. 2006; Feldman 2007). Based on nine randomized controlled trials comparing pharmacotherapy with combined psychotherapy/pharmacotherapy, Cuijpers et al. (2010) found an average standardized effect size of .23 (CI −.01 to .47) favoring combined treatment over medication alone.

Combination treatment has also been found to yield a more rapid response than monotherapy (Malhi et al. 2009; Manber et al. 2008), which can be critical when individuals are at risk of negative outcomes (e.g., job loss, hospitalization, suicide), and may enhance retention in treatment (Pampallona et al. 2004). Although combination treatment is more costly than monotherapy, the superior relapse prevention effects of CBT supports its cost-effectiveness over time when considering reduced direct (e.g., physical and behavioral health costs) and indirect costs (e.g., productivity; Antonunccio et al. 1997; Sava et al. 2009).

Utilization of Manualized Treatments

State leaders may assume that existing mental health providers are already providing manualized treatments such as CBT when psychotherapy is recommended. However, there is little evidence to support this assumption. While exposure to manualized treatments has become more common in clinical training programs (Weissman et al. 2006), several studies have shown that the majority of practicing clinicians report infrequent use of manualized or evidence-based interventions (Addis and Krasnow 2000; Becker et al. 2004; Mussell et al. 2000).

Challenges to Implementation

Public mental health settings, in particular, pose many unique challenges to implementation of CBT, including the complex needs of consumers, the multiple demands on the workforce and the varied organizational understanding and support for implementation (Foa et al. 2013). In addition, patients often present with complex psychosocial needs and psychiatric co-morbidities and non-adherence with treatment is common (Stirman et al. 2004). Due to these factors, policy makers may question the feasibility of implementing CBT with comparable outcomes (Addis 2002).

Effectiveness Trials

Weisz’ (2004) Deployment-Focused Model of treatment development and dissemination emphasizes the critical need for testing a clinical intervention in the setting where it will be deployed with real-world clients. In comparison with the numerous clinical trials of CBT for MDD that have provided evidence for its efficacy, there have been few effectiveness trials in community MH clinics. Merrill et al. (2003) trained clinicians to provide CBT in a community-based depression clinic. Using a benchmarking strategy, they compared the outcomes of 192 adults treated with CBT to those reported in two previously published clinical trials. They found that trained clinicians receiving clinical supervision achieved reductions in symptoms similar to those from the comparison studies.

Lopez and Basco (2011) examined the feasibility of dissemination of CBT for MDD in four Texas public MH clinics. Benchmarking was used to evaluate treatment outcomes against the Merrill et al. (2003) sample and the Cognitive Therapy treatment arms of the Sequenced Treatment Alternatives to Relieve Depression (STAR*D) trial (Thase et al. 2007). Despite being significantly more depressed and more likely to be unemployed and receiving public health insurance than the comparison samples, pre-post treatment effect sizes were comparable; 1.22 for the Beck Depression Inventory (BDI; Beck et al. 1961) and 1.24 for the Quick Inventory of Depressive Symptomatology—Self-Report, (Rush et al. 2003). Unfortunately, none of these studies included control groups, thereby limiting the conclusions that can be drawn.

Utilizing a quasi-experimental design, Simons et al. (2010) tested the feasibility of training community therapists to provide CBT for MDD. Twelve therapists at a not-for-profit community clinic in Indiana were trained in CBT with a two-day workshop followed by 16 group telephone consultations held over 1 year. Outcomes for adults treated by these therapists prior to their CBT training were compared to patients treated following their training. Results showed that those treated following therapist training in CBT had greater improvement on both the BDI and the Beck Anxiety Inventory (Beck et al. 1988) than those treated prior to training (effect sizes d = .59 and .60, respectively).

Weisz’ (2004) Deployment-Focused Model also suggests that the ideal test of the effectiveness of an intervention is its performance against treatment-as-usual (TAU). In state MH systems such as in Texas, pharmacotherapy generally constitutes TAU. Unfortunately, no community MH study has attempted to explore outcomes of participants receiving individual CBT to those receiving pharmacological treatment alone. The goal of the present study was to test of the effectiveness of CBT implemented in multiple community MH centers within Texas against pharmacotherapy for MDD.

The study design compares the outcomes of individuals receiving CBT with a matched case control group receiving pharmacological treatment. The sample represents predominantly individuals with severe and/or chronic depression, multiple comorbid diagnoses, and a variety of psychosocial support needs. The participating therapists were master’s level clinicians with varying levels of job experience, little previous experience with CBT, and multiple job demands. The primary goal was to determine whether adult outpatients with MDD receiving CBT in conjunction with pharmacotherapy in the public MH system experience a greater reduction in depressive symptoms than a matched sample who received treatment-as-usual. In addition, we explored the impact of several moderator variables on treatment effectiveness in the CBT group, including demographic characteristics, severity of depression, psychiatric co-morbidities, treatment engagement, and model fidelity.

Methods

Context of the Study

In Texas, public mental health services are provided by 39 local mental health authorities (LMHA) covering the 254 counties within the state. LMHAs are governmental entities charged with the provision of an array of mental health services to qualifying individuals within the authority’s catchment area. In 2004, Texas undertook a comprehensive redesign of the service system focused on using a standardized assessment process to identify the evidence-based services and supports that would be most likely to meet an individual’s recovery needs (Cook et al. 2004). Adults who enter services receive a diagnostic interview, a symptom-based assessment, and a measure of functioning across life domains.

For individuals diagnosed with depression, adults receive algorithm-based medication treatment (along with case coordination services) as a first-line treatment. If they fail to have full remission of their depression after two trials of medication, they become eligible for CBT. The study began during the implementation of this system redesign and provided some of the initial training of clinicians in CBT. Because LMHAs were restructuring their systems to meet these new service obligations, many individuals eligible to receive CBT according to the assessment procedures were unable to access the service due to an inadequate number of trained, licensed clinicians within the agency. This provided an opportunity to study CBT as it became available to some individuals served by the system while others continued to receive usual care.

Sample Selection

Therapists

Participating therapists were recruited from publicly-funded community mental health clinics (CMHC) in Texas. Fifteen CMHC administrators expressed an interest in participating in the research. Clinics that chose not to participate reported that they currently had too few adults eligible for psychotherapy, had no therapists eligible to participate, or did not feel they had the time to devote to a research project. Clinics that opted to participate were asked to identify one or more therapists for further consideration. Eligible therapists had to have a master’s degree in a mental health field, have a license allowing for the provision of therapy, or be under supervision pursuing professional licensure. Although 17 therapists consented to participation initially, seven either left employment or did not enroll participants in the study. Four additional therapists meeting the same eligibility criteria were recruited in a second wave to replace the initial therapists.

CBT Sample

Adults deemed eligible for psychotherapy services through the standardized assessment process and for whom a participating clinician was available were referred for screening for study eligibility. Clients who met the following criteria were approached for consent to participate: (a) age 18 or older, (b) current diagnosis of MDD, as determined through regular clinic procedures, (c) no evidence of current psychotic symptoms, and (d) Quick Inventory of Depressive Symptomatology (QIDS; Rush et al. 2003; described later) score of 11 or greater, indicating at least moderate symptomatology. Clients were excluded from participation if they had serious suicidal risk, necessitating immediate crisis services, or were diagnosed with Bipolar Disorder. Clients were not excluded based on other comorbid conditions and were permitted to receive other treatments or services as required or needed. Other services included medication management, case management, and crisis services; no other psychotherapies were provided. Very few exclusionary criteria were used in order to increase the generalizability of the results to the public mental health system. A total of 83 participants were prospectively enrolled in the study.

Treatment-as-Usual (TAU)

Since all of the participating LMHAs had too few trained staff to meet the “demand” of individuals eligible to receive CBT, a comparison group was selected from those adults who continued to receive algorithm-based medication treatment despite qualifying for combination treatment. A control group was created from the state-level database, which included diagnostic and clinical assessment information for all adults served in the system.

A propensity-scoring algorithm (Travis et al. 1996) was used to select a control group with equivalent characteristics. Only potential comparison subjects meeting study eligibility requirements were considered (e.g. age 18 or older, diagnosed with MDD, no evidence of psychosis or Bipolar Disorder, initial QIDS score of 11 or higher). Initially, individuals receiving CBT in the study were matched with potential comparison subjects on all categorical characteristics: (a) served within the same calendar year, (b) served at the same LMHA, (c) of the same gender, and (d) of the same ethnicity. Second, the authors utilized the FASTCLUS procedure within the SAS system to cluster potential control matches using the nearest centroid sorting technique (Anderberg 1973). A propensity score was calculated representing the similarity between the potential control subject and the treatment subject on the following additional variables: (a) age, (b) initial score on the QIDS, and (c) number of weeks in treatment. The individual with the smallest score, indicating the greatest similarity to the treatment participant, was selected with no control participants represented more than once. Eighty-three participants were selected for the TAU comparison group.

Therapist Training

Training for the 14 therapists consisted of a 35-hour face-to-face workshop provided by the second author, a founding fellow of the Academy of Cognitive Therapy and experienced in training novice therapists. Following the workshop, therapists participated in 6–9 months of twice weekly group telephone supervision with the expert trainer focused on further skills development. Therapists were expected to participate at least weekly in the calls, and exceeded this expectation by attending 57.3 % of the possible calls.

Treatment

Treatment for the CBT group consisted of 18 individual sessions of CBT (Beck et al. 1979) utilizing a protocol adapted from Wright et al. (2005). Although based on Cognitive Therapy, both the training and the treatment protocol emphasized the role of behavioral techniques, such as behavioral activation and problem solving. Therapists were also given a standard protocol for responding to missed appointments. Therapists audiotaped all therapy sessions with client consent.

A random sample of audiotapes, stratified across therapist and session number were selected to examine CBT fidelity. Independent raters, certified in CBT by the Beck Institute for Cognitive Behavior Therapy, received training to enhance reliability and then scored audiotapes for adherence to the CBT protocol as well as therapist competency. Ratings of adherence were measured with a scale assessing percent completion of session structuring activities (e.g. set agenda, assign homework) and recommended interventions (e.g., thought recording, activity scheduling) on a scale of 0 = not completed, 1 = partial completion, and 2 = full completion. Therapist competency was assessed with the Cognitive Therapy Rating Scale (Young and Beck 1980). Inter-rater reliability was measured on 21 % of the tape sample with intraclass correlations [ICC(1,1)] using a one-way random effects model. Inter-rater reliability was moderate for the CTRS with ICC = .62 and good for the Adherence Scale with ICC = .79. Therapists were found to be moderately adherent, completing 75.2 % of the session structuring elements and 74.9 % of recommended CBT interventions. Therapist competence on the CTRS was also moderate, with an average score of 35.4 (SD = 9.4). This is below the commonly used CTRS cut-off of 39 applied in clinical trials, but all but one of the therapists had a mean CTRS within one standard deviation of the cut-off, suggesting reasonable proficiency with CBT.

Participants in the CBT sample continued to receive pharmacological and case coordination services and have crisis services available as needed. The study captured the pharmacological treatment of participants receiving CBT, but did not attempt to impact medication treatment in any way. Treatment choices were generally consistent with the medication algorithms for MDD implemented in the treatment system and indicative of the treatment resistant nature of the depression experienced by most participants. Similarly, all individuals within the TAU group received algorithm-based medication services and case coordination; none received psychotherapy during the study period.

Participant Characteristics

Of the 166 participants, 84.3 % were female and 15.7 % were male. Participants ranged in age from 19 to 74 years, with a mean age of 43.1 (SD = 12.8). Approximately half of the participants (50.6 %) identified themselves as Caucasian, with 39.8 % identifying themselves as Hispanic/Latino, 8.4 % as African-American, and 1.2 % as Asian. All study participants were diagnosed by licensed clinicians using usual clinic assessment procedures. All received a diagnosis of MDD (84.9 % with recurrent episodes); 63 % with severe depression and 25.3 % with moderate depression. The majority of participants (55.4 %) had comorbid psychiatric diagnoses, with anxiety and substance related disorders the most common. Table 1 summarizes the demographic and clinical characteristics of participants.
Table 1

Sample characteristics

Variable

CT

n = 83

TAU

n = 83

Comparability statistic

Agea (mean + SD)

42.8 ± 13.2

43.2 ± 12.4

t = .17, p = .87

Gendera (female)

70 (84.3 %)

70 (84.3 %)

χ2 = 0, p = 1.0

Race/ethnicitya

  

χ2 = 0, p = 1.0

 Caucasian

42 (50.6 %)

42 (50.6 %)

 

 Black

7 (8.4 %)

7 (8.4 %)

 

 American Indian

0

0

 

 Asian American

1 (1.2 %)

1 (1.2 %)

 

 Hispanic or Latino

33 (39.8 %)

33 (39.8 %)

 

Marital status

   

 Never married

22 (26.5 %)

Not available

 

 Married

14 (16.9 %)

  

 Separated

12 (14.5 %)

  

 Divorced

33 (39.8 %)

  

 Widowed

2 (2.4 %)

  

Medicaid

25 (30.1 %)

26 (31.3 %)

χ2 = .03, p = .87

Employment

  

χ2 = 1.2, p = .54

 Employed

12 (14.46 %)

15 (18.1 %)

 

 Unemployed

11 (13.25 %)

7 (8.4 %)

 

 Not in labor force

60 (72.29 %)

61 (73.5 %)

 

Residence

  

χ2 = 1.7, p = .19

 Independent or family home

80 (96.4 %)

76 (91.6 %)

 

 Temporary housing or homeless

3 (3.6 %)

7 (8.4 %)

 

Psychiatric comorbidities

47 (56.6 %)

46 (55.4 %)

χ2 = .02, p = .88

 Anxiety

29 (34.9 %)

27 (32.5 %)

χ2 = .11, p = .74

 Substance-related

18 (21.7 %)

15 (18.1 %)

χ2 = .34, p = .56

 Personality disorder

10 (12.0 %)

16 (19.3 %)

χ2 = 1.6, p = .20

Baseline QIDS–SR scorea, (mean + SD)

18.3 ± 3.9

17.4 ± 3.8

t = −1.9, p = .11

Weeks serveda (mean + SD)

18.7 ± 12.5

19.4 ± 12.0

t = .34, p = .73

CT = Participants receiving Cognitive Therapy in combination with pharmacotherapy, TAU = Matched control group receiving treatment as usual (pharmacotherapy)

aVariables used in matching procedures

Measurement

All participants completed the Quick Inventory of Depressive Symptomatology—Self Report (QIDS–SR; Rush et al. 2003), a 16-item scale that measures depression symptoms identified in the Diagnostic and Statistical Manual 4th edition (APA 1994). The QIDS–SR has been shown to have good internal consistency, validity, and sensitivity to change. It has been shown to be correlated with other commonly used measures of depression (Rush et al. 2003). Scores of 5 or less on the QIDS–SR represent no depressive symptoms, and scores between 6 and 10 indicate mild symptoms. Moderate symptoms are reflected by scores of 11–15; severe by 16–20, and very severe by 21–27. The QIDS–SR was chosen as the primary outcome measure due to its use in the Texas system as well as its ability to assess changes in depressive symptoms. Within the CBT group, the QIDS–SR was administered at all therapy visits, while the TAU participants completed the QIDS–SR at intake and at each medication visit.

Additional measures were collected from the CBT group (but not the TAU group) at the 1st, 5th, 10th, 15th, and final visits. The clinician version of the QIDS was administered to CBT participants during a phone interview conducted by research staff and served as an independent assessment of outcome. Participants receiving CBT also completed the Beck Depression Inventory-II (BDI), a 21 item self-report measure that assesses affect, cognition, behavior, and functioning and has been shown to have excellent psychometric properties (Beck et al. 1996). The BDI Total Score ranges from 0–63 with 0–13 indicating no or minimal depression, 14 to 19 representing mild depression, 20–28 indicating moderate depression, and 29–63 reflecting severe depression. All research procedures were approved by the Texas Department of State Health Services Institutional Review Board and informed consent to participate was provided by all study participants.

Analysis

Univariate analyses (e.g. Chi square, independent t test) were conducted to examine any differences between the TAU and CBT groups on demographic and clinical characteristics to ensure the matching procedures were effective. All hypotheses were evaluated by fitting hierarchical linear models (HLM; Raudenbush and Bryk 2002). Level 1 coefficients were evaluated using the Akaike Information Criterion (AIC) following guidelines for model comparisons (i.e., models that differ by less than 2 on AIC are equivalent models) from Burnham and Anderson (2002) to compare models with and without random slopes to assess whether fixed or random coefficients were more appropriate. The same guidelines were used to assess therapist and site as potential Level 3 and Level 4 random effects. For models containing intervention and control cases, a partially clustered design was used (Baldwin et al. 2011; Bauer et al. 2008), which accommodates data in which participants in one condition are in cluster (e.g., therapist) that does not exist for participants in another condition (e.g., control group participants). The partially clustered design fits a fixed intercept for all participants in the data and a random intervention effect, which effectively serves as a random intercept for intervention participants only and captures variability associated with therapist or site. This approach is appropriate when multiple assessments across time are nested within individuals and has the advantage of effectively managing designs in which the number and timing of assessments varies. The HLM procedure models person-specific random effects, representing the degree to which each individual’s initial status (intercept) and trajectory over time (slope) vary from the mean intercept and slope.

The primary research question involves the degree to which group membership (CBT vs. TAU) predicts within-subject change in depressive symptomatology (QIDS scores) over time. To examine these effects, a partially clustered three-level model was estimated with repeated measures of QIDS–SR at Level 1, which were nested within participants at Level 2, who were nested within therapists at Level 3. Linear and quadratic days from initial assessment were included in the Level 1 equation and group membership included as a Level 2 predictor for all Level 1 coefficients. To better understand for whom CBT is most effective, a second set of HLM models was created to examine the role of selected demographic, clinical, and treatment process variables. Four separate three-level models were created utilizing repeated measures of QIDS–SR in Level 1. Potential predictors were grouped based on logical categories and entered into Level 2 or Level 3 of the HLM model.

Results

Preliminary Analyses

No significant differences were found between the TAU and CBT groups on any baseline demographic or clinical measures or on the number of weeks in treatment (See Table 1). Baseline symptomatology measures reflected severe depression on average, with a mean QIDS–SR score of 17.9 (SD = 3.9). This was mirrored in the mean QIDS–C scores of 17.8 (SD = 4.2) and BDI scores of 38.9 (SD = 10.2) at study entry for the CBT sample. The average length of time in treatment was 18.7 weeks (SD = 12.5 weeks) for the CBT group and 19.4 weeks (SD = 12.0 weeks) for TAU. Participants in the CBT group attended an average of 11.2 treatment sessions (SD = 7.0). Thirty-one percent of the CBT participants completed the full 18–20 sessions, with 16.9 % failing to return after 1 or 2 treatment sessions.

Assessment of Random Effects

The following four models containing only time in days and quadratic time in days parameters were fit to evaluate whether random terms were necessary for Level 1 coefficients: a random intercept model (AIC = 6,336.43); a random intercept and linear time model (AIC = 6,268.63); a random intercept and quadratic time model (AIC = 6,302.07); and a random intercept, linear time, and quadratic time model (AIC = 6,245.37). Because the random intercept, linear time, and quadratic time model had the lowest AIC, each of these coefficients were treated as random effects in all models reported herein. Random intercepts for therapist and site were evaluated by fitting the following unconditional models for intervention participants, all of which contained random intercepts for participants: therapist random intercept model (AIC = 5,044.06), site random intercept model (AIC = 5,045.76), and therapist nested within site (AIC = 5,046.06). Because the models were equivalent in their information criteria, the therapist random intercepts model was used in all models.

Participant Depression Outcomes

A review of participants’ QIDS–SR scores over time suggested that a quadratic equation represented the data more accurately than a linear model. A preliminary HLM analysis included only the outcome measure and time variables (Level 1). This analysis suggested that participants (CBT and TAU combined) had significant improvements in depression severity over time (β = −.06; SE = .01, t = −11.37, p < .0001) with the slope steeper at the beginning of treatment and declining over time (β = .00, SE = .00, t = 8.19, p < .0001). Group membership (CBT vs. TAU) was then added into the HLM model in Level 2 to test the primary hypothesis. Participants in the CBT group demonstrated a greater reduction in depression symptoms over time than participants in the TAU group (β = −.02; SE = .01, t = −2.09, p = .037). The slope of the CBT group leveled off more than the slope of the TAU group (β = .00; SE = .00, t = 2.19, p = .029). Results from the analysis are presented in Table 2 and the model is represented in Fig. 1.
Table 2

Hierarchical linear model analysis with quadratic growth curve: rates of change by group status

Variable

Parameter estimate

SE

t

Degrees of freedom

p

Predictors of intercept

     

 Intercept

16.78

0.47

36.06

959

<.001

 CT group status

0.12

0.82

0.15

94

.879

Predictors of slope (days)

     

 Intercept

−0.05

0.01

−6.46

959

<.001

 CT group status

−0.02

0.01

−2.09

959

.037

Predictors of Slope (days squared)

    

 Intercept

0.00

0.00

4.58

959

<.001

 CT group status

0.00

0.00

2.19

959

.029

CT group status reflects a dummy coded variable with participants receiving cognitive therapy assigned a code of 1 and those in treatment as usual assigned a code of 0

https://static-content.springer.com/image/art%3A10.1007%2Fs10488-014-0546-4/MediaObjects/10488_2014_546_Fig1_HTML.gif
Fig. 1

Change in depression symptoms over time for treatment groups

In addition to the rate of change in depression severity, it is important to explore the degree to which treatment leads to a clinically significant reduction in symptomatology as well as the ultimate goal of full remission of depression. Utilizing the final QIDS–SR score, 36.7 % of participants in the CBT group had a clinically significant response to treatment (represented by a 50 % or greater decrease in initial QIDS score) compared to 22.9 % of those in the TAU group (χ2 = 3.71, df = 1, p = .05). Twenty-four percent of participants in the CBT group experienced full remission of symptoms (QIDS < 6) compared to only 12.1 % of those in the TAU group (χ2 = 3.97, df = 1, p = .05).

Similar patterns of results were found for other measures of depression completed by the CBT participants only. CBT participants demonstrated significant improvement between baseline and the last available observation on the BDI (t = 8.10, df = 79, p < .0001, d = 1.00) and QIDS–C (t = 5.42, df = 68, p < .0001, d = .75), representing a large and medium effect size respectively (Cohen 1988). Utilizing the BDI, 57.5 % of the study participants demonstrated a clinically meaningful response to treatment (defined as change of 6 or more points) and 16.3 % reached full remission of symptoms (BDI score < 14). Based on results using the QIDS–C, 21.7 % had a significant response (50 % or greater decrease in initial QIDS) and 18.8 % had full remission (QIDS–C < 6).

Predictors of Outcome in CBT

Four sets of variables were explored as potential predictors of the rate of improvement in depression for participants in the CBT group. Each set of variables was entered into a separate HLM model with Level 1 representing change in depression over time as measured by the QIDS–SR and Level 2 or Level 3 representing the possible predictor variables within that category. The results of these analyses are presented in Table 3. Within the demographic characteristics of the sample (Model 1), age and gender were found to be significant predictors of the rate of improvement in CBT. Younger participants were found to improve more quickly than older participants (β = .00, SE = .00, t = 2.46, p = .014) with the advantage for younger individuals becoming smaller as treatment progressed (β = −.00, SE = .00, t = −3.33, p < .001). Female participants improved more quickly than male participants (β = .05, SE = .02, t = −2.14, p = .033), but their rate of improvement lessened over time, while male participants continued to improve in a more linear fashion (β = −.00, SE = .00, t = −2.87, p = .004). Neither ethnicity (Latino/a vs. non-Latino/a) nor marital status (married vs. not married) had a significant impact on the rate of improvement.
Table 3

Hierarchical linear model analysis with quadratic growth curve: predictors of change in depression symptoms in participants receiving cognitive therapy

Variable

Parameter estimate

SE

t

Degrees of freedom

p

Model 1: demographic variables

     

 Predictors of intercept

     

  Intercept

17.18

0.87

19.72

713

<.001

  Age

−0.02

0.04

−0.44

58

.663

  Gender

−1.24

1.27

−0.98

58

.333

  Ethnicity (Latino vs. non-Latino)

0.70

1.05

0.67

58

.505

  Marital status

−1.87

1.26

−1.48

58

.144

 Predictors of slope (days)

     

  Intercept

−0.09

0.01

−7.27

713

<.001

  Age

0.00

0.00

2.46

713

.014

  Gender

0.05

0.02

2.14

713

.033

  Ethnicity (Latino vs. non-Latino)

0.01

0.02

0.57

713

.572

  Marital status

0.01

0.02

0.35

713

.730

 Predictors of slope (days squared)

     

  Intercept

0.00

0.00

5.37

713

<.001

  Age

−0.00

0.00

−3.33

713

<.001

  Gender

−0.00

0.00

−2.87

713

.004

  Ethnicity (Latino vs. non-Latino)

−0.00

0.00

−0.58

713

.565

  Marital status

0.00

0.00

0.62

713

.538

Model 2: clinical severity variables

   

 Predictors of Intercept

     

  Intercept

17.36

0.82

21.18

764

<.001

  Hx of psychiatric hospitalization

−0.54

0.98

−0.55

65

.585

  Substance-related comorbidity

−1.27

1.11

−1.15

65

.255

  Personality disorder comorbidity

0.40

1.52

0.26

65

.792

 Predictors of slope (days)

     

  Intercept

−0.07

0.01

−6.08

764

<.001

  Initial QIDS score

−0.00

0.00

−2.11

764

.036

  Hx of psychiatric hospitalization

−0.03

0.02

−2.00

764

.046

  Substance-related comorbidity

0.02

0.02

0.97

764

.333

  Personality disorder comorbidity

0.05

0.03

1.85

764

.064

 Predictors of slope (days squared)

     

  Intercept

0.00

0.00

3.41

764

<.001

  Initial QIDS score

0.00

0.00

2.21

764

.028

  Hx of psychiatric hospitalization

0.00

0.00

1.88

764

.060

  Substance-related comorbidity

0.00

0.00

0.09

764

.925

  Personality disorder comorbidity

−0.00

0.00

−2.27

764

.023

Model 3: treatment adherence

     

 Predictors of intercept

     

  Intercept

16.39

0.51

31.93

772

<.001

  Percent of sessions attended

−2.91

2.91

−1

58

.322

  Average homework compliance

−0.34

1.07

−0.32

58

.753

 Predictors of slope (days)

     

  Intercept

−0.08

0.01

−9.11

772

<.001

  Percent of sessions attended

0.04

0.06

0.7

772

.482

  Average homework compliance

−0.03

0.02

−1.25

772

.210

 Predictors of slope (days squared)

     

  Intercept

0.00

0.00

6.1

772

<.001

  Percent of sessions attended

−0.00

0.00

−0.45

772

.653

  Average homework compliance

0.00

0.00

0.21

772

.832

Model 4: facilitator adherence and competence

   

 Predictors of intercept

     

  Intercept

16.62

0.57

29.12

784

<.001

  Facilitator adherence

−9.42

9.53

−0.99

11

.344

  Facilitator competence

−0.04

0.21

−0.19

11

.851

 Predictors of slope (days)

     

  Intercept

−0.08

0.01

−8.81

784

<.001

  Facilitator adherence

−0.16

0.15

−1.06

784

.290

  Facilitator competence

0.00

0.00

1.03

784

.302

 Predictors of slope (days squared)

     

  Intercept

0.00

0.00

5.64

784

<.001

  Facilitator adherence

0.00

0.00

1.38

784

.168

  Facilitator competence

−0.00

0.00

−0.74

784

.460

A second analysis examined the impact of clinical severity indicators (Model 2), with the results shown in Table 3. Participants reporting more severe depression on the QIDS–SR at study entry had a more rapid decline in symptom severity than those with lower baseline scores (β = −.00, SE = .00, t = −2.11, p = .036) and their response leveled off earlier than those with less severe symptoms (β = .00, SE = .00, t = 2.21, p = .028). Similarly, those participants reporting psychiatric hospitalization in their lifetime showed a more rapid improvement in depression severity (β = −.03, SE = .02, t = −2.11, p = .046), but slopes did not differ based on history of psychiatric hospitalization. In contrast, participants with and without a comorbid personality disorder had a similar rate of improvement, but the rate of improvement for individuals with a personality disorder lessened at a higher severity level (β = −.00, SE = .00, t = −2.27, p = .023). Comorbid substance-related disorders did not have a statistically significant impact on treatment response.

The third model explored indicators of engagement in the treatment process with results presented in Table 3. The percent of sessions attended (vs. those cancelled or missed) did not have a significant impact on treatment response. Similarly, therapist ratings of homework completion by participants (none, partial, full) across sessions was not a significant predictor of the rate of improvement in treatment. A final model (Model 4) explored the impact of two therapist-level variables—adherence to the session protocol and therapist competence in CBT. Neither adherence nor competence were significant predictors of treatment response.

Discussion

This study provided further support for the contention that CBT is an effective treatment for MDD within publicly-funded community mental health settings. Participants in the study showed a more rapid reduction in depressive symptoms over time than a matched sample of individuals receiving treatment as usual. Significantly more CBT participants had clinically meaningful reductions in symptoms and reached full symptom remission. However, it should be noted that full symptom remission was not common within this sample and that response and remission rates were similar to those found in the augmentation with CT arm of the STAR*D trial (Sinyor et al. 2010). This adds to the literature suggesting the need for an array of evidence-based interventions to be deployed either consecutively or in combination to meet the unique needs of individuals who fail to achieve remission with the initial treatment offering.

The literature on the predictors of treatment effectiveness with CBT is mixed, with most moderators and mediators showing a relationship in some studies and not others. In this study, females showed a better response to CBT, while most research has suggested equivalent efficacy by gender (Thase et al. 1994; Hamilton and Dobson 2002). Although some prior research has found marriage to predict greater response to CBT, this study found no differences based on marital status. Similarly, there was no impact of substance abuse or personality disorder comorbidities on treatment outcomes, which is inconsistent with findings by Fournier et al. (2008) that patients with personality disorders responded to medication therapy better than psychotherapy with the reverse trend for those without personality disorders.

In this sample, younger participants showed greater improvement, consistent with other research suggesting a benefit to early intervention (Fournier et al. 2009). Since all individuals in this study had failed at least two medication trials prior to being eligible for CBT and most had years of prior treatment, this suggests that there may be an advantage to providing combination treatment as soon as possible for individuals with severe or chronic depression. In addition, those with more severe initial symptoms and a greater number of prior hospitalization days showed greater improvement. Although this may be due in part or whole to the regression to the mean phenomenon, it is consistent with the meta-regression analysis conducted by Driessen et al. (2010) which concluded that effect sizes for psychological treatments for depression have been significantly larger for patients with greater baseline symptom severity.

The study sample included a large percentage of individuals identifying themselves as Hispanic or Latino. In fact, several participating therapists were bilingual and provided CBT in Spanish and participant materials were provided in Spanish. The analysis indicated that CBT showed no difference in effectiveness between Hispanic and non-Hispanic participants. This is an important finding for public mental health clinics, which in many states serve a disproportionate number of racial and ethnic minorities. Few studies to date have examined the comparative efficacy of CBT for depression across ethnic groups (Horrell 2008), but one large study by Miranda et al. (2003) found CBT to be equally effective for Hispanic/Latino individuals and Whites. Measures of participation in the treatment (missed appointments, homework compliance) did not predict treatment effectiveness. This finding is in contrast to a meta-analysis suggesting small but significant effects of homework compliance on treatment outcomes (Kazantzis et al. 2000). Lastly, measures of protocol adherence and therapist competence did not predict treatment response in this sample. While several studies have demonstrated small, but significant relationships between adherence or competence and outcomes, findings have been inconsistent, perhaps in part because of the complexity of measuring these constructs (Barber et al. 2007; Strunk et al. 2010).

This study adds to a growing literature supporting the effectiveness of CBT within real-world settings and illustrates some of the complexity of implementing evidence-based practices within public mental health clinics. In resource-limited settings, psychotherapy is likely to be reserved for individuals who are insufficiently treated with medication, resulting in a sample that varies significantly from the populations generally reported in efficacy trials. The study sample was significantly more symptomatic and had more comorbid diagnoses than those included in similar studies (Merrill et al. 2003; Thase et al. 2007). Therapists implementing the model generally lacked prior experience using CBT, had large caseloads, and served in multiple clinical and administrative roles within their clinics, which made learning a new intervention challenging. Although there was significant variability in competency scores, average fidelity ratings across the therapists approached those considered “competent” in published efficacy trials. Customary cut-off criteria on such measures were developed in early efficacy trials, and there is little research guidance to indicate what level of competency should be expected to ensure similar outcomes in real-world settings with adults with such complex mental health needs.

As suggested by the Deployment Focused Model (Weisz 2004), the ultimate test of an intervention is when it is conducted by providers in the real-world setting, with real-world clients against treatment-as-usual. The real-world nature of the study represents one of its most significant strengths. The study sample was representative of the Texas public mental health system and generalizability is maximized by the minimal exclusionary criteria. Self-report outcome measures were utilized, which is likely to be standard practice in public mental health settings, but included external researcher-derived outcome measures as well to assess the reliability of findings.

The study design also had a number of weaknesses that must be considered in interpretation of the findings. The quasi-experimental design utilizing propensity-score matching does not allow for full testing of the causal impact of CBT. It is possible that there were some differences between individuals who received CBT and those who did not that impacted outcomes, such as motivation for treatment, that could not be controlled for with available measures. Diagnoses were not derived with a gold-standard diagnostic interview and the use of clinician-derived diagnoses may complicate interpretation. Outcomes were assessed over the course of treatment and follow-up measures were not available, thus limiting the ability to examine differences in the maintenance of effects. In addition, although the study included a rigorous measurement of therapist competency and adherence by external experts, competency did not consistently reach traditional cut-offs. However, it seems reasonable to conclude that the current findings of effectiveness are unlikely to be diminished if higher levels of competency were achieved.

The study also illustrates some of the challenges of implementing an evidence-based psychotherapy within public mental health settings. Several of the clinics initially approached about participation in the research declined, primarily due to a lack of readiness to implement CBT (e.g., no therapists hired, failure to identify appropriate clients, lack of buy-in). Most of the prominent frameworks for practice implementation point to the importance of activities to ready the organization for implementation (Meyers et al. 2012; Fixsen et al. 2005; Rogers 2003). Although all of the state public mental health clinics were required to implement CBT, many were not ready to begin active implementation efforts when approached by the researchers.

Therapist training efforts were both time-intensive and somewhat costly. Therapist participation on supervision calls was greater than the stated expectation (at least once per week) and therapists stated that they appreciated the frequency, however organizations may be challenged to support this level of training without external funding. To sustain implementation through staff turnover, organizations will likely need to develop internal expertise to provide on-going supervision. This is a common model of practice dissemination, yet research has not adequately addressed the effectiveness of second (or third) generation coaching. Additionally, training resources can be wasted when staff participate in intensive training but do not ultimately implement the practice. In this study, seven of the seventeen therapists consenting to participation did not provide CBT to study participants. Several factors contribute to this challenge, including selection of appropriate staff for implementation, retention of staff, and effective support. In this study, several therapists took other positions both within and outside of the participating clinics. In fact, mental health clinics may increase staff turnover by providing therapists with skills that are in high demand within the community. To improve wide-scale dissemination, research should continue to identify cost-effective ways to develop therapist competencies, such as through blended training models or computer simulations, as well as identify key characteristics of staff most likely to develop adequate competence.

Research on the significant impact of major depression on quality of life and cost to society suggests the need for an effective public health approach. Public mental health systems, which serve as the primary health home for many adults with chronic or severe depression, would seem an important setting for ensuring access to the most effective available interventions. Along with previous research, the current study suggests that many of the reasons why state mental health systems have been slow to embrace the value of CBT or recognize the need for structured implementation efforts are not valid. Significant advances could be made through strong leadership and funding for implementation efforts by federal partners, such as SAMHSA and NIMH, the development of tools and guidelines to advance implementation of CBT, and the tracking of provision of CBT and related outcomes at a national level. State systems could support implementation through partnerships with academic institutions with experience in CBT models and implementation science, and through financial incentives supporting workforce development, high-fidelity implementation, and patient outcomes.

Acknowledgments

Research reported in this publication was supported by a grant from the National Institute of Mental Health under award number R34MH074749. The content is solely the responsibility of the authors and does not necessarily represent the official view of the National Institutes of Health. This work was prepared while Monica Basco was employed at the University of Texas at Arlington. The opinions expressed in this article are the author’s own and do not reflect the view of the National Institutes of Health, the Department of Health and Human Services, or the United States government.

Copyright information

© Springer Science+Business Media New York 2014