Mental health care provided in most practice settings uses procedures that are different from the treatments found to be effective in research settings (Institute of Medicine 2001). Quality improvement interventions are effective at increasing rates of evidence-based depression care (Katon et al. 1995, 1996; Wells et al. 2000). Unfortunately, managed behavioral health care settings are unlikely to be able to engage in large scale quality improvement interventions for psychotherapy because no efficient, easily administered measures of the extent to which psychotherapy procedures provided are evidence-based are available. In this study, we present initial findings on the development of a patient-report measure of the extent to which therapists use procedures of Cognitive Behavioral Therapy (CBT), Interpersonal Therapy (IPT), or Psychodynamic Therapy.

Psychotherapy for depression is an appropriate area to begin development of a patient-report measure of psychotherapy procedures. Depression has a high prevalence and early age of onset, making depressive disorders a leading cause of disability worldwide (Murray and Lopez 1996; Lopez et al. 2006). Two forms of well-defined psychotherapy for depression, CBT and IPT, have been well-established as effective interventions for depression (USDHHS 1999). Yet many depressed patients do not receive evidence-based psychotherapy (Wang et al. 2000; Young et al. 2001). Quality improvements efforts greatly improve the outcomes for depression (Gilbody et al. 2003), yet are not feasible for psychotherapy without ways to measure quality care. An efficient measure of the extent to which psychotherapy procedures consistent with CBT or IPT are used could enable managed behavioral health organization to document the impact of quality improvement interventions for psychotherapy for depression.

Current methods to determine if psychotherapy is evidence based are not feasible for quality improvement efforts. Trained observers code audio or videotapes of sessions to determine adherence to a particular psychotherapy. Although not a measure of psychotherapy, caregiver reported adherence to principles of MST have proven to be reliable and valid predictors of child outcome in randomized trials, as well as usual care settings (Schoenwald 2008). In fact, caregiver reports of adherence to principles of MST have proved better predictors than therapist reports of adherence (Schoenwald 2008). In this study, we reviewed existing rating scales and psychotherapy literature to develop a patient self-report measure of Cognitive Behavioral, Interpersonal, and Psychodynamic Therapies that could easily be administered as part of a quality improvement effort. We include Cognitive Behavioral and Interpersonal Therapies because they are well validated forms of care. We include Psychodynamic Therapy because 25% of therapists in practice use this approach (Norcross et al. 2002). The goals of our study were to present preliminary evidence of the psychometric characteristics of our patient-report measure of psychotherapy for depression.

Methods

We sought to develop a patient-report instrument to describe three psychotherapy approaches used in the treatment of depression. The goals of the analyses presented here were to evaluate the (a) psychometric properties of the measure, (b) preliminary data on validity of the scale, and (c) propose a revision of the measure based on these initial evaluations.

Measures

Psychotherapy Techniques

The instrument, the Psychotherapy Practice Scale—Patient Depression Care Version (PPS Patient), was designed to provide a tool to describe the psychotherapeutic techniques used in the treatment of depression from the patient perspective. The instrument focuses on three therapeutic approaches: CBT, IPT, and Psychodynamic Therapy. CBT and IPT techniques were included because these approaches have strong empirical support for the treatment of major depression (Butler et al. 2006; Chambless et al. 1998; Chambless and Ollendick 2001; Frank and Spanier 2006). Psychodynamic Therapy has limited empirical support but was included given the prominence of this orientation in clinical training and practice.

Content and item development was guided by review of clinical literature on efficacy and essential therapy components and existing observation coding tools of adherence and competence (including rating scales used in the NIMH Treatment of Depression Collaborative Research Program). Generated items were reviewed in consultation with at least two clinical experts in each of the three therapeutic approaches to ensure inclusion of key therapeutic techniques essential to each type of therapy. Items were further refined by conducting cognitive interviews with 12 current psychotherapy patients who completed the measure.

The questionnaire includes 30 items to assess techniques that are key components of CBT (11 items), IPT (9 items), and Psychodynamic Therapy (11 items). Item content is provided in Table 1. The instrument asked the patient to rate the frequency that their therapist used a particular technique in the course of treatment on a 7-point scale ranging from ‘never’ (1) to ‘always’ (7). Items were presented in random order.

Table 1 Item—subscale correlations
Full size table

Patient Demographic Variables

We used administrative data to determine age and gender for patients. Survey data measured patient education.

Clinician Demographic Variables

We used administrative data to determine age, gender, education level and years of clinical experience. Survey data measured ethnic/cultural identity.

Clinician Therapeutic Primary Orientation

Clinicians were asked to select their “main theoretical orientation” when treating depression. Options included Cognitive Behavioral, Interpersonal, psychoanalytic, psychodynamic, supportive, and other. Because many of our clinicians were eclectic, we developed a measure of the extent to which they practiced each of our three major orientations. Using clinician responses on the frequency with which they delivered CBT, IPT and Psychodynamic Therapy, we classified clinicians into low (<25% quartile), medium (25–75% quartile) and high (>75% quartile) categories on each of the three therapies.

Recruitment and Participants

We selected patients from OptumHealth Behavioral Solutions (formerly United Behavioral Health) who were recently diagnosed with major depression, who had at least three outpatient psychotherapy visits within the prior 6 weeks, and who had been treated by a high-volume network provider. A high-volume provider was defined as a clinician (MD, PhD, or MSW) who had treated 10 or more adult patients in the past year. We surveyed 2,417 eligible patients. Patients were sent a cover letter and consent form, patient survey instrument (described below), and an additional consent to allow us to contact their psychotherapist by mail. All patient participants received a $10 gift card. With patient permission, we then mailed survey packets (cover letter, consent form, signed copy of patient consent, and clinician survey instrument) to their treating clinician. All study procedures were approved by the RAND and UCLA Human Subjects Protection Committees.

Analyses

Examination of Psychometric Properties

We examined the internal consistency reliability (Cronbach’s alpha; Cronbach and Warrington,1951) with the goal of achieving at least an alpha of 0.7 for each of the three hypothesized scales. We also examined the correlation between hypothesized composites. We expected that the IPT and Psychodynamic Therapy composites would be most highly correlated given that IPT is derived from Psychodynamic Therapy. We expected the composites to have low to moderate intercorrelations. We examined item total scale correlations which allowed us to determine how highly an item correlated with its hypothesized scale and how that correlation related to the item’s correlation with other scales. To further determine the psychometric properties of each item, we examined the percent missing for the item and the variability of each item.

Examination of Factor Structure

Although we designed our patient report measure to have a three-factor structure, we were uncertain whether the measure would demonstrate this structure in a sample of patients seen by clinicians in practice who were predominantly eclectic in orientation. Thus, the use of exploratory factor analysis seemed the most appropriate analytic technique for this stage of measurement development research (Floyd and Widaman 1995; Reise et al. 2000; Weersing et al. 2002). Exploratory factor analyses using oblique (promax) rotation with varimax prerotation and squared multiple correlations as communality estimates priors were used to examine the initial composites and determine if the three therapeutic approaches would form three correlated subscales. The number of factors was determined using eigenvalues, adequate loadings (<0.40) on one factor with other loadings 0.20 lower on other factors, and the interpretability of the rotated factor pattern matrix.

Preliminary Validity Analyses

We examined whether clinician ratings of their own techniques predicted patient self-rated subscale scores for similar therapeutic orientations. For patient reported scales, we use subscores designed to measure CBT, IPT, and Psychodynamic Therapy. Individual items were standardized (z-scores with a mean of 0 and standard deviation of 1), to ensure that an item’s variability did not impact its weight in the subscore. Scores were transformed into T-scores, with a mean of 50 and standard deviation of 10 for ease of interpretation. We compared scores on each of the three patient-reported subscales with clinician-reported theoretical orientation (low, medium, high) using t-tests. To determine the effect of the three types of clinician reported therapy orientation on patient reported behavior, we used a multivariate analysis of variance (MANOVA) with clinician rated behavior (CBT, IPT, Psychodynamic Therapy) as independent variables and patient reported clinician behavior (CBT, IPT, Psychodynamic Therapy) as dependent variable.

Development of a Short-Form Measure

To determine whether it is possible to shorten the measure, we examined each for poor discrimination, negative impact on internal consistency, percentage of missing, and loading on the wrong factor.

Results

Participants

A total of 420 patients (17.4% response rate) and 159 high volume clinicians responded to the survey. Patients who responded were mostly adults between the age 35 and 54 (53.8%), female (74.2%), and received only psychotherapy (versus psychotherapy and medication management, 71.1% vs. 28.9%). Their clinicians were mostly master’s level therapists (62.4%), with largest percentage falling below age 30 (66.9%).

Evaluation of 30-Item Instrument

Cronbach’s alphas for each subscale were high, CBT = 0.93, IPT = 0.93 and Psychodynamic Therapy = 0.86 (Nunnally and Bernstein 1994). Scales were highly correlated with one another (r = 0.78–0.85). As predicted, the measures of the two dynamic therapies (IPT and Psychodynamic Therapy) were more highly correlated (r = 0.85) as compared with the correlation of CBT to IPT and Psychodynamic Therapy (r = 0.82 and 0.78, respectively).

Psychometric properties evaluated included item-subscale correlations, percentage of missing for each item, and variability of the item (Table 1). Results showed appropriate primary correlations (e.g., CBT technique items correlated most highly with the CBT subscale, etc.), as well as moderate secondary correlations between the subscales. Although, correlations were fairly high among all subscales, the lowest correlation was observed between CBT and Psychodynamic Therapy items. However, the highest correlation was observed for CBT and IPT items, rather than IPT and Psychodynamic Therapy items, as originally hypothesized. The Psychodynamic Therapy subscale had larger percentages of missing items than did the CBT and IPT subscales. Variability of items was relatively similar across scales.

Factor Structure

Table 2 presents the factor loadings, eigenvalues, and percentage of variance for each factor. There were two eigenvalues over one (14.74, 1.22, 0.91, 0.65, 0.54 and so on) with the third eigenvalue approaching one (0.91). When items were assigned to a factor that had standardized regression coefficients of at least 0.40, the three factor solution was interpretable. The three rotated factors accounted for 61% of the total variance. The first factor accounted for 81% of the proportion of total variance and the seven items that loaded on that factor appeared to tap a “structured, behavioral orientation,” most consistent with CBT techniques. However, it did include one item about encouraging social activities, which was originally developed as an IPT item. The second factor accounted for 7% of the proportion of total variance and seven items loaded on this factor. These items were originally Psychodynamic Therapy and IPT items and appeared to reflect a focus on past childhood experiences as well as an examination of relationships and problems, which we called “insight about relationships”. The third factor accounted for 5% of the proportion of total variance and 5 items that reflect a “general talking” factor, where patients had control over their session. Most of these items were from the Psychodynamic Therapy scale, but an item from each of the other two scales associated with “talking” loaded on this factor. There were also items that loaded on more than one factor. Items that loaded on to more than one factor were considered nonspecific with low discrimination, as listed in Table 2. Thus, three factor solution was not necessarily consistent with the original hypothesized orientations, instead they reflected (1) structured specific behavioral factor (most similar to CBT), (2) insight oriented factor (consistent with both Psyhodynamic Therapy and IPT), and (3) unstructured talking factor.

Table 2 Rotated Factor Loadings from a Principal Axis Factor Analysis of Items from the PPS (N = 420)
Full size table

Preliminary Evaluation of Validity

A multivariate analysis of variance (MANOVA) was conducted to determine the effect of the three types of clinician reported therapy orientation (CBT, IPT, and Psychodynamic Therapy) on three dependent variables (patient reported CBT, IPT or Psychodynamic Therapy). Clinician reported therapy orientation was categorized into low, medium, and high for each orientation. Significant differences were found among the clinician reported therapy subscale on the patient reported dynamic subscale (Wilks’ λ = 0.91, F (24,159) = 2.12, P < 0.05), suggesting the clinicians who self rated their therapy as higher on the Psychodynamic subscale were significantly more likely to have patients rate them high on the Psychodynamic subscale as well. Whereas clinician rated subscales of IPT and CBT were not as predictive of patient ratings on the therapy subscales.

Univariate analyses of variance (ANOVAs) for each dependent variable were conducted as follow-up tests to the MANOVA. The ANOVAs of the patient reported Psychodynamic Therapy (F(2,159) = 4.87, P < 0.01) and CBT (F(2,159) = 3.86, P = 0.023) were significant, whereas the Patient Reported IPT was nonsignificant (F(2,159) = 0.87, P = 0.41). Although main effect was not found for IPT, the level of IPT techniques endorsed by clinicians corresponds to the mean patient rated IPT scores, suggesting a similar trend. Table 3 presents the means and standard deviations of the patient reported therapy orientation for the three levels of each clinician reported therapy orientation (CBT, IPT, Psychodynamic Therapy) groups.

Table 3 Patient mean scores (30 Items) by clinician (Low–Medium–High) behavioral orientation
Full size table

Shortened Version

Based on the psychometric properties of the full scale (see Table 4), we considered a shortened version of this patient self-report measure. We retained items that had best psychometric properties for its hypothesized scale for the shortened version, which resulted in 15 items assessing CBT (6 items), IPT (6 items), and Psychodynamic Therapy (3 items). These items are in bold in Table 4.

Table 4 Evaluation of 30-item version
Full size table

The shortened instrument demonstrated good psychometric properties and an increased differentiation between factors. We conducted exploratory factor analyses using oblique (promax) rotation with varimax prerotation and squared multiple correlations as communality estimates priors to determine the initial composites (see Table 5). The number of factors was determined using eigenvalues and the interpretability of the rotated factor pattern matrix. There was only one eigenvalue over one (6.85, 0.93, 0.63, 0.19 so on) with the second eigenvalue approaching one (0.93); the average eigenvalue was 0.52. When items were assigned to a factor that had standardized regression coefficients of at least 0.30, the three factor solution was interpretable and all items loaded highly on the corresponding three therapy approaches.

Table 5 Rotated factor loadings from a principal axis factor analysis of 15 item version from the PPS (N = 540)
Full size table

Discussion

The psychometric and validity analyses of our Psychotherapy Practice Scale—Patient Depression Care Version offer evidence that patients are able to identify psychotherapeutic techniques and that developing a short patient measure of evidence-based care is promising. With further development and testing, this scale measuring the extent to which psychotherapy is evidence-based could be a vital tool in improving quality of psychotherapy in managed mental health care settings. This tool could be a cost effective method for identifying quality care and measuring improvements in providing evidence-based psychotherapy for depression.

This initial study is a solid first step towards developing a brief measure of the extent to which depression care is evidence-based. To develop this measure further, it should be used within ongoing studies of CBT, IPT and Psychodynamic Therapy where coded videotape validation is available to validate the measure. Alternative report formats might also be considered, including forced choose options, which might improve the reliability and validity of the measure. With clear validation, the measure could then be used to better measure care as usual in practice settings, as well as serve as a measure of evidence-based care in quality improvement interventions.

Although this first study of a patient measure of evidence-based care for depression provides useful information, several limitations should be noted. First, only 17% of our patient sample participated in this study. This number is quite typical of survey response rates in managed health care studies, but nonetheless raises questions about the representative nature of our sample. Second, providers in managed behavioral health care centers tend to be eclectic in their approach to psychotherapy. This was true of our clinicians. This prohibited us from being confident that our findings regarding the shortened scale would hold in settings where more rigorous adherence to a particular therapy occurs. Third, while our measure should evaluate if treatment is more consistent with evidence-based care, without including assessment of the intensity and sequencing of intervention approaches, the measure would not be an indication of the fidelity of care to a particular model. Finally, the scales in this study were highly related. The extent to which this represents an accurate representation of eclectic therapy as opposed to a measurement problem within the scale cannot be determined.

Despite limitations, this is the first study to our knowledge to propose a short patient-administered measured of evidence-based care for depression. A short patient-administered measure is particularly appealing because unlike providers, patients would not inherently have a bias towards reporting about evidence-based care as would clinicians who may feel they “should” be providing care according to guidelines. With further validation, this patient measure of evidence-based depression care could be a very promising tool for measuring and improving quality of psychotherapy.