Design
This study was part of the Professional Alliance with Clients in Treatment (PACT) study—a multi-site prospective naturalistic clinical cohort study among adolescents in outpatient youth mental health care (YMHC) and youth addiction care (YAC). For the present study, we used data collected at the first treatment session and at four months post-baseline. We assessed therapeutic alliance as early as possible—i.e. at the end of the first treatment-session—to minimize possible confound due to the effect of early symptom improvement on the perceived therapeutic alliance. We assessed treatment-outcome four months post-baseline because most symptom improvement occurs in these first months of treatment [20,21,22]. This study was funded by The Netherlands Organization for Health Research and Development (no. 729101014) and approved by the Medical Ethical Board of the University Medical Center Leiden (P.15.001).
Participants
From April 2015 to September 2016, 161 youths were invited to participate in the study from the usual inflow of patients at three YMHC and two YAC facilities in the Netherlands. Eligible patients were 13–23 years old, who started outpatient mental health or addiction treatment, were willing to participate in the study and provided written informed consent (if under the age of 18 years also consent from at least one caregiver). We barred patients from the study if they were cognitively incapable of comprehending the questionnaires (clinical judgement), were diagnosed with DSM-IV autism spectrum disorder, or needed inpatient treatment (clinical judgement). Informed consent was provided by 153 youths and 137 youths (89.5%) completed the baseline assessment (see Fig. 1: Consort Flow diagram). After the baseline assessment, six youths were barred from the study because they needed inpatient treatment (n = 5) or were diagnosed with autism spectrum disorder (n = 1). Four additional youths were excluded because of withdrawn informed consent. The final sample consisted of 127 youths (YMHC: N = 71; YAC: N = 56). From these, 15 youths (11.8%) did not participate in the 4-month follow-up assessment (Fig. 1). Fifty-six therapists participated in the study and they treated 1–8 youths each: 23 therapists (41%) treated one youth; 15 (26.8%) treated two youths; and 18 (32.2%) treated three or more youths.
Treatment
Participants were offered individual outpatient cognitive behavioral interventions (n = 93), family-based treatment (n = 7) or other treatment (i.e. psychomotor therapy and other psychotherapy, n = 24), and type of treatment was not specified for three participants. Median treatment duration was 6 months (interquartile range [IQR]: 4.5–8.0 months) and a median number of 7 sessions was attended (IQR: 4.0–11.0 sessions).
Assessments
All study assessments, at baseline and 4 months’ follow-up, were conducted by trained research assistants. Study assessments included questions about participants’ and therapists’ demographic background, therapeutic alliance, mental health problems, substance use frequency and diagnosis. Therapeutic alliance was assessed from the perspective of youths and therapists with the Working Alliance Inventory-12 at baseline (WAI-12; [23, 24]; Dutch translation: WAV-12; [25]). The WAI-12 is a 12- item instrument with three subscales: the affective quality of the client-therapist relationship (“Bond”), the degree of agreement on the treatment goals (“Goals”), and the level of task collaboration (“Task”), based on Bordins [26] conceptualization of therapeutic alliance. Youths and therapists were required to rate each item on a 5-point Likert scale ranging from ‘never’ to ‘always’. We used the WAI Total score (range 1–5), as an indication of the overall alliance quality with higher scores indicating better quality of the therapeutic alliance.
To assess youth self-reported mental health problems at baseline and 4 months’ follow-up, the Strengths and Difficulties Questionnaire (SDQ; [27]: Dutch translation: [28]) was administered. The SDQ is a commonly applied screening and treatment-****outcome measure with 25 items with a 3-point Likert scale, ranging from ´not true´ to ´certainly true´. We used the SDQ Total score (range 0–40), with higher scores indicating more problems. The substance use section of the Measurements in the Addictions for Triage and Evaluation, Youth version (MATE-Y; [29]) was used to collect past month information on the youths' primary substance or behavioral addiction (gaming/gambling) at baseline and 4 months follow-up. Clinical psychiatric diagnosis at baseline was made by the treating therapist, who used the criteria of the Diagnostic and Statistical Manual of Mental Disorders [30].
Primary outcome measure
We used a prespecified dichotomous outcome measure reflecting a favorable versus unfavorable treatment outcome status at 4-month follow-up as the primary outcome measure for both youths in YMHC and YAC. Youths in YMHC were considered to have a favorable outcome status if their 4-month SDQ total score was lower than 12.5. In the absence of formal Dutch cut-off scores [31], we followed the procedures suggested by Jacobson and Truax [32] and De Beurs et al. [33] to determine the cut-off value of 12.5 as the average of the mean SDQ total score in a Dutch general youth population sample (M = 9.7, SD = 4.7; [28]) and the mean baseline SDQ total score in our clinical population (M = 15.3, SD = 5.4). Youths in YAC were considered to have a favorable outcome status if they had used their primary substance or displayed their primary gaming/gambling on less than five days in the 30 days preceding the 4-month follow-up, as recommended in the guidelines for routine outcome monitoring (ROM) in Dutch addiction care (Blanken, et al. 2011, Note from Dutch Expertgroup ROM-Addiction care).
Data-analysis
Since our alliance data were nested within two levels of clustering we explored the option of using multi-level modeling, but due to insufficient sample size at both levels (level-1: 56 therapists; M = 2.27 youth per therapist; range 1–8 and level-2: five treatment facilities, range 4–11 therapists per treatment facility) it was not possible to estimate effects accurately [34]. In order to address the first study goal, pertaining to the prognostic importance of youth- and therapist-rated alliance for treatment outcome, we conducted two separate multivariate logistic regression analyses including either the youth- or the therapist-rated alliance as independent variable and treatment outcome status (favorable versus unfavorable) as dependent variable. In both regression analyses, we examined the effects of potential confounders: gender, age group (≤ 16 or ≥ 17 years), treatment setting (YMHC or YAC), cultural background (Dutch or Non-Dutch), education level (low or high), baseline problem status (favorable or unfavorable) on the primary problem domain (mental health status for youth in YMHC; substance use status for youth in YAC), and baseline problem status (favorable or unfavorable) on the concurrent problem domain (substance use status for youth in YMHC; mental health status for youth in YAC). A variable was considered to be a relevant confounder when the youth- or therapist-rated therapeutic alliance regression coefficient changed with 10% or more after adding the potential confounder into the logistic regression model [35].
In order to address our second study goal, pertaining to the added predictive value for treatment outcome of incorporating the alliance ratings from youths as well as therapists, we conducted a logistic regression analysis in which both youth- and therapist-rated alliance were entered into the model, again with a favorable or unfavorable treatment outcome as dependent variable. A test of improved prediction accuracy was conducted by comparing the difference in -2 log likelihood between the two models, in which we considered a change of ≥ 3.84 (df = 1; p = 0.05) as an indication of an improved prediction model.
Finally, we used the median value of the youth-rated and the therapist-rated alliance to distinguish four mutually exclusive subgroups, with alliance rated as (1) “strong” by both youth and therapist, (2) “weak” by the youth and “strong” by the therapist, (3) “strong” by the youth and “weak” by the therapist, and (4) “weak” by both youth and therapist. We then conducted a logistic regression analysis with the subgroup categories as independent variable to predict treatment outcome status, using the first (strong-strong) and the fourth (weak-weak) category as reference category in two separate analyses. To estimate the treatment outcome status of youth with a missing 4-month follow-up assessment, we did not use statistical imputation, but instead asked the treating therapist to provide a 'best estimate' of the youth's outcome status. All statistical analyses were conducted with IBM SPSS Statistics for Windows, version 25.0 (IBM Corp., Armonk, N.Y., USA).