Assessing the sensitivity of treatment effect estimates to differential follow-up rates: implications for translational research

Abstract

We develop a new tool for assessing the sensitivity of findings on treatment effectiveness to differential follow-up rates in the two treatment conditions being compared. The method censors the group with the higher response rate to create a synthetic respondent group that is then compared with the observed cases in the other condition to estimate a treatment effect. Censoring is done under various assumptions about the strength of the relationship between follow-up and outcomes to determine how informative differential dropout can alter inferences relative to estimates from models that assume the data are Missing at Random. The method provides an intuitive measure for understanding the strength of the association between outcomes and dropout that would be required to alter inferences about treatment effects. Our approach is motivated by translational research in which treatments found to be effective under experimental conditions are tested in standard treatment settings. In such applications, follow-up rates in the experimental setting are likely to be substantially higher than in the standard setting, especially when observational data are used in the evaluation. We test the method on a case study evaluation of the effectiveness of an evidence-supported adolescent substance abuse treatment program (Motivational Enhancement Therapy/Cognitive Behavioral Therapy-5) delivered by community-based treatment providers relative to its performance in a controlled research trial. In this case study, follow-up rates in the community-based settings were extremely low (54 %) compared to the experimental setting (95 %) giving raise to concerns about non-ignorable drop-out.

Appendix: Details on propensity score weighting

The propensity score weights were estimated using a Generalized Boosted Model (GBM) which is a non-parametric estimation technique that can account for a large number of pre-treatment covariates. GBM adaptively captures the functional form of the relationship between the covariates and treatment selection with less bias than traditional approaches (McCaffrey et al. 2004; Harder et al. 2010; Lee 2009). The “twang” package in R was used to conduct the weight estimation (Ridgeway et al. 2012).

Following the weighting procedure, we calculated the standardized mean difference of each of the pretreatment variables to measure the similarity of the CYT and propensity-score weighted comparison EAT samples. The standardized mean difference is defined using the following formula:

$$ \frac{{\hat{\mu }_{t} - \hat{\mu }_{c} }}{{\hat{\sigma }_{t} }}, $$

where $\hat{\mu }_{j}$ denotes the estimated mean value for the treatment and comparison conditions (j = t and c, respectively); $\hat{\sigma }_{t}$ denotes the estimated standard deviation in the treatment condition (the CYT/experimental youth) for a given pretreatment variable. Values of 0 for a standardized mean difference thus represent no difference in means while values of + or −1 represent one standard deviation difference between the two groups. Absolute standardized differences greater than 0.25 are considered to be ‘moderate effect size differences’ (Cochran 1968). To control for the possible confounding effects of pretreatment group differences that remain after weighting (Neugebauer and Van der Laan 2005), any variables for which the Absolute Standardized Mean Difference (ASMD) was greater than 0.25 were included in our regression models.

We began this analysis with 60 pretreatment variables related to substance abuse patient placement criteria established by the American Society for Addiction Medicine (American Society of Addiction Medicine 2001); these 60 variables have been used in previous investigations examining the effectiveness of adolescent drug treatment (McCaffrey et al. 2004). Many of these characteristics have been shown in prior work to influence substance abuse treatment outcomes, including pretreatment levels of substance use (Alford et al. 1991; Shoemaker and Sherry 1991; Jenson et al. 1993; Kennedy and Minami 1993), symptoms associated with emotional well-being and criminality (Alford et al. 1991; Brown et al. 1996, 2000; Myers et al. 1995), academic/scholastic attendance/performance (Rush 1979; Shoemaker and Sherry 1991), employment (Rush 1979), sociodemographics (Alford et al. 1991; Cady et al. 1996; Friedman and Glickman 1986; Friedman et al. 1986; Shoemaker and Sherry 1991), and social/familial substance use (Richter et al. 1991; Shoemaker and Sherry 1991).

Since controlling for variables unrelated to the outcomes may not improve estimates of treatment effects (Wooldridge 2001), we selected from this initial set of 60 pretreatment covariates only those pretreatment covariates that explained at least one percent of the variance in our outcomes for inclusion our propensity score model. This resulted in a total of 22 pretreatment variables which are listed in Table 1. Missing values on the 22 pretreatment variables were low (mean = 0.60 % and max = 4.87 %) and were controlled for in the propensity score model by balancing the two groups on missing value indicators for each variable in addition to balancing on the variables themselves.