Evaluating the Impact of California’s Full Service Partnership Program Using a Multidimensional Measure of Outcomes

Abstract

This study evaluates the impact of California’s full-service partnership (FSP) program using a multidimensional measure of outcomes. The FSP program is a key part of California’s 2005 Mental Health Services Act. Secondary data were collected from the Consumer Perception Survey, the Client and Service Information System, and the Data Collection and Reporting System, all data systems which are maintained by the California Department of Mental Health. The analytic sample contained 39,681 observations of which 588 were FSP participants (seven repeated cross-sections from May 2005 to May 2008). We performed instrumental variables (IV) limited information maximum likelihood and IV Tobit analyses. The marginal monthly improvement in outcomes of services for FSP participants was approximately 3.5 % higher than those receiving usual care with the outcomes of the average individual in the program improving by 33.4 %. This shows that the FSP program is causally effective in improving outcomes among the seriously mentally ill.

Appendices

Appendix

Our econometric analysis incorporates a two step approach. We evaluate the validity of our instruments using two-stage limited information maximum likelihood (LIML) which provides finite-sample bias reduction (Angrist and Pischke 2009; Baum et al. 2007). We then scale the results down using an instrumental variables Tobit model.

Instrument Validity

To be valid, our instruments must meet the following four criteria (Angrist and Pischke 2009). First, the set of instruments must be strongly correlated with participation in the FSP program. We test for weak instruments using the approach described by Stock and Yogo (2005) for the LIML estimator. Second, the set of instruments must be conditionally independent of the outcomes. Third, the set of instruments must have no effect on the outcome other than through the first stage channel. In other words, the second and third criteria state that the set of instruments must be uncorrelated with the second stage error term, conditional on the included covariates. We provide support that our instruments satisfy the last two criteria by examining whether the instruments have any direct effect on outcomes by including them in the main equation and performing tests of statistical significance (Wooldridge 2006). An additional test is Hansen’s J test of overidentifying restrictions (Hayashi 2000). Overidentification tests assume that one instrument is valid in order to test the validity of the remaining instrument. We establish this validity below. The fourth criterion is that each instrument must be monotonic. This condition can be satisfied by using a linear functional form in the first stage equation. It addition, since both of our instruments are dummy variables, this criterion is automatically satisfied.

Removing Bias from the Measured Causal Effect

While the above approach is appropriate for examining instrument validity, it does not take into account potential censoring of the dependent variable. Because the dependent variable is constrained by the structure of the Consumer Perception Survey (CPS) to vary from 1 to 5 and because some consumers may have desired to choose answers which correspond to a number higher than 5, we consider the dependent variable to be censored from the right (similar to top coding). Such censoring is indicated by the high percentage of responses that pile up at the top value of the scale (12.7 %) which does not occur at the bottom value of the scale (0.29 %). In such cases, using ordinary least squares or LIML may result in predictions outside of the [1, 5] interval and can also result in biased parameter estimates. To account for censoring we estimate an instrumental variables (IV) Tobit model where the upper limit is censored (Cameron and Trivedi 2009). This model yields parameters virtually identical to the LIML method described above, but allows us to appropriately scale these parameters down to account for the impact of upper limit censoring and thus estimate marginal effects that avoid overstating any causal effect that we find. This makes a large difference in results as will be seen below. Both the IV LIML models and IV Tobit models are estimated with robust standard errors.

We estimate the following equations with corrected standard errors:

$$ {\text{Ln}}\left( {\text{Outcome}} \right) = \beta_{0} + \beta_{1} \hat{F} + \beta_{2} D + \beta_{3} DSM + \beta_{4} C + \beta_{5} Y + \varepsilon $$

(1)

$$ F = \alpha_{0} + \alpha_{1} IV + \alpha_{2} D + \alpha_{3} DSM + \alpha_{4} C + \alpha_{5} Y + \eta $$

(2)

where Outcome represents outcomes of services, Ln represents the natural logarithm, F is the number of months an individual has continuously participated in the FSP program (or, alternatively, F is a dummy variable indicating whether a client has had any participation in the FSP program), and $\hat{F}$ refers to the predictions from Eq. (2). The vector D contains variables for sex, race/ethnicity (White, African-American, Asian/Pacific Islander, Hispanic, other), and age (ages 18–25, 26–34, 35–44, 45–54, 55–64, 65–74, 75 and older). The vector DSM contains psychiatric diagnostic indicators (schizophrenia, bipolar disorder, depression, anxiety disorder, personality disorder, substance abuse, alcohol abuse, other disorder, unable to diagnose). The vector C contains county-level fixed effects. County-level fixed effects account for all non-time varying differences across counties and parametrically account for the clustering of clients by county. The vector Y contains year fixed effects. Finally, IV contains a vector of instrumental variables. The symbols ε and η refer to the error terms.

There are two instruments. The first is an indicator of whether a consumer received help in completing the CPS from a paid staff member. A paid staff member includes the consumer’s case manager or clinician, a staff member other than the consumer’s case manager or clinician, and professional interviewers. This instrument is a proxy for the literacy level of a consumer: the help indicator.

Nationally, a self-reported mental health problem is associated with lower literacy, even after controlling for sociodemographics including educational level. This is consistent with research indicating that individuals with SMI have relatively low literacy, even among some whose education level is high (Grace and Christensen 1998; Christensen and Grace 1999). In a study by Lincoln et al. (2006), low literacy was positively correlated to depressive symptoms, but was not related to mental health-related quality of life in well-controlled regression models. In addition, limited literacy has been positively correlated with having a psychotic disorder (Lincoln et al. 2008). As a proxy for low literacy, we expect the help indicator to be positively associated with the probability of entering the FSP program, and, conditional on the inclusion of psychiatric diagnoses in the second-stage equation, exogenous to mental health outcomes.

In order to statistically test the exogeneity of the help indicator using an overidentification test, we must include a second instrument that is clearly exogenous. This instrument is the season of a consumer’s birth. While unfamiliar to most non-psychologists/psychiatrists, the relationship between season of birth and psychiatric diagnosis is well established through over 100 studies that have examined the relationship between mental illness and season of birth. For a review that also explores the possible reasons for this relationship, see Castrogiovanni et al. (1998). Individuals born in the winter and spring are more likely to develop schizophrenia, individuals born in the first quarter of the year are more likely to have bipolar disorders and major depressive disorder, and individuals with seasonal affective disorder are more likely to be born in May (Castrogiovanni et al. 1998). This suggests that those born during the months from October to May will be consumers with specific psychiatric diagnoses. Thus, our instrument indicates being born during these months.

Because consumers born during these months are overrepresented among consumers, such consumers will necessarily be more likely to enter the FSP program simply because there are proportionately more of them, other things equal. We refer to this instrument as the season-of-birth indicator. Conditional on the inclusion of psychiatric disorders in the second-stage equation, this instrument will be exogenous to mental health outcomes.

Results

Testing of the instruments was performed using the LIML equations as shown in Tables 6 and 7. In Table 6, the Cragg-Donald Wald F statistic for the joint statistical significance of the instruments was 10.34 which is greater than the critical value for 10 % maximal LIML size, 8.68, showing that the instruments are sufficiently strong. The corresponding statistic in Table 7 is 21.18, which is also larger than the critical value of 8.68. Note that the Stock and Yogo (2005) critical values when using two-stage LIML are legitimately smaller than the Stock and Yogo (2005) critical values when using two-stage least squares.

Table 6 Logarithm of outcomes of services: FSP versus usual care (months of participation)

Table 7 Logarithm of outcomes of services: FSP versus usual care (any participation)

The inclusion of the two instruments in the main equation (Eq. 1) along with the original values of F and estimating using ordinary least squares with robust standard errors shows that the instruments have parameters close to zero and are not statistically significant at the 5 % level either singly or jointly (equation using monthly participation measure: help index, parameter = 0.006, p = 0.12; season of birth, parameter = 0.004, p = 0.12; partial F test of set of instruments, p = 0.09; equation using any participation indicator: help index, parameter = 0.006, p = 0.13; season of birth, parameter = 0.004, p = 0.12; partial F test of set of instruments, p = 0.09) supporting the assumption that neither instrument directly influences outcomes and that each is conditionally independent of outcomes. In addition, in reduced-form equations, both instruments are positively signed.

Finally, overidentification tests of the instruments failed to reject the hypothesis that the overidentifying instrument, the help index, is exogenous. In Table 6, the Hansen’s J statistic is 0.003 (p = 0.95) and in Table 7 the corresponding statistic is 0.91 (p = 0.34). In addition, in Table 6, the exogeneity of time in the FSP program was rejected (χ ²: 4.91, p = 0.03) indicating the instruments were necessary to obtain reliable estimates of the causal effect of the FSP program. The corresponding statistic in Table 7 rejects the exogeneity of any participation in the FSP program (χ ²: 3.81, p = 0.05).

Note the large differences between the parameter on “Any participation in FSP” in Tables 5 and 7. The model in A2 predicts values outside of the [1, 5] interval, while the model in Table 5, which accounts for censoring from the right, only predicts values within the [1, 5] interval, yielding much smaller and more credible marginal effects.