Community Mental Health Journal

, Volume 42, Issue 3, pp 303–313

What Predicts Supported Employment Program Outcomes?

Authors

    • Dartmouth Medical School
    • NH-Dartmouth Psychiatric Research Center
  • Haiyi Xie
    • Dartmouth Medical School
  • Gregory J. McHugo
    • Dartmouth Medical School
  • John Halliday
    • Institute for Community InclusionUniversity of Massachusetts
  • Rick A. Martinez
    • Johnson & Johnson
Community Psychiatric Practice

DOI: 10.1007/s10597-006-9037-8

Cite this article as:
Becker, D.R., Xie, H., McHugo, G.J. et al. Community Ment Health J (2006) 42: 303. doi:10.1007/s10597-006-9037-8

Abstract

Numerous state systems and local mental health and vocational rehabilitation programs are currently attempting to implement supported employment. This cross-sectional survey of 26 mental health agencies, partnering with federal-state vocational rehabilitation, identified differences in access to supported employment services and rates of competitive employment (efficiency) as well as predictors of access and efficiency. Access varied from 2 to 100% and was related to the percentage of supported employment specialists per consumers with serious mental illness served by the mental health agency (funding). Efficiency varied from 7 to 75% and was related to implementation of the critical components of evidence-based supported employment and to the local unemployment rate. To help mental health clients achieve their employment goals, state systems and local programs should address consolidation of resources in supported employment and the quality of implementation of supported employment.

KEY WORDS

severe mental illnesssupported employmentvocational rehabilitationemployment outcomes.

Introduction

According to the President’s New Freedom Commission on Mental Health (2003), Achieving the Promise, working in one’s community is central to recovery and should be a major goal of the mental health system. Developing new identities, such as worker, and moving past the mental health patient role, is changing the quality of people’s lives (Alverson, Becker, & Drake, 1995; Bailey, 1998; Steele & Berman, 2001; Provencher, Gregg, Mead, & Mueser, 2002).

Research studies have identified supported employment as an evidence-based approach to helping people with serious mental illness gain competitive employment. A dozen randomized controlled trials comparing supported employment to other vocational interventions such as prevocational training, sheltered work, and transitional employment show that supported employment produces better competitive employment outcomes (average 59% in supported employment vs. 21% in traditional programs) (Bond, 2004a,b; Twamley, Bartels, Becker, & Jeste, 2004; Latimer et al., 2005). Two meta-analyses have yielded similar findings (Crowther, Marshall, Bond, & Huxley, 2001; Twamley, Jeste, & Lehman, 2003).

Currently, numerous states and local programs are attempting to implement supported employment (Ganju, 2004). However, the quality and outcomes of these programs are likely to be mixed (i.e. regional variation). Several correlational studies have examined the relationship between quality of employment services and program-level outcomes. In one study, Gowdy and colleagues (2003) found better employment outcomes in programs with several critical features, such as de-emphasis of prevocational programming, rapid assessments, and individualized follow-along supports. In another study, higher employment outcomes were correlated strongly with overall supported employment program fidelity (Becker, Smith, Tanzman, Drake, & Tremblay, 2001). In a third study, one supported employment program with higher fidelity had better employment outcomes than a second supported employment program with moderate fidelity (Drake, McHugo, Becker, Anthony, & Clark, 1996). A fourth study that used a 33-item supported employment fidelity scale found small and non-significant correlations with nine employment outcome indicators, but two of the four subscales had significant correlations with employment outcomes (Bond, Picone, Mauer, Fishbein, & Stout, 2000). Recently, another multi-site study found a positive correlation between fidelity to supported employment and employment outcome (McGrew & Griss, 2005).

To understand variation in competitive employment for people with serious mental illness, Drake and colleagues (1998) reported on a statewide study in which a variety of vocational services were offered. Rates of competitive employment were higher in mental health centers that focused directly on competitive employment rather than on prevocational programs and that spent a larger proportion of their budget on vocational services. The relationship between the local public mental health and vocational rehabilitation providers appeared to be an important factor at some of the centers.

Several studies have examined the relationship between labor market indicators for the general population and for people with serious mental illness. Overall, the results are mixed. In one study, Catalano, Drake, Becker and Clark (1999) found that the primary labor market did not affect the unemployment rate of people with serious mental illness who were seeking employment. They conjectured that people with mental illness may seek work in the secondary labor market, which includes part-time, temporary, and high-turnover jobs and, therefore, are not affected by the primary labor market conditions. Two other studies found that employment outcomes were unrelated to local unemployment rates (Purlee, 1993; Drake et al., 1998). Other studies, however, have found that local economic conditions do influence employment outcomes (Wong et al., 2004; Cook, 2005).

Although nearly all state mental health systems have endorsed supported employment services as part of their state mental health plan (Ganju, 2004), few studies address factors that affect access (percentage of eligible people receiving supported employment services) and efficiency (percentage of people receiving supported employment services who are working). The purpose of this study was to determine the extent of differences in access and efficiency and the factors that predict access and efficiency.

Methods

Programs

The Johnson & Johnson–Dartmouth Community Mental Health Program is a private–public–academic initiative to encourage collaboration between state departments of mental health and vocational rehabilitation in implementing evidence-based supported employment services for people with serious mental illness in a broad array of sites in a sustainable manner (Johnson & Johnson Community Mental Health Program, 2002). The Corporate Contributions Program of Johnson & Johnson provides financial support that is matched by state departments of mental health and vocational rehabilitation. The process is facilitated through partnership with the New Hampshire - Dartmouth Psychiatric Research Center. A total of 26 sites from seven states (Connecticut, Kansas, Maryland, Oregon, South Carolina, Vermont, Washington, DC) were included in this study. There were three sites in each state with the exception of one state that had five sites (Connecticut) and another that had six sites (Maryland). Most of the mental health agencies were comprehensive community mental health centers. In two states, core service agencies contracted with psychiatric rehabilitation programs for supported employment services.

Design and Measures

This study was based on a cross-sectional survey of 26 sites that were participating in the Johnson & Johnson–Dartmouth Community Mental Health Program during the period January to June 2004. Quantitative measures were used to predict access and efficiency. Because there were no existing standardized measures for some of the indices, the constructs developed are transparent and meant to capture meaningful differences.

As part of the Johnson & Johnson–Dartmouth Community Mental Health Program, local supported employment supervisors regularly submit access and employment outcomes, and this was independent of assessing other program features. We averaged two quarters (1–3/04 and 4–6/04) of data for access and efficiency to improve reliability and validity. A combination of telephone interviews and paper-and-pencil assessments was used to collect data from key informants concerning both access and efficiency at each site.

The supported employment supervisors at the local mental health agencies submitted data on: the number of adults served in the mental health agency, case mix (i.e. proportions of people with schizophrenia or substance abuse co-morbidity), number of full-time equivalent staff positions in vocational services, supported employment program staff turnover, supported employment program tenure, contact with local vocational rehabilitation agency including referral rate, timing of eligibility determination, and number of clients served by the mental health agency and vocational rehabilitation agency during the two study quarters. The vocational rehabilitation agency supervisors answered the same questions regarding contact between the mental health agency and the vocational rehabilitation agency.

Other sources of data were the US Census Bureau and local area unemployment statistics. The following variables were extracted from these data sources.

Dependent Variables

Access. The average percentage of eligible people in the mental health agency’s community support program for adults with severe mental illness (i.e. ages 18–60) who received supported employment services during two quarters (1–3/04 and 4–6/04).

Efficiency. The average rate of competitive employment during two quarters (1–3/04 and 4–6/04). The competitive employment rate was determined as the number of people from the mental health agency who received supported employment services and worked in a competitive job during the quarter divided by the number of people who received supported employment services. Competitive employment is defined by community jobs that pay at least minimum wage (paid directly by the employer to the employee) that any person can apply for, including full-time and part-time jobs.

Independent Variables

Focus on Competitive Employment. The number of full-time equivalent (FTE) supported employment specialists vs. the number of FTE positions in vocational services that include sheltered work, day treatment, transitional employment, and supported employment services in the mental health agency.

Funding. Total number of FTE supported employment specialists per total number of people served in the community support program for adults in the mental health agency.

Relationship Between Mental Health and Vocational Rehabilitation. The degree of collaboration between the local mental health agency and vocational rehabilitation agency for providing supported employment services. Local supported employment supervisors from the mental health agencies and vocational rehabilitation agencies answered questions independently about the amount of face-to-face contact between mental health and vocational rehabilitation staff, frequency of referrals to vocational rehabilitation from the supported employment program, and amount of time for determining eligibility for vocational rehabilitation agency services. The answers were coded on five-point scales. A fourth question queried about the number of people from the supported employment program who received vocational rehabilitation agency services during the period 1–6/04. For this question, the answers of the supported employment supervisor and the vocational rehabilitation supervisor were averaged and divided by the average number of adults with severe mental illness served by the mental health agency in the quarters 1–3/04 and 4–6/04. The percentage was recoded into a five-point scale by ranking the percentages into quintiles. Average agreement between the two raters across the four questions was 0.56, which indicates good agreement. The sum of the four coded scores was classified into three categories of good, mixed, and poor. The categories and corresponding scores were based on practice guidelines.

Supported Employment Fidelity Scale Score. The Supported Employment Fidelity Scale (Bond, Becker, Drake, & Vogler, 1997) assesses adherence to the evidence-based principles of supported employment. The Supported Employment Fidelity Scale contains 15 items, each rated on a five-point response format, ranging from 1=no implementation to 5=full implementation. The scale is divided into three sections: staffing, organization, and services. The total score of the Supported Employment Fidelity Scale was obtained during site visits by supported employment trainers who were experienced raters. Most of the programs were rated during the period 1–6/04. Eight sites were rated within 3 months following the desired rating period.

Supported Employment Program Tenure. The number of years that the mental health agency has had at least one FTE supported employment specialist.

Supported Employment Staff Stability. The number of supported employment staff and supervisory positions vacated during the period July 2003 to June 2004.

Background Variables

Case Mix. The percentage of people diagnosed with schizophrenia in the mental health agency’s community support program for adults with severe mental illness. Percentage of people with substance abuse co-morbidity was also computed.

Metropolitan Area. Population count (from the 2000 Census) of the mental health agency service area that was greater than 50,000 as an indicator of population density. The binary variable was defined ‘yes’ if the population count was greater than 50,000.

Regional Job Market. Local area unemployment rate for June 2004 (Retrieved January, 2005, from http://www.data.bls.gov/PDQ/outside.jsp?survey=la).

Transportation. Availability of public transportation in the mental health agency locale was rated good, mixed or poor. This rating was based on questions about whether any public transportation was available and whether it was available in evenings and on weekends. If there was no transportation in the evenings and on weekends, it was coded poor. If transportation was available, including weekends and evenings, it was coded good. If availability included either evenings or weekends, it was coded mixed.

Data Analyses

We used multiple regression to predict the percentage of people receiving supported employment services (access) and the percentage of people receiving supported employment services who work in competitive jobs (efficiency). To predict access, we developed two models. The first model included three independent variables, which were funding, focus on competitive employment, and relationship between mental health and vocational rehabilitation. In the second model, we added several background variables: case mix, metropolitan area, regional job market and transportation. Because none of the background variables was predictive, we interpret the first model.

To predict the percentage of people receiving supported employment services who work in competitive jobs (efficiency), our independent variables were Supported Employment Fidelity Scale score, supported employment program tenure, supported employment staff stability, and relationship between mental health and vocational rehabilitation. Again, we fitted the model with independent variables only, and then the model with both independent and background variables. Of the background variables, only regional job market was a significant predictor. Therefore, our final model included all independent variables plus regional job market as predictors.

Results

Substantial variation occurred across the 26 sites for both of the dependent variables. For access, the percentage of clients receiving supported employment ranged from 2 to 100% (M=28.7%, SD=25.2%). For efficiency, rates of competitive employment ranged from 7 to 75% (M=44.9%, SD=19.3%). The results of the analyses are summarized in Tables 1 and 2 including both the bivariate correlations and the multiple regression in which the factors were entered simultaneously. Table 1 shows that funding is a highly significant predictor of access. Overall, the model is highly significant. Collectively, all three independent variables explained 82% of the variance in access, and 79% of the variance after adjusting for the number of predictors in relation to sample size (i.e. adjusted R2). The correlations between the non-significant predictors and access are: metropolitan area (r=0.035), regional job market (r=−0.219), transportation (r=−0.347), case mix – schizophrenia (r=0.082), case mix – substance abuse co-morbidity (r=−0.034).
Table 1

Predicting the Percentage of People Getting Supported Employment Services (ACCESS) at 26 Study Sites

Predictors

Bivariate Correlation

Parameter Estimate

SE

p-value

Intercept

 

0.065

0.10

0.52

Funding

0.90**

13.475

1.46

<0.0001

MH/VR Relationship

−0.25

−0.022

0.04

0.56

Focus on competitive employment

0.15

0.041

0.09

0.66

*p<0.05; **p<0.01; F=31.16; p<0.0001; R2=0.82; Adj R2=0.79.

Table 2

Predicting the Percentage of People Obtaining Competitive Employment (EFFICIENCY) at 26 Study Sites

Predictors

Bivariate Correlation

Parameter Estimate

SE

p-value

Intercept

 

−0.302

0.45

0.51

MH/VR Relationship

0.08

0.034

0.05

0.47

SE fidelity scale score

0.51**

0.014

0.01

0.05

SE program tenure

0.43*

0.003

0.01

0.63

SE staff stability

−0.08

−0.013

0.02

0.43

Regional job market (unemployment rate)

−0.57**

−3.214

1.41

0.03

*p<0.05; **p<0.01; F=3.59; p=0.0188; R2=0.49; Adj R2=0.35.

Table 2 shows that both the Supported Employment Fidelity Scale score and regional job market were significant predictors of efficiency. The overall predictive power for this model was significant, and 49% of the variation (35% after adjustment) was explained by all of the predictors. Supported employment program tenure is significantly associated with efficiency as a bivariate relationship, but due to its significant correlation with the Supported Employment Fidelity Scale score (r=0.44, p=0.025), it becomes a non-significant predictor after adding other variables into the model. The correlations between the non-significant predictors and efficiency are: metropolitan area (r=−0.365), transportation (r=0.051), case mix – schizophrenia (r=−0.393), case mix – substance abuse co-morbidity (r=−0.129).

There was minimal variation in the relationship between mental health and vocational rehabilitation and no significant prediction. Eighty percent of the sites (n=20) were rated as having a good relationship.

Discussion

The analyses revealed three key findings. First, mental health agencies with a higher percentage of supported employment staff per number of adults served in the community support program (funding) provided greater access to supported employment services. Simply put, a larger number of people who provide supported employment services translates into greater service capacity.

Second, the key components of supported employment as described in the Supported Employment Fidelity Scale are related to work outcomes. Agencies that fully implement the critical components of supported employment have better work outcomes. Supported employment fidelity displaces the significance of program tenure statistically. In other words, while program experience may be important, attaining high fidelity is the critical step to achieving good outcomes. Furthermore, this study gives support to the importance of implementing the critical components of the practice rather than adapting the model to local conditions.

The third key finding was that people with serious mental illness who are accessing supported employment services and living in areas with high unemployment rates are less likely to be working. Another large multi-site study also recently found that employment rates in economically disadvantaged areas were lower (Cook, 2005). Nevertheless, the programs that followed more closely the critical components of supported employment had higher work outcomes than the programs that did not follow supported employment. In our study, for competitive employment outcomes, the predictive power of regional job market (unemployment rate) and program fidelity was similar. High fidelity is unlikely to completely overcome the impact of unemployment, yet program fidelity has an impact even when regional job market is controlled for. While program leaders cannot affect the local unemployment rates, they can influence the degree of program fidelity. Hence, programs should resist falling into a trap of low expectations for employment outcomes in areas of high unemployment and focus instead on high fidelity implementation of the supported employment model.

A central objective of the Johnson & Johnson Community Mental Health Program is to foster collaboration between the mental health and federal-state vocational rehabilitation systems. Historically, the two systems have had difficulties collaborating to help people with serious mental illness who are trying to gain employment (Noble, 1998). Close collaboration was expected as part of participating in the Johnson & Johnson program and was promoted through financial incentives and program monitoring. Collaboration developed through joint training and conferences, committees, shared funding of state supported employment coordinator positions, and accessing funding mechanisms together. Sites were expected to report simple employment outcomes on a regular basis, which promoted looking at outcomes together. Involvement of evidence-based practice training centers (often associated with an academic institution) provided training, monitored outcomes, and facilitated supported employment programs. In one state, vocational rehabilitation co-located vocational rehabilitation staff in the mental health agencies. There was consistently good collaboration between the mental health and vocational rehabilitation agencies across the 26 sites.

Several limitations of this study warrant attention. First, employment outcomes were provided by local supported employment supervisors rather than by an independent assessment, which may affect reliability and validity of the data. Second, we do not have data regarding access and efficiency prior to the Johnson & Johnson-Dartmouth Community Mental Health Program. Third, the lack of variance in the relationship between mental health and vocational rehabilitation may reflect a measurement problem. The scales, however, were built on practice guidelines and designed to distinguish between real world settings.

Financing supported employment continues to be a major problem (New Freedom Commission on Mental Health, 2003). In future studies, it would be helpful to describe the supported employment funding sources and the creative ways they have promoted increased access. Policy research needs to determine how resources from mental health, vocational rehabilitation, Medicaid, and Social Security can be aligned to support evidence-based vocational services.

Finally, a factor that was not addressed in this study was the availability of benefits counseling. Surveys continue to reveal that people with mental illness want to work (McQuilken et al., 2003), but they fear losing Social Security benefits, which prevents some people from seeking employment. Recent evidence shows that providing comprehensive benefits counseling increases the percentage of people who obtain competitive employment (Doug Bailey, personal communication, March 3, 2005) and their wages earned (Tremblay, Xie, Smith, & Drake, 2004).

Acknowledgment

This study was supported by a gift from the Johnson & Johnson Corporate Contributions Program.

Copyright information

© Springer Science+Business Media, Inc. 2006