Connecting high school dropouts to employment and education: an impact study of the National Guard Youth ChalleNGe Program

Abstract

This paper presents findings from the multi-year, random assignment study of the National Guard Youth ChalleNGe Program (NGYCP), an intensive residential program for high school dropouts. The “second chance” program gives youth an opportunity to earn a high school credential and prepare to enter the workforce or continue on to post-secondary education. The NGCYP study sample includes 1173 youth, ages 16–18, from ten program sites across the USA. Positive impacts on employment and educational outcomes were sustained 3 years after entering the study, with some evidence of older participants benefiting more than younger participants. Implications for research and practice are discussed.

JEL codes: J1, I26

1 Introduction

Young people who drop of out of high school face long odds of success in a labor market that increasingly values education and skills. In the USA, between 3.5 and 6 million young adults, ages 16 to 24, can be classified as high school dropouts, those without a high school credential and not currently enrolled in school (Northeastern University Center for Labor Market Studies 2009, U.S. Bureau of Labor Statistics 2014). While many of these young adults eventually earn a high school credential—a diploma or more often a General Educational Development (GED)¹ certificate—a long delay may place them at a serious disadvantage in competing for jobs and going on to college.² Early work experience is closely linked to future earnings and labor market success.³ In addition, there is growing evidence of the limited value in the labor market of the GED without further educational attainment (Heckman et al, 2010). More dramatically, for those that do not reconnect to the educational system and attain at least a high school credential, the individual and societal costs are high. High school dropouts face higher rates of unemployment, poverty, criminal involvement, and health difficulties. As a result, this population contributes significantly less in taxes from earnings while requiring more public spending in other areas such as incarceration costs and public assistance benefits (Northeastern University Center for Labor Market Studies 2009).

In response to the range of issues associated with dropping out of school, prevention and “second chance” programs have been developed to assist or reconnect this higher-risk youth population. These programs often offer a mix of education, training, employment, and supportive services to youth ranging in age from roughly 16 to 24 years of age. The programs themselves vary widely in terms of the specific populations they target (such as youth with disabilities or those involved in the justice system), the scale (such as large federally funded national programs vs. smaller community-grown models), the residential or non-residential setting, and the particular emphasis or combination of services offered. In recent years, there has also been an education shift in focus from GED obtainment as a goal towards high school diploma-granting opportunities and post-secondary education trajectories (Bloom, 2010). Ultimately, the goal is to help these youth successfully transition to adulthood with improved skills and/or credential and become productive, contributing members of society.

This paper presents results of the multi-year evaluation of one of these programs targeted at high school dropouts, the large-scale, residential, National Guard Youth ChalleNGe Program (NGYCP). The evaluation was led by MDRC, a nonprofit, nonpartisan research organization. This paper presents longer-term impacts on a range of educational and vocational outcomes for participants approximately 3 years after they joined the study. Shorter-term results published by MDRC showed promising results on high school credential receipt and employment, among other outcomes.

2 Background on success of youth interventions

Interventions targeting youth who have dropped out of high school have had mixed results, among those that have been rigorously evaluated. Indeed, a review of training programs operated under the Job Training Partnership Act in the 1980s found that these second chance programs produced few (and sometimes even negative) impacts on earnings (Bloom 2010; Orr et al. 1997). For example, JOBSTART, New Chance, and Job Corps have all struggled to produce sustained improvements in employment or earnings, although they have shown success in increasing GEDs and vocational certificates (Cave et al. 1993; Quint et al. 1997; Schochet et al. 2003, 2006). In response to the challenge of achieving enduring effects in this population, researchers note the importance of tracking youth over time to see how the impacts unfold in the short-term, medium-term, and longer-term (Rodríguez-Planas 2010). Relatedly, the question has also been raised about whether these relatively brief programs for high school dropouts should theoretically result in long-lasting effects or just a shorter burst of changes or outcomes for a young person.

Long-lasting effects are especially challenging for residential programs, in which participants often struggle to maintain positive changes as they reenter their peer groups and communities. For example, adolescents may arrive home to difficult living situations or friends that may be a negative influence. As a result, gains made during residential programs often diminish or disappear when participants return to their communities (Godley et al. 2001).

Although some programs may have been insufficiently intensive, or were evaluated prematurely, researchers and other experts on youth programming have suggested that many of the prior programs erred by not utilizing a positive youth development (PYD) model (Bloom 2010). While other programs might focus on “fixing” a young person’s problems, the hallmark of the PYD approach is a view of youth as positive resources to be developed (Roth and Brooks-Gunn 2003; Larson 2000; Lerner 2005). Similar to ChalleNGe, the federally funded YouthBuild program utilizes a PYD approach to individually support the youth it serves. YouthBuild gives youth leadership opportunities throughout the program, which provides vocational training and instruction towards a GED or high school diploma.

3 The ChalleNGe model

In the late 1980s and early 1990s, the Center for Strategic and International Studies (CSIS) led an initiative to develop new approaches to working with young high school dropouts. CSIS concluded that features of the military structure could be beneficial for at-risk youth and that the National Guard, with its strong community service mission, was well-suited to operate such a program (Cullinan et al. 1992). Congress funded a ten-site pilot of ChalleNGe in 1993, and funding was made permanent in 1998. The program currently operates in more than 30 locations across the USA and approximately 121,000 young people have graduated from the program since it began.

Utilizing a PYD model and run through the National Guard Bureau, ChalleNGe is an intensive program designed to “reclaim the lives of at-risk youth” who have dropped out of high school and give them the skills and values to succeed as adults.⁴ As one program director said, “It’s not about tearing these kids down. You have to build them up” (Bloom et al. 2009). The 17-month program is divided into three phases: the pre-ChalleNGe phase (2 weeks), the residential phase (20 weeks), and the post-residential phase (1 year). During the first two phases (totaling 22 weeks), the participants live at the program site, often on a military base. Throughout the program, ChalleNGe uses its specific setting, mix of activities, and the relationships it tries to create with youth—along with the education and training it offers—to promote positive change in the youth it serves. This change can potentially be seen through a youth’s attitudinal shift, improved education and employment outcomes, or other markers as steps towards independence and adulthood.

To start, the 2-week orientation period can be very challenging, both physically and mentally, much like military training. Candidates who complete pre-ChalleNGe are formally enrolled in the program as “cadets” and move to the residential phase. The curriculum for this phase is structured around eight core components that reflect current thinking about how to promote PYD: leadership/followership, responsible citizenship, service to community, life-coping skills, physical fitness, health and hygiene, job skills, and academic excellence. Cadets spend the largest share of each day in the education component. At the time of the study, most programs helped participants prepare for the GED exam, and a small number offered a high school diploma.

The program environment is described as “quasi-military”: the daily schedule is highly structured with almost no downtime, and staff closely monitor cadets at all times.⁵ A typical 100-bed program employs a multidisciplinary staff of approximately 50 people. The largest contingent of the staff is the cadre (or team leaders), who directly supervise the cadets day and night. The cadre come from a variety of backgrounds; however, a large majority have some military experience, ranging from one term of service to retirement from a full military career. In addition, all programs have a minimum of six full-time instructors. The instructors teach the GED subject courses, including math, science, writing and language arts, and computer skills, as well as other parts of the core components, including responsible citizenship and job skills. The ChalleNGe programs vary among states in the sources of their instructional staff, with some hiring instructors directly and others using teachers from local school districts or community colleges. Finally, a team of counselors work with cadets both in the context of one-on-one career and psychological counseling, as well as group classes, including life-coping skills, job skills, and health and hygiene. Most of the counseling staff hold either bachelor’s or master’s degrees in psychology, social work, mental health, alcohol and drug counseling, or school counseling. Note that while ChalleNGe utilizes a military framework (such as discipline, facilities, and staff) to accomplish its objectives, there are no military service requirements for participants.

Showing competency on the core components and having an acceptable placement lined up are requirements of successfully completing the residential phase or “graduating” from ChalleNGe. Throughout the residential phase, the cadets work with program staff to arrange a post-residential placement. Acceptable placements include employment, education, and military service. After graduation, participants move into the 1-year post-residential phase, which includes the placement and a structured mentoring program. The post-residential phase was developed to support youth’s transition back into the community and to directly address the common challenge of erosion of effects upon reentry into the community. The ChalleNGe mentoring program differs from traditional mentoring in that young people nominate their own mentors. Mentors tend to be drawn from family friends, extended family members and godparents, school and extracurricular staff, and religious leaders. ChalleNGe initiates the mentoring relationship during the residential phase, after the staff has screened and trained the mentors.⁶

Although the National Guard Bureau collects extensive data on program participation and participant outcomes that indicate the positive influence the program can play in participants’ lives, ChalleNGe had not been rigorously evaluated prior to this study (National Guard Bureau 2010).

4 Methods

4.1 Design and data

Youth were recruited for the study through participating ChalleNGe sites.⁷ The 12 participating sites were not chosen randomly; rather, there was an effort to identify programs that had stable staffing and that tended to receive more applicants than they could serve, a prerequisite for conducting a random assignment evaluation. In addition, the Department of Defense authorized a modest amount of funding to support enhanced recruitment efforts by the programs that participated in the evaluation.

Although 12 programs across the country agreed to participate in the evaluation, there were some incoming classes in which the number of applicants was too small to allow random assignment to take place. As a result, random assignment was conducted in only 10 of the 12 participating programs in late 2005 through early 2007. Random assignment was conducted for 18 incoming class cycles across the ten sites, for a total of 3074 participants. Due to different program sizes and numbers of participating class cycles, the number of participants varied greatly by site. At the low end, one site included 92 research participants, and, at the high end, one site included 941 research participants.

To be eligible for the ChalleNGe program, youth must have dropped out or been expelled from school, be unemployed, drug-free at the time of entry into the program, not currently on probation or parole for anything beyond juvenile status offenses, not serving time or awaiting sentencing, not under indictment or charged, and not convicted of a felony or capital offense. The program is open to both males and females, although about 80 % of the participants nationwide are male.

As part of their application to a participating ChalleNGe program, applicants signed consent forms to participate in the study and completed a baseline survey prior to randomization. ⁸ Of the 3074 participants, 2320 youth were assigned to the program group and 754 youth were assigned to the control group.⁹ Follow-up surveys were conducted, on average, at approximately 9 months, 21 months, and 3 years after participants entered the study.

This study focuses solely on outcomes from the 3-year survey in order to assess longer-term impacts.¹⁰ The survey was fielded to 1507 youth, yielding a total of 1173 sample members (722 program group and 451 control group) who completed the survey, for an overall response rate of 78 % (79 % of program group members and 76 % of control group members).

It is important to note that the survey fielding sample was not drawn to represent the full research sample. Given the varying samples from the individual sites, the impact results discussed below pool across all sites and provide impact estimates that weight each of the ten participating sites equally. Similarly, the random assignment strategy that maximized the number of youth served by the programs throughout this period created varying random assignment ratios. Therefore, the survey sampling aimed to adjust for these issues as much as possible. For example, the stratified sampling plan drew a higher percentage of sample members from smaller sites and from the control group. Within these parameters, the subsample was drawn randomly from the full research sample. However, as discussed below, there are limitations on the study’s generalizability beyond the survey sample. For the remainder of this paper, the survey sample will be the primary discussion of this study.

4.2 Participants

Demographic information about study participants indicate that ChalleNGe is serving a diverse group of high school dropouts (see Table 1). Approximately 88 % were male and the average age at the time of study enrollment was just under 17 years of age.¹¹ Just over 40 % of the sample described themselves as white, while over 30 % identified as black and another 18 % as Hispanic. Twenty-six percent of youth reported someone in their household received public assistance and fewer than half of the participants lived in two-parent households. More than two thirds of the sample (83 %) had completed tenth grade or lower. Eighty-three percent reported having been suspended from school, and 18 % had been convicted of a crime.

Table 1 Baseline characteristics of 3-year survey respondents

4.3 Measures

This study focuses on key program outcomes encompassing the major objectives of the intervention, including self-reported measures of youth educational, vocational, and employment outcomes.

• GED/high school diploma was measured using two single-item questions asking whether youth had received a high school diploma and whether they had received a GED.

• College credit was measured using a single-item question asking if youth had received any credit towards a college degree.

• Vocational training and certification was measured using two independent single-item questions that asked if youth had participated in vocational training and if youth had received a trade license or training certificate.

• Military enlistment was measured using a single-item question asking if youth had enlisted in the military since random assignment.

• Employment and earnings measures were derived from a series of questions including the start and end date of each job they had since random assignment, hours worked per week, and wages. These measures compile the self-reported information on employment and earnings in the 12 months prior to the survey interview and at the time of the interview.

• Involved in a productive activity combined responses to a series of questions and includes participation at the time of the survey interview in any of the following: employment, school, GED programs, vocational activities, the military, or a residential program. A separate measure utilized the earlier GED/high school diploma measure and ongoing participation in any of the following: employment, post-secondary education, vocational activities, the military, or a residential program.

• Living independently was based on a single-item question that asked where a youth lived at the time of the survey interview.

• Married or living with a partner was based on a single-item question that asked youth about their current marital status.

4.4 Statistical procedures

First, we conducted tests to verify that random assignment was successful in creating statistically equivalent groups at baseline. Then, program impacts tested for statistical significance in comparisons of the regression-adjusted means of the program and control groups for all outcome variables.¹² An intent-to-treat analysis is utilized, and thus, all members of each group were included regardless of their participation in the ChalleNGe program. As mentioned above, given the large variation in site sample sizes, the comparisons are drawn across all sites for a pooled estimate. The regression model controlled for sample member baseline characteristics and was weighted to account for differences by sites in each of the following aspects: site sample size (so each of the ten sites contributes equally to the results), survey response rates, and program versus control ratios. Given the aim of the impact analysis to understand the effects of the program across different sites, the decision was made to weight each site equally. This means, however, that the impacts presented do not represent the distribution of youth for the entire study. For the subgroup analyses, impacts for each subgroup were calculated, and an H-statistic was generated to assess whether the difference in impacts between subgroups was statistically different.

5 Results

5.1 Initial equivalence and attrition

Table 1 presents a comparison of baseline characteristics between participants assigned to the treatment group and those assigned to the control group. The table shows a handful of statistically significant differences between the two groups, but one group does not appear more disadvantaged or “at risk” than the other. For example, on average, those in the program group were more likely than the control group to have completed eleventh grade. However, this group also reported higher rates of conviction and drug or alcohol use. Overall, a logistic regression predicting treatment status among respondents confirmed that baseline characteristics did not significantly predict treatment status among those who responded to the 38-month survey (p = 0.25).

As discussed earlier, although the 3-year follow-up survey did not attempt to represent the full sample of baseline participants, analyses were conducted to assess the degree to which survey respondents (the sample included in the current study) were similar to the remainder of the larger sample at baseline. An analysis of baseline characteristics between these two groups revealed some small, but statistically significant, differences that limit the generalizability of the results (see Appendix Table 4). As was to be expected given the sampling plan’s aims, the survey respondents included significantly fewer members of the program group than the remaining sample in an effort to represent the program and control groups more equally. In addition, compared to those who were not included in the 38-month survey, survey respondents were more likely to be male, white, or Hispanic and less likely to be black, less likely to receive public assistance, more likely to have been convicted, and differed in their family composition and who suggested that they should apply to ChalleNGe. A logistic regression confirmed that there are systematic differences of baseline characteristics between the two groups that limit the generalizability of the results beyond the survey sample. Again, these differences were to be expected given the survey sampling plan, the variation in site sample sizes, and the demographic differences between the programs’ applicants.

5.2 Intervention received

Based on data from ChalleNGe’s Management Information System (MIS), among all youth assigned to the program group, 82 % started the pre-ChalleNGe phase. Sixty-eight percent of the program group completed the 2-week Pre-ChalleNGe residential phase and enrolled in the residential phase. Fifty-three percent of the program group graduated from ChalleNGe by finishing the residential phase and its requirements. Among those who completed pre-Challenge and formally enrolled in ChalleNGe, approximately 78 % graduated from the program. These percentages are similar to national averages for ChalleNGe program completion among enrollees. Unfortunately, participation rates among the survey sample were not available.

At the time of the 3-year follow-up survey, 56 % of participants in the treatment group reported that they were still in contact with their mentors. In addition, a total of eight youth assigned to the control group attended ChalleNGe. Survey data also indicated that at the 3-year mark, within the control group, 13 % had attended a residential program other than ChalleNGe. They had received an average of 9 months of high school classes and an average of 4.4 months of GED classes. Within the treatment group, they had received an average of 7.6 months of high school classes and an average of 4.8 months of GED classes (this may include both classes taken within the context of ChalleNGe as well as in other contexts). Notably, as previously mentioned, this study employs an intent-to-treat analysis, including all youth who were assigned to a given group during randomization regardless of actual participation in the program.

5.3 Impacts

Results of the impact analyses indicated that participation in the ChalleNGe program led to significant improvements on educational and employment outcomes (see Table 2) 3 years later. Youth in the program group were significantly more likely than the control group to have obtained a high school diploma or GED (p < .01, effect size = 0.33), with 72 % of those in the program group having earned a GED or high school diploma, compared with 55 % of the control group.¹³ Notably, this impact was driven largely by differences in GED receipt. The program group was also nearly twice as likely as the control group to have earned at least some college credit (p < .01, effect size = 0.34). Additionally, youth in the program group were more likely than the control group to have received vocational training since entering the study (p < .01, effect size = 0.14). However, there were no differences reported between the two groups with respect to receipt of a trade license or learning certificate.

Table 2 Program impacts at the 3-year follow-up

Exposure to ChalleNGe led to better employment outcomes for those in the program group, compared to the control group, 3 years later. Specifically, the program group was employed for a greater number of months with higher earnings (p < .01, effect size = 0.20 and 0.16) than the control group in the past year. The program group reported earnings of approximately $13,500 in the previous 12 months, about $2,300 higher than the control group average of $11,250. At the time of survey, the program group was also more likely than the control group to be employed and earning an hourly wage of $10 or more (p < .05, effect size = 0.12). It is interesting to note, given the National Guard involvement in the program, that there were no significant differences in military enlistment between the two groups.

Looking across outcomes, at the time of the survey, the two groups also differed on whether they had a high school credential and were involved in another productive activity, such as employment, educational or residential programs, vocational training, or military enlistment (p < .01, effect size = 0.23). The program group was more likely than the control group to be living on their own, but rates of marriage or living with a partner did not differ.

5.4 Impacts for subgroups

Given that effects of other youth interventions have varied by certain subgroups, such as age in the JobCorps evaluation, we investigated whether ChalleNGe is more or less effective for specific groups of participants. Specifically, analyses examined impacts based on age, high school academic performance, whether a family member suggested ChalleNGe participation (a possible sign of family involvement), and previous involvement with the justice system. Among these, differences mainly emerged in looking at effects by participant age. The results presented in Table 3 suggest that the program may have slightly more positive effects, for both educational and employment outcomes, on older participants (those who enter at age 17 or 18) than on younger participants (those who enter at age 16). Significant differences in impacts by age group were observed for receiving a high school diploma or GED and earning college credit (p < .05), with older youth faring better on both variables. ChalleNGe appears to have reduced high school diploma receipt (p < .05) among the subgroup of younger sample members; some of them probably obtained a GED through the program when they otherwise would have obtained a regular diploma. Among older sample members, those in the program group were also more likely than the control group to have been employed in more months and earned more money (p < .05).

Table 3 Selected impacts, by age at random assignment

Given the relatively small number of young women included in the evaluation, this study was unable to test directly for differences in impacts between men and women. However, a sensitivity analysis removing the women from the sample resulted in a similar pattern of impacts to that of the full sample. In addition, small sample sizes within some sites limited investigation of differences in impacts across the ten sites. Moreover, the evaluation did not include a systematic cataloging of site characteristics as part of the implementation research, compromising our capacity to make site-level attributions.

6 Discussion

The results indicate that the National Guard Youth ChalleNGe Program has significant positive effects on youth participants 3 years after entering the program. Specifically, compared to the control group, youth who participated in the ChalleNGe program earned more money and were more likely to obtain a high school diploma or GED, earn college credit, and be employed. These labor market outcomes differentiate ChalleNGe from some of the youth interventions mentioned earlier, such as JOBSTART and New Chance.

The results of this evaluation suggest that the skills, competencies, and credentials of vulnerable youth can be improved and sustained in the context of sufficient support and opportunities. By employing a PYD approach, the ChalleNGe program provides youth with an environment that allows them to gain such assets during these crucial transitional years. Once off-track, youth face serious challenges as they attempt to re-engage and become a participating, productive member of society. As such, those who, for a substantial period of time, are not in school, not working at a job, not in the military, or not married to someone connected to any of these institutions are far more likely during later adulthood to struggle with poverty, be on welfare, work inconsistently, and be unmarried (Danziger and Ratner 2010).

Moreover, in a twenty-first century labor market that increasingly prioritizes education and skills, young people who drop out of high school face greater challenges (Berlin et al. 2010). It appears that ChalleNGe can provide educational, occupational, and economic benefits that endure for 3 years. Particularly since post-secondary education is becoming increasingly necessary in the twenty-first century economy, the fact that program participants were significantly more likely than those in the control group to go on to receive college credit is promising. Given the high percentage of youth who received a GED certificate, continuing on to post-secondary education seems crucial. Studies have shown that GED holders earn significantly less than high school graduates (Tyler 2005).

Beyond the educational outcomes, it is encouraging that ChalleNGe was able to produce improvements in employment and income 3 years later. The employment rates—about half of the sample employed—mirror that of earlier follow-up survey responses. Even though respondents are getting older, this lack of an increase in employment over time is not surprising given the sharp rise in unemployment nationally that accompanied the most recent recession. In 2009, when the survey was administered, the unemployment rate for youth reached 15 %, the second-highest rate since the government began tracking these data.¹⁴

Overall, ChalleNGe appeared to help youth move along the path in the transition to adulthood. The young people had received a high school credential and were involved in another productive activity to further themselves or contribute to society, whether it be further school, training, or employment. In addition, the young people were showing their independence by moving out of their parents’ homes and living on their own.

It should also be noted that not all of those assigned to the program group actually made it past registration and the pre-ChalleNGe phases to enroll in (68 %) and graduate from (53 %) the ChalleNGe program. The impacts of the program on the group of ChalleNGe graduates are thus likely to be larger than the impacts reported in this study. It is a notable accomplishment that, even with the use of intent-to-treat analyses, the program was able to achieve sustained effects on educational and employment outcomes within an at-risk population. Due to the variation in dosage of intervention received within the treatment group, it will be important for future research to explore the influence of dosage on program impacts.

However, as noted earlier, MDRC has presented earlier impacts for this study sample. Comparing the 3-year results with those from a survey administered at 21 months, there is a narrowing gap between the two groups on earning a high school diploma or GED (from 24 to 16 percentage points). While impacts on other educational outcomes did not measurably shift, the impact on employment at the time of the survey increased slightly (from 5 to 7 percentage points). Overall, larger percentages of the research sample had earned college credit and participated in vocational training at the 3-year mark, indicating the sample’s continued progress—at an average age of 20 years old—to continue improving towards adulthood. Given these changing ages and the relatively young age of participants, longer-term follow-up of the research sample could further tell the story of the transition to adulthood and the possible longer-term effects of the ChalleNGe program.

Subgroup analyses also provided valuable insight into variation of impacts among participants. Specifically, stronger educational impacts were observed for those entering the program when they were older (17 or 18 years old) than for those entering when they were younger (16 years old) for obtaining a GED or high school diploma and college credit. One reason for this difference in impact is that youth who are 16 years old may be more likely to return to high school after dropping out than youth who are 17 or 18 years old. In fact, within the younger group, significantly fewer youth in the treatment group than in the control group received a high school diploma, indicating that participation in ChalleNGe may have inadvertently discouraged high school attendance among younger youth by making it possible for them to earn a GED. It has been well-established that a high school diploma is of more value in the labor market than a GED. This impact could be viewed as a potentially negative unintended consequence of the program and suggests the need for caution in making opportunities for earning a GED available to youth who are still of an age when they could remain in or return to high school.¹⁵

Differences in impacts between older and younger participants may also be influenced by the fact that older participants may be more mature and better able to take advantage of the opportunities offered by the ChalleNGe programs. The JobCorps evaluation similarly found evidence of greater effects for older participants. In addition, qualitative differences between students who leave school at 16 years old versus those who leave school at 17 or 18 years old may contribute to greater impacts among older participants. Generally, these results suggest the benefit of focusing second chance programs such as ChalleNGe on older youth who may have fewer options for obtaining high school credentials and who may approach the experience with greater seriousness and focus.

Although this study has a number of strengths, including the use of random assignment, data collected 3 years following entry into the ChalleNGe program, and a large, national sample, there are also a number of limitations. As noted earlier, differences between those who responded to surveys and those who did not respond may have biased the results and limit the generalizability of results beyond the survey sample. It is hard to know the direction of the bias, however, based on the differences in baseline characteristics between the survey respondent sample and the remaining random assignment sample. The survey sampling plan did draw sample members randomly, within the confines of the plan discussed earlier, so there are no selection bias issues. This sampling plan may limit generalizability in one sense, but the plan and the pooled, weighted impact estimates attempt to more equally represent the different programs in the study.

Similarly, the study could not include all youth who applied to these programs during the random assignment period. As discussed, some cohorts did not include enough applicants to allow random assignment to occur at a site. The weighting scheme utilized in the pooled analysis—which gave equal weight to each site—helps to account for these potential sampling variations. Specifically, programs that fared better with recruitment, and included more youth in the sample, do not overwhelm the smaller program samples in the analysis of outcomes. In addition, the study relied solely on self-reported measures of outcomes, which are influenced by individual bias and subjectivity.

Program effects also may have been inflated by the fact that participants in the study were drawn from ChalleNGe sites with stable staffing that received more applicants than they could serve, possibly indicating more successful programs. In addition, the ability to analyze and understand site-level differences in this evaluation was limited due to sample sizes and the implementation research involved. The analysis pooled results across sites to create the best overall estimate possible of ChalleNGe, using weights and regression adjustment. However, while technically the pooled results do not represent a national impact of ChalleNGe, the study sites do look like other ChalleNGe sites based on the available national performance data. Further implementation research to catalog the site differences and interpret site-level variation could be helpful for program administrators, funders, and policymakers.

Moreover, although it is significant that this study was able to collect data on participants more than 3 years after entry into the study, the field would benefit from studies that follow participants for a longer period of time. Such studies would help to determine whether improved educational and occupational outcomes are sustained throughout the complicated transition to adulthood and beyond. Finally, although the ChalleNGe program serves a vulnerable group of youth by virtue of their status as high school dropouts, due to the voluntary and intensive nature of the intervention, it tends to attract a youth who are particularly motivated to make change, limiting the external validity of the study. Specifically, the results cannot be generalized beyond the youth who would be interested and eligible in the ChalleNGe program.

7 Conclusions

Despite these limitations, this study significantly adds to the body of knowledge about youth interventions that can have enduring effects across key years in the transition to adulthood among high school dropouts. The young people in this study were off-track, with dwindling opportunities to re-engage and become participating, productive members of society. Rather than target their deficits and problematic behaviors, ChalleNGe utilized a PYD approach to “reclaim the lives of at-risk youth” and 3 years later the evidence remains in the employment and education outcomes discussed here. More broadly, the findings of this evaluation suggest that an emphasis on PYD may be particularly useful for youth who have dropped out of high school. The results of this study provide hope that ChalleNGe can offer some youth a second chance to become productive members of our society.

Appendix

Table 4 Comparison of 3-year survey respondents and remaining baseline sample