Evaluation of Substance Abuse Prevention and Treatment Programs

Abstract

In this chapter, program evaluation is examined as a way to systematically utilize the scientific method to determine whether adolescent substance use prevention interventions implemented with families, within schools and the workplace, and at the community level achieve their intended goals. General purposes and approaches to program evaluation are described, as well as the processes associated with the planning and conduct of evaluations, with a focus on addressing common challenges for evaluators and stakeholders. Finally, a summary case involving an evaluation of a large, national school-based substance use prevention intervention is used to illustrate concepts and processes important to the evaluation of substance abuse prevention and treatment programs.

Appendices

Appendix: Application Example—The Adolescent Substance Abuse Prevention Study (ASAPS)

The evaluation of the Adolescent Substance Abuse Prevention Study (ASAPS) was conducted with the goal of assessing the implementation and effectiveness of a two-component universal, school-based substance abuse prevention curriculum delivered by police officers who had previously been trained, and were currently teaching the Drug Abuse Resistance Education (D.A.R.E.) program. The research team for the evaluation were located at The University of Akron Institute for Health and Social Policy in Ohio, and stakeholders who collaborated with the team included D.A.R.E. America organization leadership, trainers, officers, and students, substance abuse prevention researchers, research methodologists, and statisticians from across the United States, curriculum specialists and teachers, and the Robert Wood Johnson Foundation (RWJF).

Evaluation planning begun in 1999 when RWJF funded a project to revise and evaluate the D.A.R.E. curricula currently being implemented in schools. At that time, the principal investigator contacted and invited stakeholders, researchers, and educators to participate in two planning groups: (1) a curriculum workgroup, and (2) a research design workgroup. Both groups worked concurrently on planning curriculum revisions, as well as the study’s evaluation design. This process took nearly 2 years to complete and included pilot studies, capacity building for the administration of the study, and school and police department recruitment in six large cities across the United States. The new curricula, Take Charge of Your Life (TCYL), was implemented in 120 middle schools in six sites across the continental U.S. and data collection for the outcome evaluation study began in the fall of 2001. The 9th grade component was delivered to the same cohort of students in the 2003/2004 academic year. Students were followed annually for data collection until they were in the 11th grade (2005/2006).

Defining Program Goals and Processes of the ASAPS

The overarching goal of the new curriculum was to delay substance use initiation or reduce current levels of use in an ethnically and socially diverse U.S. middle and high school populations. Curriculum materials were developed by prevention experts using criteria for effective prevention programming derived from existing meta-analyses and reviews of the literature. The curriculum workgroup used these final recommendations to develop the new middle and high school curricula, using a problem driven format based on authentic dilemmas and issues faced by teens as they are pressured or tempted to experiment with or use tobacco, alcohol, illegal drugs, or inhalants. The primary instructional strategy used student-to-student engagement through instructor guided in-depth discussions, role-playing of skills and concepts, and small group problem-solving.

The chosen framework enables students to actively utilize the intended ideas and skills as they develop their own understandings and capacities to be in control of situations where they are pressured to use tobacco, alcohol, and drugs. In order to attain these objectives, the curriculum focused on the following specific targets/constructs for change (immediate and intermediate outcomes) at the student level:

• Consequences of substance use: understand the nature of and risks (personal, physical, social, legal) associated with the use of alcohol, drugs, tobacco, and inhalants.

• Beliefs and attitudes toward substance use: examine and understand their own beliefs and attitudes related to alcohol, drug, tobacco, and inhalant use.

• Normative beliefs regarding peer use of substances: correct misperceptions about the rates of substance use by same-age adolescents.

• Decision-making skills: make positive quality of life decisions.

• Communication skills: communicate clearly and interact positively in social and interpersonal situations.

• Resistance/refusal skills: develop and use resistance skills.⁹

The final curricula was composed of ten, 40 min lessons administered in the 7th grade, and seven, 40 min sessions delivered to the same students (cohort) when they were in the 9th grade. These targeted constructs were understood to operate within a larger, ecological set of influences ranging from family to community levels. The result of research and development of the two-part curricula was the theoretical model which guided both the intervention as a whole, and the research design for evaluating the program. This theoretical model is depicted in Fig. 20.2.

Fig. 20.2

Program theory for adolescent substance abuse prevention program

In this model, substance use is seen as a direct outcome of the students’ knowledge and beliefs about the nature and effects of various dangerous substances, their attitudes toward the use and consequences of abuse of the various substances, and the level of personal skills which may allow them to effectively resist pressure to participate in the use of any illicit substances. Thus, the intervention targeted for change the student’s normative beliefs, skills (immediate outcomes), personal attitudes and values (intermediate outcomes) which in turn were expected to reduce the student’s intentions to use substances (intermediate outcome) and substance use (ultimate outcome). As can be seen in Fig. 20.2, these constructs are conceptualized to mediate the effects of the intervention on substance use. For this reason, the mediating constructs/variables are treated as proximal/immediate or intermediate outcomes of the intervention, while substance use is viewed as the distal/ultimate outcome.¹⁰ In order to create a more comprehensive framework for a substance prevention program, the model also included constructs thought to directly influence the student’s normative beliefs, communication skills, and personal attitudes and values and indirectly influence the student’s substance use. These school, community and student risk indicators were included and are represented in Fig. 20.2 as social bonding, school, and community risk factors (National Institute on Drug Abuse 1999).

Formative Evaluation

The formative evaluation of the new, 7th grade curriculum began with separate focus groups of 7th grade students, their parents, middle school teachers, and D.A.R.E. police officers to elicit feedback on the curriculum goals, content, and materials. Students were asked to provide input on ways to revise the problematic situations presented, as well as for the inclusion of any other problems they felt were relevant to same-age adolescents. In addition, students were queried about their opinions regarding having police officers teaching in the classroom. Parents’ attitudes about drug abuse, prevention programming, and having police officers in the classroom were also explored. The same process was followed with local police officers who had experience with the D.A.R.E. program. Feedback on the ability/motivation of officers to implement this type of lesson and their ability to involve students in active discussion of the problems presented were solicited. The officers were also asked to evaluate how realistic each problem was and the accompanying activities and to provide input/insight into problematic situations they have been exposed to in the D.A.R.E. classroom. These problems were incorporated into the lesson plans. Finally, middle school teachers were recruited to review the lesson plans and instructor training manual.

The first trial delivery of the middle school curriculum (pilot study I) was tested in one urban middle school. All 7th grade health or social studies classes received the middle school curriculum (n = 153). All data collection instruments were tested in the feasibility study including measures of outcome and mediating constructs, fidelity of implementation, and exposure. Two local D.A.R.E. officers participated in a two day training on the curriculum theory, content, intended delivery procedures, and background information; each was responsible for half of the 7th grade classes. In order to gather as much information as possible, each class was observed by trained raters from the research institute. Officers were observed by one or more members of the curriculum workgroup during delivery of each lesson. Feedback was exchanged on the lesson and revisions were made to the lessons based on these sessions and the feedback from the focus groups.

To test the specific objectives of the study, a number of study instruments were developed, including instructor observation sheets, short, lesson specific, evaluation surveys for instructors and students, a student survey with measures of the targeted outcomes and mediators, risk and protective factors, and demographic information. These instruments relied heavily on existing measurement instruments, especially those measuring students’ substance use as well as normative beliefs, decision, communication and refusal skills, and personal attitudes and values related to substance use. Based on an extensive review of the literature, it was evident that in addition to using many existing measures, new or adapted measures of many of the study constructs would be necessary. These newly developed measures were pre-tested in the trial delivery of the middle school curriculum by the institute’s research staff.

A second pilot study was conducted in nine middle schools using a non-randomized, no-comparison group design; a convenience sample of nine schools, including urban, suburban, and rural public schools and one private (Catholic) school. Data on mediators and outcomes were collected from 462 students. The results of the second pilot study were promising with significant changes in the expected direction for the immediate outcomes of normative beliefs, refusal skills, and the perceptions of harm for ATOD use. Instructors implemented the program with high fidelity, and feedback was positive toward the feasibility of implementing the program on a larger scale. The positive results of the two pilot studies indicated the curriculum and officer training were ready for the larger efficacy study.

Outcome Evaluation

The research workgroup implemented a randomized, longitudinal control trial, to evaluate the intervention. School districts (middle and high school clusters) were randomly assigned to either a treatment (intervention schools that would receive the new TCYL curricula) or a control condition. However, ethical concerns prevented the study group from requiring the control condition schools from refraining from implementing any prevention programming; consequently, the control condition became the condition of ‘prevention programming as usual.’

Sampling

High schools and all their feeder middle schools were selected as the clusters of interest, with the school district becoming the unit of sampling. A sample size of approximately 40 school districts per condition was expected to provide statistical power of 0.80, assuming an alpha level of 0.05 and a two-tailed test of hypothesis. To insure diversity in ethnicity and geographic location, six large metropolitan areas were selected for recruitment, with the ‘core’ central city school district identified as the center point for a 50 mile sampling radius (with the goal of including urban, suburban, and rural school districts) for each region of the United States. These regions were centered in the following metropolitan areas: (1) New York City, NY/Newark, NJ,¹¹ (2) Detroit, MI, (3) St. Louis, MO, (4) New Orleans, LA, (5) Houston, TX, and (6) Los Angeles, CA. A sampling frame was developed for the area surrounding each ‘core’ district, with the inclusion criteria of public school districts with middle schools that housed 7th and 8th graders and four-year high schools. To ensure diversity across schools, the six sampling frames were stratified by risk status, which was calculated using a function of the proportion of students eligible for free lunch (an indicator of SES/poverty) and minority enrollment in the school.

Organizational Structure of the Study

Given that this was a large evaluation study that spanned six metropolitan regions of the United States, a well-planned administrative structure was a critical component for a successful implementation of the study plan. Two principal investigators oversaw all the study activities, a senior researcher oversaw the sampling and assignment of districts to intervention condition, another senior researcher oversaw curriculum implementation and monitoring, and a senior researcher oversaw data analysis. A project manager coordinated the recruitment of school districts and police departments and training of police officer/instructors, and a data manager oversaw data collection and processing. In addition, regional coordinators were hired to monitor on-site recruitment, communication, data collection, and retention of schools and subjects. They also supervised full-time site coordinators for each of the six regions who oversaw part- and/or full-time data collection personnel and acted as liaisons between the schools, police, and community, while supervising curricula delivery and data collection procedures.

Recruitment and Retention of Schools

The local recruiting process was then directed by the site coordinator in each region. An explanatory brochure and a cover letter were sent to the selected school district superintendents requesting their participation in the study. They were informed that their agreement to participate in the study would not guarantee that their schools would receive the program and that if they are included in the control group, they would not be able to deliver this program for two years to allow the 7th grade cohort to transition to their high school. The mailings were followed with a telephone call from the Project Manager at the Institute; this correspondence provided information to the coordinator as to the gatekeepers and appropriate person(s) with which to continue recruitment of the district schools. The actual recruitment activities for each school district varied; however, the process generally included site visits by senior research staff members to explain the study to district personnel, principals, teachers, and parents/PTA associations, along with visits to the local police/sheriff’s department to recruit local officers to train for instructors for the TCYL curricula.

School districts were required to agree to be assigned to either the control or intervention (treatment) condition to participate in the study. Once district personnel agree to participate, similar materials, a letter of support from the district superintendent, and phone calls were made to the principals of the middle schools and high schools to elicit their cooperation. In recognition of the costs associated with participation with the study, each school was offered $500 per year of participation in the study and resource materials relative to substance abuse prevention as incentives to participate. They were also promised copies of the final study report.

After all agreement letters were signed, the district was randomly assigned to either the intervention/treatment or control condition. The core districts were randomly assigned first (to achieve a balance of control and intervention inner-city districts) and the schools surrounding the ‘core’ district were assigned randomly without any attempt to achieve a balance of control and intervention districts (although the sample was fairly balanced within each region).

Retention of schools was one of the primary roles of the site coordinators. In addition to the incentives, each regional site coordinator convened a ‘Community Advisory Group,’ consisting of local community leaders representing the schools, business community, local political groups, social service agencies, community coalitions, and/or law enforcement. The Advisory Group’s role included supporting the project within the community, assisting in the interpretation of study findings, and serving as a support network to establish the program in the middle and high schools after the completion of the research project in both the experimental and control communities.

Recruitment, Retention, and Tracking of Students

Parents and students in both the intervention and control schools were required to sign a consent form to participate in the survey administration (all intervention students received the curriculum, regardless of survey consent status). Parents and students were also asked to provide information for persons who would know the whereabouts of the student for follow-up purposes. Parents and students were reminded about the need for the signed consent forms and incentives; to increase signed consent forms, incentives were offered to students in terms of classroom-wide activities, such as pizza parties. The consent forms were available in both English and Spanish.

One week before the first intervention class (7th and 9th grades), a baseline survey was administered to students who had provided appropriate consent. To ensure anonymity, student was assigned a unique identifier. This key file was then stored separately from any student data for use in matching coded student surveys to the data file at each data collection point. Once the students completed the survey, they placed the coded survey forms in the blank envelope, sealed the envelope, and deposited it in a slit on a sealed box that was located at the front of the classroom; this procedure was used at each data collection point.

Site coordinator and team members were responsible for student tracking. Students who were not present at a data collection session but were still enrolled in school were approached individually by the site coordinator and asked to complete the forms in an empty classroom or other private place to ensure confidentiality. Students who had left the district were lost to follow-up. The decision not to follow-up students who left the school district was made after an attrition pilot study was conducted by the research group and the findings indicated that the cost for locating these students was prohibitive (Stephens et al. 2007).

Data Collection and Measures

The data collected can be summarized by five general construct categories: (1) substance use outcomes, (2) mediating outcomes, (3) risk factors, (4) moderators of the intervention, and (5) indicators of implementation fidelity. The outcomes and individual risk factors were measured as part of the main student survey; other variables required data collection from additional and sometimes multiple sources, such as the officer/instructors, curriculum trainers, site coordinators and ethnographer’s interviews with students, and key community and school informants. As there were many sources of data collected for this evaluation, this example will focus only on data collected from student surveys (outcome evaluation) and implementation fidelity observations (process evaluation).

Baseline student surveys were administered immediately prior to program delivery in 7th grade, and a post-test was administered approximately 90 days after the completion of the ten session curriculum. A third post-test was administered in the 8th grade, and pre- and post-tests were administered before and after the implementation of the 9th grade curriculum (seven sessions). Post-tests in the 10th and 11th grade were administered approximately one and two years after the 9th grade post-test. Neither the officer/instructor nor the classroom teacher was present during data collection to assure students that their responses on the survey would be confidential. To maintain generalizability with national data systems and other prevention program evaluations, survey measures were adapted from ongoing studies, such as Monitoring the Future Study (Johnston et al. 2013), Center for Substance Abuse’s Core Measures (1999), and prior studies of D.A.R.E wherever possible. Any changes to the original measures were tested for validity and reliability in the feasibility study.

Since the curriculum is specifically aimed at reducing or preventing the use of alcohol, tobacco, marijuana, and inhalants, this construct was measured using the self-report of students’ use of tobacco, alcohol, marijuana, and inhalants on a paper-pencil, confidential student survey at seven points in time. The age of first use as well as pattern of use (lifetime, last 12 months, and last 30 days) and amount of use where appropriate (binge drinking) were measured using questions taken from the Monitoring the Future survey. To measure age at first use of substances, students were asked, for example, “How old were you the first time you had a full drink of an alcoholic beverage?” with ordinal response options ranging from “never” to “15 or 16.” Substance use was operationalized with questions such as, “How many TIMES (if any) have you had alcoholic beverages to drink (more than just a few sips)….” “during the last 12 month” or “during the last 30 days.” Seven response options ranged from “never” (coded as zero), to “40 or more times” (coded as 6).

With regard to mediating variables, intentions to use substances was measured with drug-specific questions, asking how likely the student was to try alcohol, tobacco, or marijuana in the next 12 months. Responses ranged from 1 (definitely will) to 5 (definitely will not). Attitudes toward the use of tobacco, alcohol and marijuana were measured by two items for each drug. Students selected a response to complete the stem, “I think it is okay for students my age to…” Response items included, “smoke cigarettes once in a while,” “drink alcohol almost every weekend,” and “smoke marijuana once in a while.” Responses ranged from 1 (agree) to 5 (disagree). Normative beliefs were measured by three questions that asked students how many 10th graders the student believed used tobacco, alcohol, or marijuana in the last 30 days. Response categories ranged from 1 (more than 75%) to 5 (10% or less). Perceptions of harmful consequences resulting from substance use were measured by three items that asked students how much they thought the use of a particular substance (alcohol, tobacco, or marijuana) affects how the brain works. The response categories ranged from 1(none) to 5 (a lot).¹²

Refusal skills were assessed by responses given to three hypothetical scenarios involving the opportunity to use tobacco, alcohol, or marijuana. Students were asked to read the scenario in which a substance was offered by a peer. Each student selected the best refusal response to that offers from a list of possible responses given the person being offered does not want to use the substance. Responses were weighted according to the level of assertiveness demonstrated. For example, a response of “no, maybe later” was assigned a lower score than a response of “no thanks, I don’t want to smoke.” Scores ranged from 0 (least appropriate response) to 2 (best response chosen). The student survey also included risk and protective factors and demographic characteristics including self-reported age, sex, race/ethnicity, and family composition.

Implementation Fidelity

Fidelity of implementation was measured by the amount of intervention material actually covered in each session, the number of intervention sessions completed, the amount of role-playing, demonstration, and discussion that occurred during a sample of two sessions, the number of times the instructor reinvented or altered the material or delivery, and overall quality of delivery. These constructs fit with the recommendations of Baranowski and Stables (2000) that implementation and reach are critical components of process evaluation. These data came from four sources: (1) independent classroom observation checklists completed by trained site staff, (2) post-instruction self-report surveys by officer/instructors, (3) post-instruction self-report surveys by students, and (4) attendance records for each student in each of the intervention sessions.

Data Processing and Analysis

Data collection was overseen by the site coordinators and their team members. Student surveys were scored/scanned electronically and sent to a master data base maintained by the Institute to assure the highest possible quality. Data were assessed for completeness, that responses were within range of the values for each variable, and that data were internally consistent. Particular attention was paid to assuring that the longitudinal data on the cohort sample could be linked across all waves of data. As soon as data were received, linkages were established and Site Coordinators were contacted to remedy any discrepancies. Data from observation forms and implementation fidelity surveys were also scanned and cleaned at the Institute and stored electronically.

Data Analysis

The unit of sampling was the school district, but the curricula were delivered in classrooms within those schools, and the surveys were administered to students. Hence, students were nested within the classroom, school, and district. This complex sampling design presented the potential for biased estimates for the standard errors of any hypothesis tests. Therefore, all analyses were conducting using statistical procedures which adjusted the standard errors.¹³ Logit models were utilized for binomial outcomes, in addition to weighted least squares, while maximum likelihood models were used in path modeling and structural equation models with outcomes at the ordinal and interval levels.

Evaluation Study Findings

13.1 Process Evaluation (Implementation Fidelity)

The process evaluation reported here focuses on the implementation fidelity of the officer/instructors delivering the new curriculum in the 7th and 9th grades and the exposure of students to the curriculum (as measured by lessons attended). Classroom observations conducted in the 7th grade and 9th grade all showed high implementation fidelity. Coverage of the content of the lessons in 7th grade ranged from 34 to 100% with a median of 72 and 81% content coverage for each of the two lessons observed. For the 9th grade, the proportion of content covered in each of the two lessons observed ranged from 12 to 100% with a median proportion of 70 and 78% for each of the observed lessons. Instructors’ use of the appropriate activity in their instruction was somewhat lower, with a median proportion of 63 and 44% delivering the two observed lessons with the appropriate instructional activity in the 7th grade and 50 and 60% using the appropriate instructional strategy during the observed lessons in the 9th grade (Sloboda et al. 2009a, b). While these numbers may seem low, they were relatively high compared to other studies reported in the literature (Ennett et al. 2011; Hallfors and Godette 2002; Ennett et al. 2003).

Attendance records indicated there was adequate exposure to the curricula with 69% of 7th grade intervention students attending all ten lessons and another 27% of students attending eight or more of the ten lessons. In the 9th grade, 44% of the intervention students attended 100% of the seven lessons and 17% of the intervention students attended at least five (71%) of the seven lessons.

Outcome Evaluation

The outcome evaluation was completed when the cohort of students were 11th graders. A detailed reporting of the data analytical procedures and finding is reported in Sloboda et al. (2009a, b). Forty-two high schools and 59 middle schools were assigned to the intervention (TCYL curriculum intervention) and 41 high schools with their 63 feeder middle schools were assigned to the control condition. A total of 19,529 students completed consent forms prior to the administration of the 7th pre-test (intervention group n = 11,314; control group n = 8215). Baseline surveys were completed by 10,028 intervention students and 7302 control students. During the course of the study, three high schools were lost to follow-up—two were destroyed in Hurricane Katrina and one opted out of the study. Baseline data were used to check the randomization process between the intervention and control schools; demographic characteristics and substance use outcomes showed no significant differences between the groups, thus confirming the randomization procedure was successful. Attrition analyses at the 11th grade post-test showed that overall, older students, female students, non-White students, and students who reported the use of alcohol, tobacco, or marijuana were more likely to drop out of the study. The only source of differential attrition was that students who identified as “other-race” were more likely to drop out of the control group than the intervention group (Sloboda et al. 2009a, b; Teasdale et al. 2009).

The primary goal of the intervention was to reduce or delay the onset of substance use among the cohort of students who received the TCYL curricula. The program theory proposed that the curricula would change the targets of normative beliefs, consequences of substance use, attitudes towards substance use, refusal, decision-making, and communications skills (immediate outcomes/mediators). Changes in these constructs would result in students’ intentions to avoid substance use (intermediate outcome) and changes in intentions would result in lower use of tobacco, alcohol, and marijuana (ultimate outcome). The findings did not support the hypothesis that the curriculum would have an impact on these outcomes. In fact, although there had been intervention effects on normative beliefs and refusal skills when students were in middle school, these effects were no longer significant in the 11th grade. There was no intervention effect on past 12-month outcomes of alcohol use, getting drunk on alcohol, or marijuana use. Surprisingly, alcohol use and getting drunk on alcohol in the past 30 days showed a significant effect in the opposite direction as expected, as did the 30-day use of tobacco and marijuana and the self-reported past two weeks binge drinking outcome. That is, the intervention group reported, on average, using tobacco, alcohol, and marijuana at higher rates than the control group in the 30 days prior to the 11th grade survey administration. The risk ratios for these effects ranged from 1.09 for alcohol use to 1.21 for cigarette smoking. In order to further understand these puzzling findings, subgroup analyses was utilized to test the moderating effects of prior substance use (as reported at baseline), sex, and race/ethnicity were conducted.

The analyses by substance use status, sex, and race/ethnicity did provide the evaluators with some insight for whom the program was and was not successful in reducing substance use. Males in the intervention group appeared to be driving the higher rates of alcohol use, getting drunk and bingeing on alcohol, while females in the intervention group were more likely (than their control group counterparts) to binge drink and smoke cigarettes. Non-White students in the intervention group had higher rates of cigarette use than non-White students in the control group, and White students in the intervention group had significantly higher alcohol outcomes than their White counterparts in the control group. The most surprising findings were those found when the nonusers at baseline were compared to substance users at baseline. Students who reported no use of alcohol at baseline in the intervention group were more likely than nonusers of alcohol in the control group to report binge drinking in the past two weeks, and alcohol use and getting drunk on alcohol in the past 30 days. Similarly, non-smokers in the intervention group were more likely than nonsmokers in the control group to report smoking cigarettes at the 11th grade post-test (Sloboda et al. 2009a, b).

A single finding regarding marijuana use was found in the expected direction for the intervention group. Students in the intervention group who reported using marijuana at baseline had significantly lower rates of use in the 11th grade than did non-marijuana users in the control group. In summary, the intervention appeared to work only in early marijuana users to reduce marijuana use. The intervention also appeared to have the effect of increasing smoking in females and non-Whites students, increase drinking in Whites and males, increasing problem drinking in females, and increasing smoking, alcohol use, and problem drinking in students who were nonusers in the 7th grade (Sloboda et al. 2009a, b).

What Went Wrong?

Two analytical questions were explored in an effort to provide insight into the study results. First, the basic program model of immediate, intermediate, and ultimate program targets was assessed to determine if the curriculum was targeting constructs that, if changed, would change the ultimate outcome of substance use. Path modeling was used to examine the relationships among normative beliefs, attitudes toward use, perceptions of consequences of substance use and refusal, communication, and decision-making skills on the intentions to use and actual use of cigarettes, alcohol, and marijuana. The findings proved interesting in that the effects (direct and indirect) of these mediators on each of the outcomes differed slightly; for example, refusal skills worked to reduce intentions to use cigarettes and marijuana by helping students to utilize decision-making skills to form intentions, but impacted alcohol use directly and possibly only for students who already had formed negative attitudes toward alcohol use. However, with the exception of communication skills (which appeared to increase intentions to use alcohol, and had no effect on cigarette or marijuana use), all program targets had a significant indirect or direct effect on each of the substances, with normative beliefs having the largest total (indirect + direct) effect on each substance. The effect of each of the program targets on substance use was small, and therefore the program would have to produce large changes to actually have an effect on the substance use outcomes (Stephens et al. 2009).

In fact, as noted above, while the program showed effects early on (in middle school) on normative beliefs, perceptions of consequences and refusal skills, by the 11th grade, these effects had disappeared and none of the targeted mediators showed differences between the control and intervention group. The second set of analyses used path modeling to examine the relationship of the program model constructs for baseline users and nonusers; these analyses were done to explore the single positive effect of the program on baseline marijuana users. The results of these models showed several significant effects of the treatment variable on the targeted mediators which were not seen in the intervention group overall. For nonusers, the intervention had a significant effect only on two marijuana specific mediators. Nonusers in the intervention group were significantly higher on their 9th grade marijuana refusal skills and marijuana specific normative beliefs than nonusers in the control group. For the baseline users, intervention effects were shown on mediators for each of the three substances. Baseline cigarette users in the intervention group were significantly higher than baseline users in the control group on the perceptions of harm for cigarette use. Baseline alcohol users in the intervention group were significantly higher than baseline users in the control group on normative beliefs about alcohol use and the perceptions of harm for using alcohol. Finally, baseline marijuana users in the intervention group were higher on their intentions not to use marijuana, marijuana refusal skills, and normative beliefs about marijuana use.

The TCYL intervention appeared to have no effect on the cigarette specific mediators in the nonuser group, but the program did appear to have a significant direct effect on cigarette use in the direction of higher use for TCYL baseline nonusers. There was, however, a significant program effect for the baseline user group on cigarette specific perceptions of consequences. The findings were similar for alcohol use, with the addition of a significant program effect on normative beliefs surrounding the use of alcohol for students who were baseline users of alcohol. The results for the model of marijuana use are similar for both baseline users and nonusers. The main effect of the program on marijuana use and the intentions not to use marijuana for baseline users became nonsignificant when the mediators are included in the same model, indicating full mediation of the program through marijuana specific refusal skills and normative beliefs, both of which showed positive program effects for baseline nonusers and users. Neither baseline users nor baseline nonusers showed any program effects on the global measures of communication or decision-making skills (Teasdale et al. 2009).

These findings partially explained why the intervention had beneficial programmatic impacts on marijuana use for baseline users. That is, the TCYL intervention reduced beliefs about the normative nature of marijuana use and increased refusal skills, compared to control students. In contrast, no explanation was found for why the program had the negative impacts for nonusers. It is interesting to note that students in the TCYL program did not report increased intentions to use alcohol, tobacco, and marijuana relative to control students. Based on the program theory, any impacts of the program should have worked through the targeted mediators (normative beliefs, consequences, and skills) and behavioral intentions. This was not the case, leaving the question of what did the intervention do to create the negative outcomes for baseline nonusers? If it was not the proposed theoretical mediators that influenced these outcomes, what components of the intervention impacted substance use? The evaluation group and other researcher continue to explore these findings to determine what went wrong and whether or not the intervention can be improved to change the targeted constructs.

Lessons Learned from the ASAPS

There are many lessons to be learned from the ASAPS, but this chapter focuses on those regarding the evaluation process. First, the ASAPS as an evaluation undertaking was a success in that the evaluation team collaborated with a multitude of stakeholders to come to agreement on program goals and processes, as well as how to evaluate those processes and outcomes. The shared responsibility and input of these stakeholders provided the framework for a comprehensive evaluation, the results of which continue to be utilized by substance abuse prevention specialists, educators, and public health professionals.

Second, this study illustrates the importance of incorporating program theory into the evaluation process and measuring and analyzing constructs and processes that compose that theory. While the evaluators expected the program goal of reduced substance use to be met, they were, with only one exception, not achieved. To understand this contradictory finding, they had the measures of the program theory and implementation procedures to examine to determine why the intervention did not work as anticipated. They found that while the program did impact some of the proposed mediators of substance use, the changes were not consistent or large enough to have an effect on substance use across a subpopulation of students. The program was implemented with fidelity, so this did not appear to be the weakness. Perhaps the content of the curricula was not powerful enough to change the targeted constructs? The lesson learned is that the program should be revised to strengthen its impact on these constructs before being implemented in middle and high schools.

Finally, the importance of utilizing program evaluation findings is also illustrated in this study. As a result of these analyses, the evaluators came to two important conclusions: (1) the TCYL curricula was not a universal curricula, and would be more appropriate for high risk students, and (2) it may suggest to focus on targeted subpopulations of students (users OR nonusers) rather than using “universal” programs that are delivered to a diverse population of students. These conclusions were taken seriously by the decision-maker/stakeholders (D.A.R.E. America), who decided not to implement the TCYL program until further revisions and testing had been done to insure the intervention achieved the primary goal of reducing substance use in adolescents.

Dissemination of Findings

The findings of any program evaluation should be broadly disseminated to facilitate decision-making by other program implementers, stakeholders, and researchers. The ASAPS study dissemination process included annual reports and presentations to RWJF and other stakeholders. Presentations were made at professional conferences and a large number of peer-reviewed articles on the study have been published (in addition to those articles cited in this chapter, please see Brown et al. 2008; DesJarlais et al. 2006; Hammond et al. 2008; Merrill et al. 2006; Sloboda et al. 2008; Teasdale et al. 2013; Tonkin et al. 2008).

All student survey data were also made available to the public through the Inter-university Consortium for Political and Social Research (ICPSR), and is available to researchers and students at: https://www.icpsr.umich.edu/icpsrweb/landing.jsp. By providing other evaluators, program implementers, and researchers access to the data and findings for this evaluation, the evaluators anticipate improvements in not only in substance abuse prevention programming, but in evaluation research as well.