Background

Research has identified several evidence-based practices (EBPs) for the treatment of adolescent and young adult substance use disorders (i.e., youth SUD) [1]. One exemplar is the Adolescent Community Reinforcement Approach (A-CRA [2]), a 12–14-week behavioral treatment for youth ages 12–25 that improved substance use, mental health, and social outcomes in randomized trials (as summarized in a recent systematic review [1]). A-CRA has been widely implemented in the USA, where it was developed, and several other countries (Canada, Brazil, Spain, and the Netherlands). It embodies many common EBP features for youth SUD, such as the use of strategies to modify factors related to substance use, procedures to address family and community influences, and community-based delivery options [1].

Despite the strong evidence for EBPs like A-CRA, low rates of SUD service delivery (7.6% ages 12–17 and 4.4% ages 18–25) [3] suggest that an even smaller portion of the ~ 10 million youth with SUD receive effective services. Behavioral health administrators and policymakers need research-based guidance on how to increase the availability of EBPs for youth SUD through targeted implementation efforts [1, 4,5,6,7]. To produce generalizable knowledge that can guide administrators and policymakers, researchers may examine the implementation strategies that help systems, organizations, and/or individual providers (e.g., clinicians, supervisors) successfully implement an EBP [8,9,10].

A recent review highlighted the importance of financing strategies [11], a type of implementation strategy that helps systems and organizations secure financial resources to cover implementation-related expenses. Grants and contract funding (e.g., from government or philanthropic funders) were commonly reported examples, although the review identified 23 financing strategies in total. Financing strategies are essential because EBP implementation is often a complex, long-term, and costly process [12,13,14,15,16]. For A-CRA specifically, previous research found that lack of stable funding was a major barrier to its implementation, despite favorable provider perceptions of the model [17,18,19].

Federal agencies have the influence and resources to potentially increase EBP delivery nationwide through financing strategies [6, 20]. Unfortunately, little is known about how federal financing strategies should be structured to maximize implementation outcomes for any EBP, behavioral health setting, or type of financing strategy [11]. For example, the US Substance Abuse and Mental Health Services Administration (SAMHSA) issued several rounds of grant funding that supported A-CRA implementation; first, “organization-focused” grants issued directly to treatment organizations, followed by “state-focused” grants issued to state substance use agencies. Treatment organizations provide direct clinical services and thus are the service delivery context for A-CRA implementation, whereas substance use agencies are state-level entities that support the publicly funded services delivered by treatment organizations. As one of the largest investments in EBP implementation to date, SAMHSA’s initiatives offer a natural experiment to compare the outcomes of organization-focused versus state-focused grants on a large scale, but so far, no such evaluation is publicly available.

The present study evaluated how well the two aforementioned SAMHSA grant strategies promoted A-CRA provider-level reach, one of the several key outcomes for evaluating the impact of implementation strategies [21, 22]. Reach is defined as the extent to which an EBP is used within a service system or community [23]. Many studies conceptualize reach at the service recipient or client level, but it is equally important to understand provider-level reach—i.e., the proportion of eligible clinicians and/or supervisors who start using an EBP (sometimes called “penetration” [22]), since implementation strategies often target organizations or providers (e.g., training). We defined provider-level reach as rates of A-CRA certification (which designates whether each provider adopted A-CRA and delivered it with adequate fidelity), in recognition that those other implementation outcomes (adoption and fidelity [21]) are preconditions for reach to be meaningful.

Method

We describe this study using the Standards for Reporting Implementation Studies [24] (see Additional file 1). Relevant details are summarized here, but also see our published research protocol for more details [25]. All procedures were approved by the RAND Corporation IRB (Protocol #2020-N0887), including the use of data previously collected from organization-focused grantees [18, 19, 26] and new data collection from state-focused grantees.

Study overview

The current study used a longitudinal, mixed-method [27] design to compare A-CRA reach between state-focused and organization-focused grant recipients. Leveraging SAMHSA grant initiatives that all targeted the same EBP (i.e., A-CRA) and provided the same implementation support (e.g., training and consultation) allowed for a natural experiment to examine the impact of the two financing strategies: organization-focused versus state-focused grants. Our research questions, detailed in Table 1, were as follows: (1) How did A-CRA reach rates differ by grant type? and (2) What factors influenced A-CRA reach outcomes?

Table 1 Mixed-method design for A-CRA provider-level reach outcomes study by research question
Full size table

Given that the state-focused financing strategy was designed by SAMHSA to improve implementation outcomes beyond the organization-focused initiative, for research question 1, we tested SAMHSA’s hypothesis that reach would be higher among organizations participating in the state-focused grants relative to the organization-focused grants. Our design used a natural experiment, without random assignment of treatment organizations to the different grant types, and we prioritized producing externally valid evidence on the impact of each grant strategy in large-scale initiatives. This design does not fully isolate the effects of grant type on outcomes; however, we also tested the robustness of our findings against major internal validity threats with a series of sensitivity analyses.

We hypothesized for research question 2 that A-CRA reach outcomes would be moderated by two key factors shown to influence implementation: [28, 29] external financial support for services [13, 30] and internal leadership support [31, 32]. We used qualitative interview data to identify the barriers and facilitators of A-CRA implementation, providing a complementary qualitative data source on quantitatively measured reach rates in order to extend our understanding of how reach outcomes functioned in practice (a mixed-method complementarity-elaboration function [27]).

Project context

The project design and hypotheses are grounded in the Exploration, Preparation, Implementation, Sustainment (EPIS) framework [28, 29], which describes how public service organizations implement EBPs by navigating a multi-phase process influenced by contextual factors across multilevel domains (innovation, inner context, outer context, bridging factors). In the exploration phase, the SAMHSA Center for Substance Abuse Treatment [33] selected A-CRA for implementation; other EBPs were also included in some initiatives, but we focused on A-CRA to minimize variation in outcomes due to innovation (i.e., EBP) characteristics. Briefly, A-CRA [2] is a psychosocial treatment model that uses cognitive-behavioral and family therapy techniques (e.g., functional analysis, communication skills) to replace factors supporting substance use with alternative activities and behaviors; it can be delivered in individual or group formats, with caregivers attending some sessions alone or with the youth. Chestnut Health Systems (CHS), the organization that developed A-CRA and conducts A-CRA training and research, created a certification protocol that supports high-fidelity delivery across providers through the use of a treatment manual, initial training (with behavioral role-plays), observed practice in delivering A-CRA, and consultation support and feedback until certification is achieved [34].

Through its grant-funded initiatives, SAMHSA provided funding, oversight, and leadership for A-CRA implementation to (i) four cohorts of organization-focused grantees across 26 states, awarded between 2006 and 2010, followed by (ii) four cohorts of state-focused grantees across 22 states, awarded between 2012 and 2017. During the preparation and implementation phases, organizations or states applied for, received, and executed SAMHSA grants. All grants targeted inner context (intra-organizational) factors at treatment organizations through training and certification for A-CRA clinicians and supervisors; leadership from treatment organizations selected which providers were available and interested to participate in the training. State-focused grants further targeted outer context (extra-organizational) factors at the state level and bridging factors that connected state substance use agencies and treatment organizations; with these changes, SAMHSA aimed to maximize the number of providers offering A-CRA in their states and, thus, enhance reach outcomes relative to organization-focused grants. Reach outcomes are the culmination of the implementation phase, i.e., grant funding period, after which the sustainment phase begins. Next, we describe the two grant types, and Table 2 details their characteristics.

Table 2 Characteristics of organization-focused and state-focused grant strategies
Full size table

Organization-focused grants

SAMHSA awarded ~ $900,000 USD (across a 3-year period) to each treatment organization. These grants supported A-CRA implementation by paying for A-CRA delivery, A-CRA supervision, and other related activities. Our prior work showed that grantees commonly experienced initial success in implementing A-CRA with fidelity and reducing youth substance use, but difficulty sustaining the model post-funding [17,18,19].

State-focused grants

SAMHSA awarded ~ $3–4 million USD each (across a 3- to 4-year period; sometimes extended to 6 years) to state agencies that administered publicly funded SUD services. One-third of the grant funds paid for states to develop EBP-focused infrastructure, such as funding and training. Some grants required that state agencies propose “demonstration site” organizations, which established partnerships with the state agency and implemented A-CRA first before the state began broad dissemination to other organizations.

Participants

Treatment organizations

We used CHS certification records, conducted semi-structured interviews, and collected survey data from clinicians and supervisors at treatment organizations that implemented A-CRA (inner context). For interviews and surveys, we sampled organizations within the 5-year period following the completion of grant funding. Individuals who were currently or recently employed as a clinician or clinical supervisor responsible for youth SUD treatment were eligible to participate; those knowledgeable about A-CRA implementation were preferred.

For organization-focused grantees, all 82 eligible organizations (in 27 states) were invited for interviews. Organizations that participated in state-focused grants were more numerous, so we created a comparable sample of 82 organizations (in 18 states) by randomly selecting up to five organizations per state, including up to three “demonstration sites.” To ensure a comparable sample size for state-focused grants, we re-selected 17 organizations across nine states (because five had closed, 11 no longer provided youth treatment, and one had merged with a site that was already part of our sample); CHS records showed that selected and not-selected organizations did not differ on A-CRA certification rates (χ2(1) = 1.45, p = .23) or turnover rates (χ2(1) = 0.82, p = .36), even after replacement.

The interviewed sample included 154 organizations (94% of the 164 selected), with 249 provider participants (39% clinicians, 33% supervisor-clinicians, 28% supervisors). For CHS records, data from all trained providers (566 organization-focused, 417 state-focused) at the selected organizations were included. For 11 organizations that had providers trained under both grant types, we used the first (organization-focused) observation in our primary analyses, but we explored other ways of handling those organizations in sensitivity analyses (described later).

State substance use agencies

We also interviewed administrators from the state agencies (outer context) that received SAMHSA state-focused grants. Participants were currently or recently employed in a relevant leadership position and were eligible within the 5-year period after funding ended. Thirty-two administrators from 100% of the 18 eligible states participated; we conducted group interviews when multiple individuals from the same state agency participated.

Data sources and collection procedures

Table 3 summarizes the data collection activities and specific measures used to evaluate reach outcomes. CHS maintains a database of SAMHSA grantees, from which we extracted administrative records used to contact eligible clinicians, supervisors, and state administrators for interviews and surveys. Recruitment was done via email, phone, and/or mail, and data were collected via telephone or Microsoft Teams for interviews and Confirmit for surveys. Informed consent was obtained for each activity, no personally identifiable information was collected, and we de-identified participants’ data upon collection. We offered $25 USD for each interview and survey, although some participants (mostly state agency administrators) considered participation part of their job and declined compensation.

Table 3 Measures used to compare state-focused versus organization-focused grants on their A-CRA provider-level reach outcomes
Full size table

A-CRA certification records

CHS created a database of certification outcomes for all sampled treatment organizations, which specified any certifications achieved by each clinician or supervisor trained in A-CRA during the grant period as well as relevant descriptive information.

Semi-structured interviews

Interview protocols used a combination of open-ended questions and standardized probes [37]. Protocols were tailored to each participants’ role and, for treatment organizations, whether the organization was still delivering A-CRA (see Additional file 2 for protocols). The interviews explored state- and/or organization-level approaches to implementing A-CRA, how A-CRA implementation was supported during the funding period, sustainability planning, and (state administrators only) state infrastructure developed through their grant, as well as sustainment-focused questions for future analysis. Interviews were audio-recorded and transcribed (with transcripts de-identified) and lasted approximately 45–60 min.

Provider surveys

Following each provider interview, respondents were sent a web-based survey that collected standardized measures (including potential moderating factors) and other descriptive information (see Additional file 3 for all survey items), which again include sustainment-related questions not analyzed here. The surveys took approximately 25 min to complete. Of the treatment organizations interviewed, 90% had at least one survey completed.

Measures

Grant characteristics

Participants’ involvement in SAMHSA grants was determined from CHS records and later verified during the consent process and initial interview questions. Characteristics recorded included grant type, start and end years, and length.

Reach outcomes

We used CHS administrative records to define reach [23] as the proportion of providers in a treatment organization that were certified in A-CRA by the end of the grant period, out of all individuals who were eligible for certification (i.e., completed the initial training). A-CRA clinician certification is based on proficient demonstration of A-CRA procedures (i.e., clinical techniques or activities). “First-level” certification indicates proficiency in nine core procedures and is required for independent delivery of A-CRA; optional “full” certification indicates proficiency in an additional 10 procedures (19 total) [2]. Additional optional certifications are available to indicate proficiency with transition-age youth (TAY; ages 18–25), which adds two other procedures to the first-level and full certifications, and in A-CRA supervision, which is based on demonstrated ability to supervise within the A-CRA model and rate A-CRA procedures and session fidelity. We constructed six reach variables representing the proportion of participants who achieved any certification—which was the primary outcome of interest—as well as first-level, full, TAY (first-level or full), or supervisor certifications.

We also examined descriptive variables including time to certification, whether the participant left the organization (i.e., turnover status), and fidelity scores for rated sessions resulting in passing scores. It was important to verify that each certified individual achieved fidelity (i.e., adherence to the A-CRA model with adequate competence or skill), since fidelity is associated with client SUD outcomes [34, 38,39,40,41,42,43]. An average rating of ≥ 3 out of 5—based on a review of recorded sessions—is required across the relevant procedures for a given A-CRA certification [34, 38,39,40,41,42,43].

Moderators

We proposed two prespecified moderators of A-CRA reach outcomes [25], but the survey measures for these moderators could not be analyzed due to difficulties with scoring responses (“external support” measure) or high missingness at the organization level (leadership measure, which was only administered to clinicians). Instead, we used similar subscales from the Program Sustainability Assessment Tool, a measure of eight EBP sustainment capacity domains [35, 36]; this measure was administered to all participants, so all organizations with survey responses had data for the analysis (n = 138). We used the funding stability subscale to capture external financial support for A-CRA and the organizational capacity subscale for internal leadership support of A-CRA.

Descriptive measures of A-CRA barriers and facilitators

We collected other interview and survey data that measured constructs from EPIS. Innovation measures captured participants’ perceptions of and attitudes toward the A-CRA model. Inner context measures described the treatment organizations delivering A-CRA, outer context measures described extra-organizational factors that affected A-CRA reach, and bridging factors described factors that linked inner and outer contexts (including grant-funded activities). We asked about the impacts of the ongoing COVID-19 pandemic in the state-focused sample, but those data were only applicable to reach outcomes in three states with active state-focused grants in 2020.

Analysis plan

Quantitative data analysis

To compare reach rates between grant types, we fit a series of multivariable linear regression models with reach rate as the organization-level dependent variables. Each regression included grant type and several covariates: turnover rate, number of individuals trained, number of supervisors who pursued certification, grant end date, and quarters of grant funding. These covariates helped address threats to interpreting the “grant type” variable (e.g., grant end date controls for time trends) and account for important factors identified in our qualitative analyses (e.g., turnover). Moderator variables (funding stability, organizational capacity) were entered at the organization level, along with an interaction term with grant type. Moderator variables were averaged for all ratings at that organization; if moderator items were missing, we imputed their mean value from other organizations with the same A-CRA sustainment status.

The initial examination of reach rates revealed that both TAY outcomes had sample sizes too small for an adequately powered analysis (ns = 34 for first-level TAY, 29 for full TAY), per our initial power calculations [25]. Therefore, we report only descriptive statistics for those outcomes and focused our modeling on the following four outcomes: any, first-level, full, and supervisor certification. Furthermore, any certification was the primary outcome (it subsumes the other certifications), so we used the threshold p < .05 for statistical significance rather than controlling the experiment-wide error rate.

Supplemental and sensitivity analyses

We tested alternate model specifications such as beta regression, robust standard errors, and state-level fixed effects (with and without clustering standard errors), but the findings changed minimally so we report the basic models. We also conducted a series of sensitivity analyses that characterized threats to internal validity in our design. Specifically, we examined (a) observed secular trends using a non-equivalent dependent variable [44, 45], delivery of buprenorphine, and an evidence-based medication for opioid use disorder and (b) patterns of findings across sub-samples that lent insight into the impacts of grant type (for example, we examined outcomes for demonstration sites vs. other state-focused grantees). All details of these analyses and the findings are reported in Additional file 4.

Qualitative analysis of interviews

We analyzed interviews using conventional content analysis [46] to identify the barriers and facilitators to A-CRA reach during grant funding periods. As a starting point for our analyses, we used a codebook from a prior published analysis of interviews with organization-focused grantees [17]; the second author of this study led that work.

To analyze the state-focused interview transcripts, we developed a codebook in Microsoft Excel with code definitions and exemplars [47], organized by the four EPIS domains (outer and inner contexts, bridging factors, and innovation characteristics) and further classified by determinant type of barrier and/or facilitator. We then used the NVivo qualitative software program to organize and analyze transcripts, iteratively adding newly identified codes to the codebook. Two research assistants coded most transcripts, but two interviewers also contributed. The lead author provided feedback on the initial coding and reviewed specific passages on request throughout; the coding team also met biweekly to discuss the progress and refine the coding as needed.

Finally, we developed written summaries of the content of codes (with exemplar quotes), focusing on implementation-specific barriers and facilitators. The lead author and second author (who led the previous analysis [17]) incorporated details about the similarities and differences between the barriers and facilitators that participants described for each grant type. All co-authors reviewed and helped further revise the summaries to ensure credibility and completeness.

Results

Reach outcomes

Table 4 reports reach rates for each A-CRA certification; for example, the average percentage of eligible individuals who received any certification was 72% for treatment organizations under organization-focused grants versus 36% for organizations under state-focused grants. The table also reports descriptive statistics for covariate, moderator, and certification process variables. Note that the average fidelity scores for certified providers ranged between 3.5 and 4.1, with all values ≥ 3 (i.e., adequate fidelity).

Table 4 Descriptive statistics for organizations that implemented A-CRA through organization-focused and state-focused grants
Full size table

Table 5 presents estimates from regression models which examined whether grant type (organization-focused vs. state-focused) was associated with reach outcomes. Controlling for covariates, the rates of any A-CRA certification in organization-focused grants were 27 percentage points higher (p = .01), and the rates of first-level certification were 23 percentage points higher (p = .04), compared to state-focused grants. These models explained a large portion of the variance in outcomes (R2 = .33 and .22). There were no significant differences in full or supervisor certification rates.

Table 5 Regression models for association of grant type and A-CRA provider-level reach outcomes
Full size table

Sensitivity analyses

The detailed results of the sensitivity analysis regression models are reported in Additional file 4. Briefly, we found that observed differences in reach outcomes did not result from confounding temporal trends or the characteristics of states or organizations but instead were strongly associated with grant type. For the non-equivalent dependent variable, grant type was not associated with the likelihood of an organization delivering buprenorphine. For the sub-sample analyses, the pattern of results was similar to the main analysis across sub-samples.

Moderator analyses

Neither tested moderator showed significant effects for the sub-sample of organizations (84%) with survey responses. Coefficients for the main effects on any certification were as follows: funding stability, 0.03 (p = .18), and organizational capacity, 0.00 (p = .91). Interaction effects with grant type were as follows: funding stability, 0.01 (p = .88), and organizational capacity, 0.07 (p = .07).

Barriers and facilitators identified in qualitative interviews

Table 6 reports the demographic characteristics of the providers who completed the interviews. For state administrators, we only collected information about professional experience, with the primary respondent(s) in the interview generally providing the information (21 of 32 administrators [66%] responded): experience with administering substance use treatment services ranged from 1 to 30 years (M = 14.2, SD = 8.3), and when relevant, experience providing substance use treatment services ranged from 1 to 25 years (n = 14; M = 11.7, SD = 6.4).

Table 6 Demographic characteristics of provider interview samples by grant type
Full size table

From our coding of those interviews, we summarized the key barriers and facilitators to A-CRA reach. Findings are presented by EPIS domain (innovation, inner context, outer context, bridging factors). The written summaries give an overview of all findings but emphasize the key differences between contextual influences on organization-focused and state-focused grants. Table 7 lists each of those key differences in further detail, highlighting how each finding contributed to our interpretation of quantitative reach outcomes. Finally, we provide an expanded version of that table in Additional file 5, which includes summaries of all identified codes (with illustrative quotes) and details of which participants reported each code by grant type (organization-focused, state-focused) and role (provider, state administrator); most codes were reported by all grant types and roles.

Table 7 Summary of key barriers and facilitators to A-CRA provider-level reach outcomes from qualitative interviews
Full size table

Innovation: characteristics of A-CRA

Respondents described numerous innovation factors as strengths of A-CRA, such as its content and procedures, structured format, and evidence-based status, and the training and certification support offered by CHS. These facilitators were often endorsed regardless of an individual’s certification status, though some respondents who stopped using A-CRA described its being overly structured and inflexible as a barrier. Respondents also broadly described the A-CRA certification process as complex and burdensome (whether they completed it or not), which was a major barrier to reach. A key difference between grant types was that state-focused grantees reported more frequent and serious barriers around fit of A-CRA with their client population compared to organization-focused grantees; across grant types, A-CRA being a good fit for the youth and families served was considered an important facilitator. Overall, providers had more detailed responses about A-CRA characteristics, though state administrators reported similar perceptions based on their discussions with providers.

Inner context: treatment organization factors

Respondents across roles identified organizational characteristics that facilitated reach of A-CRA, including leadership, supervisors, and clinicians who supported the model; “champions” who helped provide resources and encouraged certification; and organizational policies that aligned with A-CRA’s use (e.g., availability of client incentives). Conversely, the lack of support from key individuals (often related to the negative perceptions described under the innovation domain) and misalignment with organizational policies (e.g., strict time requirements for direct service delivery) created barriers to A-CRA reach. Turnover of providers and leaders was another major barrier, especially when it disrupted support for A-CRA or required the replacement of certified providers. Finally, one prominent facilitator uniquely described by organization-focused grantees was an engagement in strategic planning to promote the adoption and use of A-CRA. State-focused grantees described minimal strategic planning around A-CRA, so this facilitator was rarely present for them.

Outer context: community, state, and national factors

State administrators provided rich perspectives on outer context factors that influenced A-CRA reach, though providers commented on similar factors. Across grant types, states’ policy contexts influenced A-CRA reach; for example, policies that encouraged the provision of EBPs facilitated A-CRA use, whereas a common barrier was limited reimbursement for non-service components of A-CRA (e.g., certification and supervision activities). Among state-focused grantees, some respondents described state substance use agency administrators as helpful in guiding the implementation process, which also encouraged organizational buy-in. However, state administrators also described challenges to fulfilling that role, with the specific challenges varying from state to state (e.g., limited support from state leaders, decision-making power lying with local authorities, turnover of state agency staff). Finally, some state-focused grantees also described attitudes among caregivers, community members, and government agencies that increasingly de-prioritized addressing youth substance use (perhaps related to increased legalization of cannabis)—which decreased youth referrals and engagement in A-CRA.

Bridging factors: treatment organization links to external resources

The most salient bridging factor for many respondents was their SAMHSA funding. For state-focused grants, state-led training activities and comprehensive funding for A-CRA training, certification, supervision, and delivery were identified as major facilitators of reach, bridging between outer context (state-focused grant) and inner context (enactment of grant activities at treatment organizations) through statewide coordination of activities. However, providers noted that some state administrators did not encourage the use of A-CRA or help identify referral sources, creating barriers to reach. A few states were able to extend efforts beyond training and funding into policy development (e.g., developing new funding mechanisms to support A-CRA) and broader implementation supports (e.g., statewide learning collaboratives); such activities promoted reach but were rare and challenging to execute due to aforementioned state policy barriers. Organization-focused grants provided similar benefits as state-focused grants (i.e., training and comprehensive funding) through a more direct mechanism, albeit to individual organizations only. Organization-focused grantees’ experiences differed in that they worked with SAMHSA directly, a collaboration often described as helpful but also as intensive and overwhelming (e.g., due to grant-related assessment and reporting requirements). Treatment organizations involved in state-focused grants typically had fewer requirements and less interaction with grant administrators (state or SAMHSA). Finally, providers considered their partnerships with other organizations for referrals and on-site service delivery (e.g., schools, juvenile justice) as important facilitators (or barriers, when ineffective) across grant types.

COVID-19 impacts

For the three states where the COVID-19 pandemic overlapped with their state-focused grant, participants described several impacts on A-CRA implementation. Effects on A-CRA delivery (innovation domain) were mixed; despite the benefit of increased use of telehealth as an accessible treatment modality, providers noted that some clients experienced connectivity and equipment issues, making it harder to engage them in A-CRA via telehealth. Respondents also described how the pandemic exacerbated existing barriers like staff turnover (inner context) and difficulties working with partners for A-CRA referrals (bridging factors).

Discussion

Effective financing strategies are needed to support EBP implementation in SUD services. We compared two financing strategies from the SAMHSA on the implementation outcome of A-CRA provider-level reach (i.e., certification rates). The shift from organization-focused to state-focused grants was associated with a 27 percentage point decrease in provider certification rates at treatment organizations, counter to our hypothesis. This finding appeared to be driven by differences in required first-level certification, with rates of the optional full certification < 1/3 in both grant types. We used a rigorous non-experimental design, with extensive sensitivity analyses supporting our interpretation that grant type was the major contributor to variation in reach, and confounding temporal, state, or organizational factors did not explain the findings.

Interviews with providers and state administrators suggested that achieving A-CRA certification was challenging and resource-intensive regardless of the type of financing strategy. Barriers and facilitators spanned all EPIS domains, as well as the construct of interactions and relationships during the implementation process (e.g., need for alignment among individuals in different roles within the inner context, partnerships with outer context organizations), which has rarely been reported in studies using EPIS [28, 29]. Importantly, organization-focused grantees benefitted more from certain facilitators, such as strategic planning around A-CRA and more intensive involvement with SAMHSA, whereas state-focused grantees encountered certain barriers more frequently, such as difficulties for state administrators in executing implementation support and engaging with treatment organizations or A-CRA being perceived as a poor fit for the organization’s client population. These qualitative findings helped lend valuable insight into why reach rates were lower under state-focused grants—especially since we could not identify quantitative moderators.

We designed this mixed-methods study to provide an in-depth understanding of financing strategies and identify the practical implications for policymakers. The findings contradicted our hypothesis, which makes it even more imperative to use this research evidence to inform EBP financing strategy design. Indeed, SAMHSA created the state-focused grant mechanisms in response to concerns about organization-focused grants, which in our earlier work [17,18,19] had declining rates of sustainment—57% of organizations discontinued A-CRA by 3 years post-grant. Therefore, policymakers need to consider the strengths and limitations of both types of financing strategies when deciding which to use or which warrants scalability—as well as consider other financing strategies not tested here. We plan to conduct focus group discussions with state and federal policymakers to help identify implications of our findings; for example, we will ask policymakers what strategic planning strategies could help state agencies best navigate systemic challenges (e.g., state policies, budget priorities) when scaling up EBP initiatives.

State-focused grants might have additional benefits not identified in our organization-level analysis. Reach at the state level (i.e., number of certified A-CRA providers relative to population of youth with SUD) could be higher in state-focused grants, which trained a much larger number of organizations than received organization-focused grants. Another important future direction is our planned state-level case study analyses [48, 49], in which we will incorporate additional data sources (e.g., state-level reach rates, documents provided by state administrators) to compare and contrast states’ state-focused grant activities and outcomes [25]. The effectiveness of state-focused grants likely depends on each state’s policy context—which can vary dramatically [50]. This conclusion is supported by our findings that certain states were successful at developing infrastructure to support ongoing A-CRA implementation (e.g., new funding mechanisms).

It is also possible that state-focused grants produced higher A-CRA sustainment rates, as SAMHSA intended, among certified providers—though this seems unlikely given that successful initial implementation (e.g., reach) provides the necessary foundation for later sustainment [51]. Responding to calls to better articulate strategies for EBP sustainment [52], we are now measuring A-CRA sustainment among the state-focused versus organization-focused grant-funded programs. Ultimately, we expect that the financing strategies with the greatest policy impact will be those that produce both widespread and long-term implementation of EBPs, maximizing public health impact [53].

Our study builds on a small but emerging area of research on provider-level reach rates as an implementation outcome. Two previous studies examined provider reach for SUD-specific treatments—contingency management at an opioid treatment program [54] and alcohol and opioid use disorder treatments in a collaborative care service delivery model [55]—but examined only one or two implementation sites. Other studies examined system-level reach rates of youth EBPs including multidimensional family therapy for SUD and behavior problems [56], parent-child interaction therapy for behavior problems in young children [57], and six different mental health EBPs fiscally mandated for adoption in Los Angeles County [58, 59]. None of these studies compared reach rates between alternate implementation strategies, but all of the system-level studies identified financing strategies as important for sustaining EBPs in public behavioral health systems. The Los Angeles County study is a rare example of research examining implementation outcomes (including reach) from financing strategies [58]. The current study went beyond that county-level analysis by examining reach outcomes from different federal-level financing strategies (grants), for an SUD-focused EBP, and across various states and treatment organizations.

Researchers hoping to conceptualize their own studies of financing strategies in behavioral health, medicine, public health, and/or prevention may find it instructive to consider recently published recommendations from a policy adaptation of the EPIS framework [60]. Those recommendations were to specify the function and forms of the policies of interest (e.g., what resources state-focused and organization-focused grants provided, how those resources related to A-CRA reach outcomes); describe phases of policy implementation across inner and outer contexts (e.g., EPIS guided our data collection, analysis, and interpretation) and bridging factors [61] (of which financing strategies are one example [62]); and identify the temporal roles of policy actors such as policymakers and administrators (e.g., we distinguished barriers, facilitators, and outcomes in implementation vs. sustainment phases). We have not yet addressed the sixth recommendation, considering outer and inner context adaptations, but we intend to assess how contexts could be adapted to support A-CRA reach in the policymaker focus groups.

Despite its strengths, this study had several limitations. First, this work cannot directly inform efforts to improve the feasibility of the A-CRA certification process (e.g., identifying pragmatic fidelity measures [63]), which could make it easier to finance A-CRA as well as achieve full or supervisor certification. Second, we were not able to evaluate clinical outcomes of A-CRA; we confirmed that providers had adequate fidelity scores, which are associated with clinical outcomes [38, 39], but monitoring clinical impacts remains a major challenge for most EBP implementation initiatives. Third, we focused on provider-level A-CRA reach, but client-level reach must be separately considered; e.g., a recent analysis of the fiscally mandated EBP adoption in Los Angeles County found only 17% of youth in need of EBPs received them, with further disparities for minoritized ethnic groups and immigrants [64]. Fourth, we did not assess adherence to required grant activities, which could have introduced unmeasured variability within each grant type; our case studies will consider this issue to the extent possible. Finally, some aspects of our study design made the planned moderator analyses challenging, most notably missing data and interpretability issues (described under the “ Method” section) and survey responses being collected post-implementation, which may not represent how a factor operated during implementation. To balance that limitation, we maximized the explanatory power of qualitative data to identify influences and contextualize the experiences of our participants.

Conclusions

This study is the first published comparison of reach outcomes between alternate financing strategies to support EBP implementation. This work fills a critical gap in how implementation research findings can guide administrators’ and policymakers’ investments in EBPs for youth SUD. Having identified critical issues in A-CRA reach and sustainment outcomes for state-focused and organization-focused grant financing strategies, we are poised to begin collaboratively exploring policy solutions. We hope that this work will encourage researchers, policymakers, and system/organization leaders to collaboratively study optimal ways to implement diverse EBPs in financially sustainable strategies, including the use of novel analytic approaches (e.g., non-equivalent dependent variables). Continued efforts will be essential to understand how these strategies operate with service and administrative systems and organizations to ultimately improve the effectiveness of providers and the lives of clients.