Introduction

Co-occurring mental health and substance use disorders are prevalent, have serious consequences, and remain undertreated—especially with evidence-based practices (EBP).1,2,3,4 Many individuals with opioid use disorder (OUD) have co-occurring disorders such as post-traumatic stress disorder (PTSD) and/or major depressive disorder (MDD).5,6,7 Most individuals with OUD have experienced trauma and close to two-thirds go on to experience co-occurring PTSD.6,7,8 Approximately half of those with OUD also have lifetime major depression9 and 20% have a 12-month diagnosis of a mood disorder.10 Undiagnosed and untreated co-occurring disorders may interfere with treatment of OUD and with return to baseline functioning.

Primary care is an opportune setting to deliver treatments for co-occurring substance and mental health disorders because individuals with substance use disorders, including OUD, tend to access primary care more frequently than specialty substance use settings.11,12 The Collaboration Leading to Addiction Treatment and Recovery from Other Stresses (CLARO) initiative is one of the first studies to evaluate a collaborative care model for co-occurring disorders in primary care settings.13,14 The multisite CLARO randomized controlled trial (RCT) assesses the effectiveness of a collaborative care intervention for OUD co-occurring with PTSD and/or depression, implemented in low-resource settings.11,15

Various factors within any organizational setting may influence the implementation of interventions to treat those with co-occurring disorders. In primary care, unique challenges may surface in part because both substance use and mental health disorders tend to be underidentified and undertreated in primary care settings. For example, patient characteristics, such as having multiple disorders that may interact, might pose a challenge to primary care providers or the organization (health system) itself may not be adequately equipped to treat individuals with co-occurring disorders. Broadly, factors like the organizational culture of a clinic may impact the extent to which new practices are adopted by providers. Similarly, an intervention may show positive outcomes however lack of leadership support may complicate the extent to which practices are sustained even if financial resources are stable. For example, previous research on co-occurring substance use disorder treatment has documented contextual barriers including the importance of administrative leadership; challenges related to staff turnover, training and supervision; and financial considerations like potential reductions in billing or process changes to financial services.16 These potential factors, while not necessarily exclusive to co-occurring disorder interventions, point to the value of understanding contextual factors in relation to interventions for co-occurring disorder.

Understanding the range of contextual factors, or determinants, of sites adopting an EBP can be critical to the success of its implementation.17 The first 2 years of CLARO (2020–2021) focused on implementing the collaborative care model in 13 clinics operating within three distinct healthcare systems in a western US. Understanding the context of these clinics early in the intervention implementation process was considered an important component of implementing the CLARO RCT. In particular, it seemed probable that organizational factors of the participating sites might impact the way collaborative care was implemented at each site, as well as its potential sustainability.

For these reasons, it is vital to understand the landscape of each organization prior to implementing an intervention and treat it as part of the implementation planning process. Identifying potential barriers and facilitators to implementation may reveal areas that are in need of additional support or that should be tracked over time.18,19 This study contributes to the developing research area that links pre-implementation research to implementation plans.17 In a study aimed at testing psychometric Consolidated Framework for Implementation Research (CFIR) measures, Fernandez et al.20 argue that inner setting constructs—those that focus on organizational context specifically—can be used to inform or enhance implementation approaches; for example, shining a light on deficient constructs can lead to making or measuring changes to those areas over time. Weir et al.21 took this approach further by identifying implementation barriers and then developing strategies based on the CFIR to help facilitate the implementation of a cluster randomized trial in hospitals.

The current study describes results of a survey administered in three healthcare systems during the pre-implementation phase designed to assess contextual factors that may influence implementation of the CLARO intervention. Guided by the well-established CFIR19,22,23 comprising 37 characteristics across five domains believed to influence implementation of EBPs, this research examines the question, how do perceptions of contextual factors vary among the three systems, and across types of staff? The discussion section also addresses how the research team used these results to inform the RCT.

Methods

Overview of study design

The purpose of the survey was to develop a pre-implementation understanding of organizational context of clinics participating in the CLARO RCT. The target population for the survey was clinic staff in the primary care clinics, including clinicians. The intent was to survey as many staff members from each healthcare system as possible, and to gather data from staff working in a range of clinic roles, as compared to taking a sampling approach.

The survey included informed consent language that explained the survey to respondents and specified that their participation was voluntary, that they could skip items or stop taking the survey at any time, that personal data would be kept confidential, and that data were for research use only and would not be shared with outside parties. Those invited to take the survey were entered into a sweepstakes for the chance to win one of 13 $100 gift cards for at least one winner per clinic.

To pre-test the survey, some CLARO team members and representatives from the healthcare systems were invited to test the survey and provide feedback about question wording and the length of time it took to complete the survey.

Recruitment procedures and survey administration

A total of 541 staff at the 13 clinics were sent emails asking for their voluntary survey participation. Inclusion criteria included all staff members including clinicians and other clinic staff members who were likely to interact with patients that may qualify for CLARO. Exclusion criteria were staff who worked in dental, facilities, or other areas in which interaction with potential CLARO patients would be unlikely. The healthcare system leaders most directly involved with the CLARO research team also did not take the survey because of their close involvement with the CLARO RCT; in some cases, they were asked to provide input on the survey prior to launch (e.g., to test the length of time the survey would take). Types of staff that took the survey included medical providers (including physicians, physician assistants, and nurses), behavioral health providers, and other non-clinician staff (including medical assistants and key administrators). This approach allowed for obtaining a relatively comprehensive view of each clinic.

For the three healthcare systems, CLARO RCT start dates (meaning the date when patient enrollment into CLARO was scheduled to begin) ranged from November 2020 to February 2021. The pre-implementation survey was administered to clinic staff members in the 3–4 weeks prior to the date on which RCT enrollment began at each clinic. Because of restrictions related to COVID, no in-person activities in the clinics were possible, and so all facets of survey administration were conducted virtually.

In each system, a clinic champion or administrator sent an email to all staff the week prior to survey launch to let staff know they may receive a legitimate (i.e., not spam) survey invitation email connected to CLARO. These emails also included language that mirrored the IRB-approved survey language to emphasize the voluntary nature of the survey. Survey respondents received an email with a unique URL for the survey from the RAND Survey Research Group. Any respondents who had not yet completed the survey received a weekly email reminder while the survey remained open.

Response rates

Among the three healthcare systems, response rates ranged from 50 to 68% across systems and 56 to 85% across role (Table 1), which is considered acceptable for a voluntary, organizational survey.24,25 Table 7 in Appendix 1 provides demographic information about the respondents.

Table 1 Response rates (RR) by role, across healthcare system
Full size table

System 1, a Federally Qualified Health Center (FQHC), contained clinics located in both urban and rural parts of the state. System 2 was also an FQHC with clinics located in a rural part of the state, different than system 1. In system 3, which was affiliated with a university, clinics were located in an urban area.

Measures

This study adapted previously validated CFIR scales from Fernandez et al.20 and Kegler et al.26 and incorporated the Brief Opioid Stigma Scale from Yang et al. (Table 2).27 At least one scale related to each of the five CFIR domains. The scales from Kegler et al.26 were developed and validated by those authors to help researchers increase the likelihood of implementation success during any stage of implementation including during the planning phases. Their set of measures addressed each of the five CFIR domains and were easily adaptable. Similar to Kegler et al.,26 Fernandez et al.20 developed measures focused on the Inner Setting CFIR domain. Their study provided scales useful for examining facets of culture, climate, leadership, and resourcing. Yang et al.’s27 Brief Opioid Stigma Scale, a 6-item scale focusing on perceptions about individual and community-level stigma, was also included in the survey. Though not part of the CFIR, given the focus of the CLARO RCT, the individual section of the stigma scale contributes to the understanding of Inner Setting, and the community-level section, to Outer Setting. These measures provided coverage of concepts across the major CFIR domains. Additionally, the authors reviewed concepts to make sure they made sense to the context of the CLARO RCT. For example, the survey did not include items on constructs such as Design and Quality of Packaging (a CFIR Intervention Characteristic construct) or External Policies and Incentives (a CFIR Outer Setting construct) because these were less salient to the CLARO RCT and to the aim of this research focused on understanding the clinic setting.

Table 2 Summary of survey measures
Full size table

All scales used in the survey were 5-point Likert scales (strongly disagree to strongly agree). To adapt these scales to the context of CLARO, minor edits were made to item wording (e.g., using the term “clinic”). Additionally, given the pre-implementation timing of the survey, the constructs chosen reflected the types of antecedent constructs that Damschroder et al.22 later identified; for example, items that were evaluative or retrospective were not included (e.g., items like, the intervention was “used by colleagues who were happy with it”26(p.4184)) since this survey was administered prior to the start of the intervention.

Survey analysis

In the survey, all respondents were asked to complete CFIR scale and demographic items. Behavioral health providers (BHPs; include those with LADAC, LCSW, LMHC, LMSW, LPCC, or PsyD credentials) and medical providers (MPs; include those with MD, PA, NP, or RN/LVN credentials) were asked some additional questions at the end of the survey about their clinic workflow (e.g., extent to which their appointments were in-person or virtual). Any respondent that answered 80% or more of the core questions were included in the analysis. This 80% cutoff rule was used because when missing data were analyzed, an 80% cutoff typically included respondents who had answered at least some items in the main sections of the survey (e.g., respondents that answered the CFIR questions but then did not complete the demographic items located at the end of the survey).

In the survey, questions asked about the respondents’ clinic environment, not about the healthcare system as a whole. Responses were aggregated to the healthcare system level to protect the confidentiality of respondents (e.g., some clinics had fewer than 30 staff total), and to facilitate a comparison across the three distinct systems. Furthermore, each system relied on system-wide top leadership teams, and in some instances, staff members worked at more than one site in a system.

Constructs were analyzed based on percent of favorable responses in which ratings of agree or strongly agree were considered favorable in order to gauge relatively strong versus weak context areas. Most survey items were phrased in an affirming way (e.g., “This clinic does a good job…”). Therefore, in most instances, construct scores that fell below 50% favorability (e.g., in which less than 50% agreed or strongly agreed) were considered areas for improvement. Three constructs, complexity and the two stigma scores, were exceptions in that due to question wording, high percent favorable ratings indicated deficits or areas for improvement. Scores were not reversed for those three constructs because it was simpler and clearer to keep those items as is; for example, it was unclear if reverse scoring the items would reflect respondents’ survey selections (e.g., disagreeing or strongly disagreeing that “most people believe that a person who is addicted to opioids cannot be trusted” may not be the same as agreeing or strongly agreeing that “most people believe that a person who is addicted to opioids can be trusted”).

Results

The main research question addressed how organizational factors varied among the three systems and across staff. Table 3 compares scores across the three healthcare systems. To depict how constructs varied by role (Table 4), respondents from all healthcare systems were combined and then data were broken out by role. This allowed us to examine role-based differences without unintentionally identifying individuals due to small sample sizes.

Table 3 Percent of favorable responses by healthcare system
Full size table
Table 4 Percent of favorable responses by role
Full size table

There were several similarities across the three healthcare systems. Notably, most areas for improvement focused on inner setting (organization level) CFIR constructs. Culture stress, level of resources, and complexity were low-scoring areas when data were analyzed by healthcare system, indicating that these were particularly salient areas for potential improvement in all three systems. Perceived community stigma was consistently high among the three systems, whereas perceived individual stigma was consistently low.

Among the three systems, individual-level factors tended to be areas of strength. For example, openness to innovation (healthcare system average: 92%; role average: 81%) and knowledge/beliefs about the intervention (healthcare system average: 93%; role average: 79%) both had high favorability ratings.

In terms of variation across systems, system 2 results indicated some additional low-scoring areas for improvement in learning climate, leadership engagement, and reflecting and evaluating. Additionally, for some areas for improvement common to the three systems, system 2 had lower favorability scores than the other two systems, such as with level of resources (28% favorable versus 49% for system 1 and 44% for system 3).

As Table 4 indicates, role-based areas for improvement were similar to the system-level comparison data. Culture stress, level of resources, and complexity were low-scoring areas among all roles. BHPs and RNs both indicated relative advantage as a low-scoring construct, and BHPs specifically indicated engaging champions as a low-scoring area to watch. Similar to system-level results, perceived community stigma was consistently high (close to 50%) among those in different roles (with the exception of RN staff members who had an item n = 0). Tables 5 and 6 summarize the most salient areas for improvement by system and by role.

Table 5 Summary of areas for improvement represented by range with a low of less than 50% favorable across the 3 healthcare systems
Full size table
Table 6 Areas for improvement across all clinic staff by role
Full size table

Discussion

This study presents a unique contribution to implementation literature by applying quantitative CFIR measures to assess pre-implementation contextual data about three healthcare systems. Rather than focusing on a single aspect of the CFIR, the survey built on previous research to capture insights from each CFIR domain and added additional insights about stigma perceptions. Surveying staff in different roles allowed for a comprehensive view of clinic context, and the research team used the findings to inform the implementation.

By understanding both weaknesses and strengths, these results provided useful feedback to the CLARO research team about the healthcare systems in the pre-implementation period. Overall, there were several consistencies across the three systems and across roles. Most areas for improvement at the system- and role-levels focused on inner setting (organizational focused) constructs whereas individual-level (staff focused) constructs tended to be areas of strength. Perceived community stigma also tended to score relatively high across systems and across roles. Some survey findings seem analogous to contextual factors identified in previous research such as concerns about resources.16

With regard to variation, system 2 results indicated some additional areas for improvement with inner setting and process constructs when compared to systems 1 and 3. Notably, as the CLARO RCT progressed, some challenges with system 2 gradually developed, shedding light on the pre-implementation survey findings. Emergent challenges included a complex and hierarchical leadership approach, combined with high staff burnout and patient skepticism about the health system and providers. As an example, in system 2, the research team eventually learned that while the population included many who could technically qualify for CLARO, many clinic staff believed they would not be able to recruit patients due to perceived stigma in the rural community (e.g., it was highly likely that a given patient would know everyone working in the clinic, from the front desk staff administrator to the pharmacist, so some worried patients may fear a damaged reputation). Implementation was later discontinued in system 2 due to low levels of adoption and fidelity to the intervention model.

There was a fair amount of consistency when comparing survey data by role in that some inner setting and intervention constructs stood out across most roles similar to system-level results. In terms of variation, BHP results indicated that the process construct, engaging champions, was an area for further examination. Additionally, MPs seemed to have more favorable views about relative advantage as compared to BHPs and RNs. The possible reasons for those differences are less clear and would benefit from additional research.

This research builds on the work of Fernandez et al.20 and Weir et al.21 by not only capturing contextual data from sites participating in an EBP in a pre-implementation stage, but also in the work the research team did to report and discuss the findings with the participating healthcare system leadership. A 2-page summary report was developed for each healthcare system that showed the range of responses for each construct. The goal was to provide the healthcare systems with objective feedback that they could use to reflect on CLARO RCT implementation as well as about their healthcare system writ large. Targeted discussions were held with clinic leaders where reports were discussed.

Discussions with clinic leaders focused on the many apparent strengths as well as areas for improvement including the topics, culture stress, resources, complexity, and perceived community stigma. Strengths were considered areas that clinic leaders and staff members could build on to support the CLARO RCT. For example, while many leaders might worry that employees will be resistant to something new, the high-scoring individual-level characteristic, openness to innovation, showed that staff members in these healthcare systems tended to be open to new initiatives. This finding provided a strong starting point for conversations with staff about CLARO as a new initiative.

In a second example, in one healthcare system, discussions addressed how the survey findings related to ongoing concerns about employee burnout, and what that meant for implementation. CLARO team members met with clinic leaders to talk through the steps that clinic leaders planned to take to support their clinic staff members with issues related to stress, lack of resources, burnout, and related issues like leadership turnover.

Additionally, because there were concerns about community-level stigma in all three healthcare systems, researchers spoke with leaders from each system about the nuances of what community stigma looked like in their geographic areas to help determine the biggest potential challenges related to implementation (e.g., if stigma was prominent among certain types of patients specifically, such as the unhoused).

Through these types of conversations, CLARO team members were able to probe some of the nuances evidenced by survey data to support strategies for the implementation. The CLARO team did not change IRB-approved implementation protocols, but they were able to provide suggestions or ideas about aspects of the implementation that fell within the RCT’s control to some extent (e.g., ideas for champions and other coordinators), and suggestions for clinic leaders in support of their healthcare systems. Survey data discussions were also helpful in building relationships among the CLARO team and clinic leaders.

Limitations

It is notable that this survey was launched during the period in which COVID cases were rising, prior to availability of vaccines. The participating clinics shifted to primarily virtual visits during the survey period; for some providers, like BHPs, all visits were temporarily virtual. The clinics started to increase in-person health care visits in spring of 2021, shortly after the surveys were closed. This also meant that the researchers could not engage in any in-person activities to inform staff about the survey.

This study faced some additional limitations. First, by attempting to capture pre-implementation data, some responses to items about the intervention were speculative by design. The true extent of barriers (or the emergence of unanticipated barriers) was unknowable at the time. Future research could compare pre-implementation data with follow-up data to track barriers over time. Second, methodological limitations include the potential for response bias (e.g., those with the strongest opinions tend to respond) and missing data (with a voluntary survey, it is not realistic to expect high response rates). Additionally, while surveys allow for breadth of data, they do not allow for depth. Knowing this tradeoff, the authors chose to administer a survey in order to gather a breadth of data across members of the three healthcare systems. Alternative approaches, such as interviews, would have made it more difficult to gather data that reflected a wide range of clinic staff roles.

Third, while there were some functional differences among clinics systems (for example, one system uses a group format for delivering medications for opioid use disorder while another has more limited capacity for in-house counseling), clinics within systems shared many similarities, which could have contributed to the broad similarities found in survey results. Future research can use mixed method techniques to supplement survey data with interview, or other, data to examine a broader range of research questions. Fourth, CFIR’s menu of constructs is valuable in that it can be tailored to a study’s situation; however, at the same time, it would have been too burdensome to test all 37 CFIR constructs in survey form. That meant some constructs that were less relevant to the context of the CLARO RCT had to be omitted. Finally, analysis was aggregated to the healthcare system level; further research could explore ways of taking a deeper look at individual clinic-level differences while still protecting the confidentiality of staff members.

In future studies, researchers can further develop procedures or tools for using pre-implementation CFIR data to inform (1) rapid feedback loops with clinic leadership and (2) implementation strategies. Pre-implementation data can be examined for associations with patient outcomes at a later point in time and can be useful for assessing the generalizability of study findings and planning for future dissemination of collaborative care efforts. Looking at the CFIR data post-implementation can also help to identify features of clinic context that are associated with implementation outcomes; to guide exploration in determining which factors influenced effectiveness outcomes by using the CFIR constructs; and to identify the features of the intervention that were implemented with the greatest success or that were likely to be sustained beyond the RCT.

Implications for Behavioral Health

Findings from this study have implications for understanding the implementation of complex behavioral health interventions into the primary care context. By using a pre-implementation approach and incorporating stigma measures, the authors contribute to Damschroder et al.’s22 goal for the CFIR to evolve over time. This work built on previous literature by using validated scales to increase the understanding of the healthcare systems, and the authors sought insights from staff members in different roles including clinicians and non-clinicians, as compared to solely clinic leaders or just one type of personnel. Past research has shown some CFIR determinants to be correlated with implementation outcomes.28 This approach, using an organizational survey to examine healthcare systems, also reflects a practical methodical approach that other multisite studies can use. Additionally, holding constructive discussions with clinic members about survey findings reflects an approach for providing survey feedback to participating clinics and for connecting with clinic representatives on both strengths and potential weakness or challenge areas that could impact the implementation of the RCT. This type of constructive feedback loop can be a helpful takeaway for both researchers and clinicians.