INTRODUCTION

Strategic efforts to disseminate and implement evidence-based interventions can speed wide adoption in practice and improve health outcomes.1 The Agency for Healthcare Research and Quality (AHRQ) established the Patient-Centered Outcomes Research (PCOR) Dissemination and Implementation (D&I) Initiative to identify and prioritize PCOR findings that could improve quality and health outcomes by closing gaps in practice if widely implemented.2,3 PCOR findings include those from studies comparing preventive, diagnostic, treatment, or health care delivery approaches for impact that is meaningful to the patient.4 Huppert et al. have previously described the D&I Initiative’s sequential prioritization process for PCOR findings.5 Briefly, PCOR findings are identified through public nomination, assessed for strength of evidence, potential for health impact, implementation feasibility, and alignment with AHRQ priorities. The D&I Initiative recommends implementation projects for prioritized PCOR findings to AHRQ leadership to consider for funding through the PCOR Trust Fund.

Impactful PCOR findings with good strength of evidence may not be feasible to implement in clinical practice if, for example, access to necessary components of the approach is too limited for stakeholders to adopt readily, or implementation is too costly or complex. Feasibility is the “cumulative impact of different influences that have an effect on implementation of an intervention within a specific health care system or practice.”6 Clinicians or health systems need not only to understand how to implement PCOR findings but also whether the approach will work in their settings given their patient populations and other characteristics.7 Assessing PCOR findings for feasibility of wide-scale implementation is important because the growing volume of evidence-based interventions and PCOR findings has strained clinicians’ and health care organizations’ capacity for timely implementation. Furthermore, investing public resources for wide-scale implementation projects that are not feasible in real-world settings is wasteful and unsustainable.

To support prioritization and investment decisions, AHRQ sought criteria to look systematically at evidence related to factors that affect implementation of the intervention in specific clinical settings. While the implementation science literature has described many frameworks that map out factors that affect feasibility of implementation,1 none adequately met our needs to assess which evidence-based PCOR topics were ready for investment in wide implementation. In this paper, we present the process and methodology we used to develop AHRQ’s criteria and guiding questions to assess the feasibility of broadly implementing nominated PCOR findings.

METHODS

To develop criteria and guiding questions, we conducted a literature review and key informant interviews (KIIs) to identify factors affecting implementation. We grouped factors into themes, and a technical expert panel (TEP) prioritized themes for potential use as criteria. Criteria and guiding questions were revised based on pilot-test results. Neither institutional review board (IRB) nor Office of Management and Budget (OMB) review was required.

Literature Review

We conducted a literature review to identify factors for consideration as potential feasibility criteria. Our search included peer-reviewed and gray literature published between January 2010 and September 2016, including interventional studies, review articles, key models of implementability, frameworks, reference guides, books, and D&I toolkits. We used Boolean logic to combine relevant keywords and Medical Subject Headings (MeSH) (Appendix 1) to search PubMed and the Cochrane Database of Systematic Reviews. To identify additional sources, we also retrieved articles flagged by PubMed as similar or related to relevant study articles, searched gray literature on federal agency websites, reviewed reference lists of relevant articles, and consulted with research team members and key informants (KIs).

We reviewed abstracts of identified publications against pre-defined inclusion and exclusion criteria (Appendix 1). Independently, two reviewers used a template to abstract data (Appendix 1) from each full-text publication and used inductive qualitative analysis8 to identify factors affecting the feasibility of implementation. Reviewers then grouped together similar factors from various sources to identify and define emerging themes. Reviewers resolved differences through discussion, and a senior advisor served as a tiebreaker when they could not reach consensus.

Key Informant Interviews

In parallel with the literature review, we conducted and analyzed thirteen KIIs (Table 1) to gather diverse perspectives regarding factors affecting the feasibility of implementation and to identify additional sources for our literature review. We selected KIs based on their expertise in implementation science, direct experience implementing evidence-based interventions, and/or position within organizations representing a group perspective (patient/caregivers, providers, health systems) potentially affected by implementation efforts. Non-Federal KIs were offered a small honorarium to compensate for their time.

Table 1 Organizations Represented in KI Interviews and Technical Expert Panel
Full size table

We conducted 60–90-minute semi-structured telephone interviews with each KI. KIs discussed the role of patients, providers, and/or organizations during implementation; predictors of and motivators for successful implementation; barriers to implementation feasibility; and seminal implementation research to include in the literature review. Two team members reviewed interview transcripts to identify key factors affecting implementation feasibility, resolving differences by consensus. We then triangulated factors identified from the literature and the KIIs and grouped them into themes.

Technical Expert Panel

We convened a fourteen-member TEP for a full-day meeting to develop consensus on the most important previously identified themes for assessing feasibility. The TEP, including seven KI participants, were also selected based on their availability and for their D&I expertise from multiple perspectives, including federal and state government agencies; academic and other research institutions; and health care, professional, and consumer advocacy organizations (Table 1). We asked the TEP members, in full- and small-group discussions, to assess face-validity of each feasibility theme, recommend additional factors, and test the usability of the themes as criteria to assess a set of fictional case studies. We also asked TEP members to prioritize themes for feasibility criteria and for input on developing criteria based on the case study experience.

Criteria Development and Pilot Testing

Using the TEP’s prioritized themes, we created feasibility assessment criteria intended for use by a pair of AHRQ staff to independently review PCOR nominations, guide discussions, and inform recommendations for funding. We provided definitions and guiding questions to appropriately convey the criteria and their components to AHRQ staff end-users (Appendix 2). The complexity and associated components of each criterion dictated the number of guiding questions associated with each (Table 2; Appendix 3). In collaboration with AHRQ end-users, we edited criteria wording for clarity and improved layout, added a 3-point Likert scale scoring system to help reviewers compare assessments, and included an explanation of their purpose and specific instructions for use.

Table 2 Criteria Selected Based on Literature Review (LR), Key Informant Interviews (KIIs), and Technical Expert Panel (TEP) Results
Full size table

To determine the criteria’s usability for assessing feasibility of implementing PCOR findings, we asked seven PCOR and/or D&I experts to pilot-test them against four case studies. Case studies were based on summarized and de-identified nominated PCOR findings related to screening, prevention, and chronic disease management. To ensure the criteria and guiding questions would make sense to other clinicians and quality improvement professionals, experts included AHRQ staff who would use the criteria to assess nominations for the PCOR D&I initiative, a subset of available TEP members, and other experts not previously involved in the project. To minimize burden on the experts, we assigned each pilot tester two or three case studies. Four pilot testers evaluated each case study.

Pilot testers reviewed each assigned case study, evaluated it against the criteria using the guiding questions and 3-point Likert scale, and provided narrative justification for their assessment of each criterion to demonstrate their decision-making process. Pilot testers then filled out a brief follow-up questionnaire with narrative, written feedback regarding their experience using the criteria, including ease of use, length of time to complete the assessment, and any questions they had or challenges they encountered.

In collaboration with AHRQ end-users and based on pilot testing results, we edited the criteria wording for clarity and improved the layout to support both high-level thinking for global assessment, and detailed analyses focused on each criterion’s component parts.

RESULTS

Literature Review

We identified over 300 potentially relevant citations in our initial literature searches and included 41 publications, including reviews carried out for a similar purpose, gray literature, books, D&I frameworks and toolkits, and reference guides, in our analysis. Our analysis of implementation factors and thematic groupings yielded 13 common themes important for successful implementation of evidence-based interventions (see Table 2).

Key Informant Interviews

Based on their experience, perspectives, and expertise, each KI provided context to deepen and nuance our understanding of the factors that influence feasibility that we previously identified through the literature review. KIs highlighted the importance of the following:

  • Patient-level buy-in. Heterogeneity of target populations makes it difficult to achieve. Direct-to-consumer marketing and patient engagement in the development of the intervention and implementation strategy may improve the feasibility.

  • Provider-level buy-in and engagement. Successful implementation strategies minimize clinic interruptions or improve clinical workflow and do not impinge upon provider autonomy in exercising clinical judgment and good patient care. Whether the intervention actually addresses or improves a problem area for providers can improve implementation feasibility.

  • Engagement from multiple levels of the organization, including both formal and informal leaders. Organizational leaders look for a strong business case focusing on a positive return on investment. Organizational advocates for the intervention can champion implementation, but efforts highly dependent on a few specific individuals could be at risk from staff turnover.

  • The significant role that organizational capacity and motivation may play in the readiness to implement. This may be related to the type of institution and characteristics of the setting such as size, target population, and resources.

  • Taking into account external forces, such as shifts in payment models, reimbursement mechanisms, and the broader political climate, that may affect the organization and the feasibility of implementation.

The KIs did not identify any additional factors to those compiled from the literature review but provided input on prioritizing themes for inclusion either as criteria or guiding questions (Table 2; Appendix 3).

Technical Expert Panel

When discussing the initial set of feasibility criteria presented during the meeting, the panel emphasized the need to evaluate interventions at national, local, and stakeholder levels and from various perspectives to thoroughly and accurately determine feasibility. At the national level, national policies and priorities, the political environment, and external factors such as complementary or completing funding efforts should be considered. At the organization- and implementer-level, the TEP noted factors including demonstrated benefits and outcomes of the intervention, upfront costs or investments required (e.g., training, additional staff), burden of implementation, organizational reputation, existing and evolving reimbursement structure, and applicability and/or acceptability to the specific setting and patient population served by the intervention. The panel noted that feasibility of the intervention itself and the readiness of a specific organization to implement may be intertwined notions and difficult to separate during the feasibility assessment.

The TEP prioritized nine of the 13 candidate themes identified in the literature review and KI interviews for inclusion as feasibility assessment criteria (Table 2; Appendix 3). The TEP identified acceptability to the implementers, generalizability, and alignment of the intervention with external policies and incentives as the top three most important themes. They also considered acceptability to the organization, strength of evidence on implementability, and thoroughness of plan for implementability and sustainability to be high priorities.

The TEP excluded themes that were either already applied during AHRQ’s assessments of strength of evidence and impact (appropriateness of the intervention to address the problem and the theoretical basis of the intervention) conducted prior to the feasibility assessment or that were more appropriate for use by individual health care organizations to assess their readiness to participate in an implementation effort. TEP members recommended that we provide descriptions, definitions, or prompts to assess specific components of each to clearly convey the most central aspects of the selected themes.

Although the TEP and the research team noted the importance of considering patient values and perspectives in the assessment of evidence-based interventions, the TEP agreed to exclude “Importance of outcomes to patients” from the prioritized set of feasibility criteria (Table 2; Appendix 3). This concept is captured at the beginning of PCOR D&I prioritization process by the definition of PCOR and by the assessment of impact prior to feasibility assessment. In addition, the TEP noted that patients could be implementers and captured in the criteria, “Acceptability to the implementers/organization and fit with organizational capabilities.”

Criteria Development

We retained themes unanimously prioritized by the TEP and combined them into five criteria. Three “essential” criteria address elements of PCOR findings that are necessary to indicate whether or not implementation is feasible. Two “supplemental” criteria that are rarely present but would greatly facilitate implementation serve as additional indicators that an intervention is likely to be feasible to implement.

Essential Criterion 1 combines acceptability to the implementers and acceptability to the organization since these concepts overlap. Essential Criterion 2 addresses generalizability, adaptability, and ease of achieving fidelity to core components of an intervention if nationally scaled or broadly implemented in settings other than the one for which it was designed or demonstrated effectiveness. Essential Criterion 3 addresses alignment between external policies and incentives (e.g., regulations, mandates, payment structures, collaboratives, benchmark report, competition) and the purpose of the intervention. Alignment or misalignment can affect momentum among implementers, the organization, and other stakeholders. Timing in relation to the political and national climate associated with an intervention specifically or with a broader related effort could also impact its feasibility.

The presence of a detailed, delineated plan or evidence supporting a specific implementation strategy (supplemental criteria) allows an organization to implement an intervention without the need for developing a de novo implementation strategy that may differ substantially from the strategy used in the research supporting the intervention. Evidentiary support for implementation strategies can help AHRQ determine which strategies are best suited to implement an intervention in particular settings or populations, and which interventions can likely be successfully implemented using multiple implementation strategies.

Pilot Testing

Pilot testers found the criteria and guiding questions easy to use when relevant evidence was provided or readily available to assess the case study. For example, one case study clearly described a discrete implementation strategy to improve uptake of an intervention. Pilot testers were clearly able to identify implementers and implementation stakeholders, the components of the intervention and implementation strategy, and the intervention’s complexity, and have a sense of potential burdens (e.g., staff time, costs) for implementing organizations.

Conversely, pilot testers found the criteria and questions difficult to answer when relevant evidence was lacking about implementation costs, workforce capacity requirements, and acceptability of the intervention or the implementation strategy to stakeholders. In these instances, pilot testers requested additional guidance and estimated the time and/or resources it would take to gather the additional information needed to complete the questions and assess the feasibility of implementation. Pilot testers also noted a lack of alignment between the information provided in the nomination itself and information required to assess feasibility.

Pilot testers’ feedback regarding intent of each guiding question helped us improve the wording of guiding questions and led us to develop a companion guide with information about the background, meaning, and intent of each question in the assessment form and instructions to indicate the source of information used to answer the question.

Table 3 shows the complete final criteria with the guiding questions and associated Likert scale responses for assessing nominated PCOR findings. While we initially intended to develop an informal scoring mechanism to rank nominations, we learned that there was great variation in scoring among individual reviewers’ scores. Ultimately, we found the qualitative responses to be most informative in the deliberative process. Scores provided complementary information to signal which nominations may be particularly meritorious or where there might be disagreements and need for further discussion among reviewers.

Table 3 Complete Feasibility of Implementation Assessment Criteria
Full size table

DISCUSSION

AHRQ’s experience with the Effective Health Care Program and the Evidence-based Practice Centers informed our public nomination process to identify and assess PCOR findings for strength of evidence and impact. We recognized the need for an additional step to assess implementation feasibility to help us understand and account for the implementation environment and stakeholders for each specific PCOR finding. Described here, AHRQ’s three essential and two supplemental criteria along with guiding questions (Table 3) can be viewed as a framework to systematically guide a user’s thinking about the requirements for implementing a specific intervention in a specific setting and the intervention’s appropriateness for wide implementation efforts.

Despite the quantitative scoring and narrative comments, the criteria and questions are not designed to be used as a formal benchmarking or measurement scale. Since assessments are based on the reviewers’ judgment, the criteria and guiding questions work to consistently structure assessments, help reviewers articulate the rationale for final assessments, and promote transparency in recommendations for funding. The criteria and guiding questions help inform implementation concept development and funding decisions and identify issues or gaps in information that could be addressed through stakeholder engagement or further implementation research. Through its funded initiatives, AHRQ aims to integrate quality improvement strategies and implementation science to further expand the evidence base for implementing evidence to improve quality and outcomes of care.

Criteria in Use

AHRQ’s PCOR D&I Initiative has been using its prioritization process and feasibility criteria for approximately three years.5 Two current initiatives, TAKEheart to scale and spread increased uptake of cardiac rehabilitation (CR),9 and a grant initiative to help primary care practices better identify and manage unhealthy alcohol use (UAU), provide examples of the utility of developed criteria and questions in informing funding decisions.10

CR meets most of the essential and supplemental feasibility criteria. Strong evidence exists for both the CR intervention (reduced mortality and recurrent cardiac events) and the implementation strategy to maximize referral, enrollment, and participation (automatic referral and liaison).11 CR after an acute event is widely underused, and sizable disparities exist as to who receives it. The Centers for Disease Control and Prevention (CDC), a partner in the initiative, had developed a change package to support implementation by hospitals, health systems, and clinicians.12 AHRQ has partnered with the American Hospital Association to scale and spread the increased uptake of CR nationally. Evaluation of the TAKEheart initiative will provide additional evidence about what strategies work to scale and spread potentially high-impact PCOR findings to inform future initiatives.

Screening, brief intervention, and referral to treatment (SBIRT) and medication-assisted therapy (MAT) have strong evidence in primary care to address UAU, a major public health problem. Routine adoption of SBIRT to identify UAU for adults aged 18 and older in primary care is a B recommendation of the United States Preventive Services Task Force.13 AHRQ’s Evidence-Based Practice Center systematic review on pharmacotherapy for adults with alcohol use disorders in outpatient settings provides evidence for medication management of alcohol use disorders.14 Although the evidence for successful implementation demonstrating it can be feasible is just emerging,15 addressing UAU in primary care was selected for wider implementation, in part, because there are related tools and evidence for implementing the integration of behavioral health and primary care that can support implementation of SBIRT and MAT.16 Six funded UAU projects will test different approaches to implementing the evidence, further developing the evidence base for effective implementation strategies and, if successful, providing evidence for further scale and spread.

For a variety of reasons, topics were deemed unfeasible and not recommended for AHRQ implementation investment. One potentially feasible but complex topic was considered further with a group of stakeholders, but multiple acceptance and implementation barriers as well as rapidly changing clinical treatments and implementation strategies made it difficult to discern a clear path for implementation. Next feasible steps for implementation of several topics duplicated efforts of other funding agencies. For other topics, recent policy or guideline changes were likely changing practice but national utilization data was not current enough to determine whether a significant practice gap still existed. For another topic, recent policy changes initiated a number of national efforts that were changing the implementation landscape, and no clear implementation strategy had yet surfaced. For one topic, although clear implementation strategies existed, adequate service providers were unavailable to accommodate an increase in demand that would result from implementation efforts.

Potential Use in Health Systems and Practices

Although our process was specifically designed to inform AHRQ’s decision-making, other funders, health systems, and practices may be able to use the criteria and guiding questions to systematically consider important aspects of implementation in their own settings: who would implement and how implementation efforts would affect them, and how acceptable those effects would be to them. Heath systems and practices could use the guiding questions to consider evidence behind components of the intervention and implementation strategy, and availability of implementation plans or toolkits, to determine if they are generalizable or adaptable to the realities of their own settings or to what degree trial and error may be required to implement. They could consider alignment with payment policies or incentive structures versus financial costs of implementation resources to determine the business case to implement. Health systems and practices may even benefit from more immediate and local utilization, cost, capacity, and acceptability data not readily available at a national level.

Implications for Future Research

The PCOR D&I prioritization approach, and specifically the feasibility criteria, highlights gaps in understanding about how to best implement PCOR findings and points to areas requiring more research. For example, reviewers have found a need for more information and research related to costs, workforce capacity and training requirements, and/or acceptability of the intervention or implementation strategy to implementers at various health system levels. Evidence for implementation will always lag behind new evidence for clinical interventions. Understanding which implementation strategies can be effective for different types of evidence and clinical settings in the future may help accelerate implementation of new evidence as it emerges. Gaps may also be filled by a rigorous approach to learning while implementing, including the use of hybrid trials and other developing methodologies. This is the approach being used in the UAU initiative.

Improved reporting of quality improvement efforts and implementation research findings may allow us to assess more accurately the role of the intervention elements versus the implementation strategies in determining feasibility of widespread implementation. Journal editors can contribute to the availability of needed evidence for feasibility assessment by requiring articles to adhere to implementation science reporting guidance frameworks such as SQUIRE 2.0,17 CFIR,18 RE-AIM,19 Star-I,20 MUSIQ,,21 and the CONSORT 2010 statement: extension to randomised pilot and feasibility trials.22

LIMITATIONS

We developed the criteria and guiding questions based on the literature review at the time of this project with input from a limited number of key informants, technical experts, and pilot testers. We recognize that the number of individuals involved is insufficient to represent all stakeholders and to fully validate the criteria, and results may be affected by selection bias since experts were chosen based on availability. However, the process provided valuable external guidance for AHRQ’s work. The criteria and questions may continue to evolve through user feedback and ongoing research as the implementation science field matures. Others may want to build on this work.

CONCLUSIONS

To the best of our knowledge, no prior effort has produced a set of feasibility assessment criteria to help prioritize investments for implementing evidence-based interventions in clinical practice. As evidenced in the numerous D&I frameworks currently available, implementation of research findings is complex and challenging. The current proposed criteria and guiding questions help AHRQ systematically consider factors affecting implementation and identify gaps in understanding how to implement a specific intervention in a specific type of setting to determine whether a nominated PCOR finding is suitable for widespread implementation investment with our limited resources.