BACKGROUND

Science impact assessment is crucial for understanding the benefits and value of research investments and dissemination activities toward improving public health and reducing research waste.1 A key question is whether research discoveries result in evidence-based practices (EBPs) and implementation or quality improvement efforts that lead to improvements in population health. The Foundations for Evidence-based Policymaking Act of 2018—which requires federal Cabinet-level agencies, including the US Department of Health and Human Services and the Department of Veterans Affairs (VA), to use scientifically generated evidence to justify budget investments—has accelerated the need to determine whether research evidence is making a substantial impact on policies and programs.2 However, impact assessment remains a challenge and is complicated by the need to identify and systematically track measures that accurately capture impact, link upstream research with changes in distal health outcomes, and attribute specific organizational efforts given the overall body of evidence that led to the development of the innovation.3

Located in the VA Office of Research & Development (ORD), the mission of the Quality Enhancement Research Initiative (QUERI) is to improve the quality and delivery of care for US Military Veterans through translating research findings into practice. QUERI’s strategic methodology is threefold: (1) implement evidence-based practices (EBPs) and promising innovations into routine care; (2) evaluate programs and policies to optimize care; (3) promote the use of implementation and quality improvement science methods across the VA. Since its inception in 1998, QUERI has funded more than 500 peer-reviewed systematic investigations focused on transforming Veteran care using scientifically supported methods and has grown to support a national network of 200+ investigators. To maximize scientific impact, QUERI promotes research-operations partnerships between investigators distributed at VA medical centers across the US and national, regional, and local VA operations leaders. VA operations partners are senior financial, strategy, and/or clinical leaders who work in a National Program Office (e.g., Executive Director, Office of Mental Health and Suicide Prevention), a regional health system (e.g., Chief Medical Officer of a Veterans Integrated Service Network), or a medical center (e.g., Medical Center Director) and can provide guidance and support to ensure QUERI project goals are aligned with health system needs and improve uptake and feasibility of the EBP or promising innovation (hereafter referred to collectively as EBP). These multi-level partnerships are essential for obtaining buy-in and tailoring EBPs to meet regional and local needs while ensuring alignment with national priorities and standards.

This paper describes the QUERI Impact Framework and its implementation in the VA. Developed in response to an increased demand among VA leadership to understand the real-world impacts of VA-funded programs, the QUERI Impact Framework strives to provide a pragmatic approach for measuring the overall health, economic, policy, and cultural impacts of implementation and quality improvement sciences.

METHODS

Methods and data ascertainment for analyses were considered non-research and did not require institutional review board approval per ORD Program Guide 1200.21.

Ascertainment of Evidence Supporting the QUERI Impact Framework

Established in 1998 as an intramural funding program to translate research evidence into clinical practice, QUERI initially focused on disseminating EBPs that had evidence demonstrating improved outcomes for diseases prevalent among Veterans (e.g., heart disease, spinal cord injury), and assessments of project progress and impacts typically included project completion and publications.4 In response to changes in VA priorities and an external evaluation of QUERI in 2014, QUERI began partnering with national VA leadership to address cross-cutting clinical priorities (e.g., care coordination, patient safety) through the application of implementation strategies, which support providers in using an innovation.5, 6 The 2016–2020 QUERI Strategic Plan resulted in the establishment of 15 QUERI Programs to support implementation efforts addressing clinical variation across the VA.7 As a result, QUERI Impact Framework metrics were created through an iterative process of development, deployment, and refinement (Table 1). With the shift from disease-focused investigator-initiated projects to interdisciplinary problem-focused centers where goals are driven by the health system, QUERI began to benchmark progress using key measures derived from the National Academy of Medicine Degrees of Impact Framework.7, 8 The QUERI Impact Framework expands on QUERI Program impact domains and incorporates lessons learned from QUERI initiatives, VA transformational efforts since 2015, and 2021–2025 QUERI strategic planning.7

Table 1 Multiple Levels of Influence Leading to the Development of the QUERI Impact Framework
Full size table

The 2021–2025 QUERI strategic planning process—focused on updating the 2016–2020 QUERI Strategic Plan—provided more depth to the QUERI Impact Framework by incorporating perspectives from QUERI investigators and VA leadership. In November 2018, VA operations leaders were sent an online survey to capture their priorities, involvement with QUERI-funded initiatives, and perceptions of QUERI, including QUERI’s responsiveness to VA priorities and its impact on Veteran care. Through purposeful sampling to ensure diverse geographic representation, 2–3 leaders from each regional health system, Veterans Integrated Service Network (VISN), were invited to participate in semi-structured phone interviews. In March 2019, an online survey was sent out to QUERI-funded teams, and similarly a sample of QUERI investigators were identified to ensure geographic and functional diversity and invited to participate in semi-structured telephone interviews. QUERI investigators were asked about successes and challenges related to their QUERI-funded activities, satisfaction with QUERI, and QUERI’s overall strengths and opportunities for improvement. The results of strategic planning refined the QUERI Impact Framework, promoting leaner domains and metrics and underscoring the importance of understanding the impact of non-research activities (e.g., developing and maintaining multi-level partnerships, ensuring sustainability of an EBP), measuring the impact of partnerships beyond monetary investments, assessing the geographic reach and scope of QUERI’s implementation efforts across VISNs, benchmarking impact using quality of care metrics, and understanding the perceptions of QUERI across stakeholders at the local, regional, and national levels of the health system.

QUERI Impact Framework

The QUERI Impact Framework emphasizes alignment with research and operations priorities and formation of bi-directional multi-level partnerships, which are crucial for changing clinical practice and healthcare policy and, ultimately, improving patient care and health outcomes. Table 2 describes potential indicators across five degrees of impact (Alignment, Commitment, Tailoring, Informing the field, Observing healthcare changes and generating New questions/projects or ACTION).

Table 2 QUERI Impact Framework
Full size table

Alignment with Multi-Level Priorities

Alignment with research and operations partners and national priorities ensures practical strategies and tools are developed to achieve health system performance objectives while maximizing scientific impact. This participatory emphasis is captured in the Alignment Domain focused on priorities identified by stakeholders at the local, regional, and national levels. Examples of national VA priorities include reduction of unwarranted variation across integrated clinical and operational service lines and improving access to services that support and encourage lifelong health and wellness. In the context of these priorities, VISN leadership has nominated actionable priorities, such as improving access to medication-assisted treatment for opioid use disorder, for QUERI implementation support.9

Commitment

The Commitment Domain examines the productivity of collaborations in contributing to the success of the implementation effort, emphasizing joint investment and risk-sharing. In providing tangible resources, researchers and operations leaders form bi-directional strategic partnerships grounded in a shared vision and goals, a collaborative governance structure, and flexible incentives. Operations partners show commitment through contributing funding, personnel, data, and other resources to support the implementation effort, and researchers bring their expertise in developing scientifically supported tools and strategies. Examples of resources that operations leaders have provided for QUERI initiatives include funding for equipment, travel support for site visits, and protected time for non-research personnel (e.g., pharmacists). Dashboards, clinical decision support tools, training manuals, and toolkits are some examples of products developed by QUERI investigators to support the scale up and spread of EBPs.

Tailoring to Local Context

The Tailoring Domain focuses on proximate impacts linked to health system performance plan goals, promoting a shared understanding across stakeholders for benchmarking success toward achieving goals. QUERI tracks metrics derived from the VA Medical Center Director and VISN Director Performance Plan, including the number of facilities where the EBP has been implemented, the number of VA staff trained in the EBP, and the number of Veterans served by the EBP.

Like other US health systems that span multiple geographic regions, VA faces challenges with addressing clinical variation across facilities and health disparities among specific Veteran groups.10 Because QUERI investigators are experts in using implementation strategies to enhance the uptake of EBPs for facilities with complex needs, QUERI implementation teams are encouraged to provide tailored support for specific subpopulations (e.g., Veterans residing in rural areas) and facilities that are experiencing quality gaps. These more intensive efforts, including the innovative strategies used in working with facilities experiencing quality gaps, are tracked separately.

Informing the Field

Like the National Academy of Medicine (formerly Institute of Medicine) Degrees of Impact Framework and AAMC/RAND metrics, the QUERI Impact Framework measures dissemination to stakeholders with direct decision-making and financial authority and publications and presentations at academic conferences.8, 11 Capturing the number of published journal articles is crucial because it ensures investigators, who are located at institutions that often tie academic promotion to the number of high impact journal publications and conference presentations produced from research, are encouraged to lead QUERI-funded initiatives. Additionally, the QUERI Impact Framework assesses the effectiveness of these dissemination efforts by measuring the scale up and spread of the EBP beyond the original implementation facilities.

Observing Healthcare Changes and Generating New Projects

The fifth domain, focused on long-term impacts, involves monitoring the sustainability of the EBP and long-term changes in quality of care, policy, and culture across the health system. Based on the results of the 2021–2025 strategic planning where VISN and facility leadership emphasized the importance of measuring impact using quality of care metrics, QUERI Programs were encouraged to identify relevant VA performance metrics, provider experience data from the VA All Employee Survey, and other publicly reported quality metrics identified by the National Quality Forum and the Centers for Medicare and Medicaid Services (CMS).12, 13 Understanding whether an expansion or development of new projects emanated from the original initiative can help signal a cultural shift in health system leadership’s and other key stakeholders’ perceptions of the value of implementation and quality improvement methods in improving care. Furthermore, the 2021–2025 QUERI strategic planning highlighted the importance of mentoring the next generation of QUERI investigators and capturing their impact by measuring the volume and quality of products developed by early career investigators to improve patient care through the application of implementation and quality improvement methods. These long-term impacts will be systematically captured starting with the 2021–2025 QUERI Programs, which are undergoing peer review in 2020.

Application of the QUERI Impact Framework

Iterative Process for Collecting Impacts

In 2018, QUERI began deploying an iterative process of collecting impacts across the continuum of funding, including at the point of application. The inclusion of anticipated impacts as an application requirement encouraged QUERI investigators and key stakeholders to consider the potential impact of their work on improving care for Veterans and, with a Veteran-centered perspective, together identify resources and metrics to optimize impact. The biannual collection of impacts throughout the funding award period allows for ongoing evaluation and reflection so adjustments can be made to share learnings and enhance impact.

Applying the Framework to the QUERI Programs

Table 3 describes QUERI’s cross-cutting metrics, which reflect QUERI’s strategic methodology focused on implementation, evaluation, and dissemination. Funded on 5-year cycles and led by VA investigators, QUERI Program centers support 3–7 implementation- and/or evaluation-focused initiatives that reflect a cross-disciplinary impact goal aligned with VA priorities. Since the QUERI Programs have similar structures, start dates, and implementation strategy goals, the following impact analysis reviews the in-progress impacts of QUERI Program initiatives that are utilizing at least one discrete implementation strategy to support implementation of an EBP.16 Data were abstracted from Fiscal Year 2019 reports, which include cumulative impacts from the start of the QUERI Program through September 30, 2019.

Table 3 QUERI Program Center Impacts
Full size table

RESULTS

The 15 QUERI Programs have or are currently supporting implementation of 49 EBPs. Table 3 summarizes aggregate QUERI Program impacts.

Alignment with Multi-Level Priorities

All QUERI Programs align with ORD’s priority of translating more research into real-world settings with 76% of initiatives (n = 37) reporting that the evidence behind the intervention being delivered and/or implementation strategy being deployed stemmed from VA-funded peer-reviewed awards, published journal articles, and/or systematic reviews. QUERI Programs were aligned with at least one VA priority, reporting a total of 62 national partnerships representing 37 unique organizational and/or patient care service areas (e.g., mental health, pharmacy benefits) and 9 regional partnerships (e.g., VISN Chief Medical Officer) with 7 different VISNs. Seven QUERI Programs leveraged multi-level partnerships with at least one national and one regional partner. The nature of operations partner involvement ranged from consultation (e.g., providing input on implementation efforts) to deeper collaborations (e.g., co-authoring manuscripts).

Commitment

Nine (18%) initiatives reported a total of $5,420,074 in monetary support from operations partners. Nearly all QUERI initiatives reported in-kind operations support, such as dedicated staff time, data acquisition and programming, access to provider networks, and facility recruitment support.

Tailoring to Local Context

QUERI investigators supported implementation efforts across a variety of settings (e.g., outpatient clinic, community living center) touching all VISNs and 15 sites experiencing quality gaps. QUERI Programs trained 5147 health professionals, including physicians, nurses, pharmacists, psychologists, care managers, peer specialists, technicians, coordinators, and directors.

Informing the Field

Knowledge generated through the implementation effort was disseminated to operations leaders through regularly scheduled meetings, research stakeholders through 271 journal articles, and external organizations and other members of the public through 320 presentations at non-VA conferences/meetings (e.g., Academy Health research meetings).

Moreover, 15 (31%) QUERI initiatives submitted their EBP to the VA’s Diffusion of Excellence Shark Tank or Diffusion Marketplace, which are VA programs that support the scale up and spread of innovations submitted by VA employees and selected by subject matter experts and VA leadership.

Observing Healthcare Changes and Generating New Projects

Among the 9 QUERI Programs conducting an economic analysis, most are determining the cost of the EBP and/or estimating the cost of the specific implementation strategy used. The 30 projects/analyses launched in response to operations questions ranged in size and effort, including brief consultation, workgroup participation, systematic review, analysis and policy recommendations, program evaluation, metric development, and toolkit creation and dissemination. Although it was early for most QUERI Programs to demonstrate changes in Veteran health outcomes, some demonstrated clinical improvements, such as increases in disease screening and appropriate diagnosis referrals, positive changes in clinician workflow, increased provider satisfaction, improved Veteran satisfaction, and cost savings from avoided hospitalizations. QUERI Programs shaped policy and/or operations activities in a variety of ways, including refining a tool or program to optimize national rollout and informing decision-making for national or regional adoption of a program or mandate. For example, two EBPs—an inpatient walking program to improve mobility and a data-driven approach to enhance the quality of care for Veterans with transient ischemic attack (TIA) and minor stroke—were selected by VISN Directors for scale up and spread across the region.

DISCUSSION

To our knowledge, there is no comprehensive impact framework for measuring the multi-faceted contributions of implementation and quality improvement sciences in real-world settings. Impact assessment is crucial to ensure positive benefit across different levels of a health system. With its participatory emphasis and focus on capturing health, policy, economic, and cultural impacts using a combination of quantitative and qualitative metrics, the QUERI Impact Framework strives to provide a pragmatic approach for measuring the multi-level contributions of implementation and quality improvement sciences. In capturing all stages of implementation—pre-implementation, implementation, and sustainment—the QUERI Impact Framework seeks to enhance the long-term impact of research by ensuring sustainment of the EBP after initial implementation support and funding end.9

The QUERI Impact Framework builds upon previous impact frameworks, such as the Centers for Disease Control and Prevention’s Science Impact Framework, and assesses impact on quality, cost, and patient and provider experiences across all implementation activities, not solely research efforts.17 A unique feature of the QUERI Impact Framework is its central focus on alignment of stakeholders around multi-level priorities and metrics as health systems continue moving toward data-driven consumer-centered care. The ACTION Framework includes research dissemination impacts but places a greater emphasis on multi-level partnerships and broader societal impacts, which align with QUERI’s mission to improve Veteran health by utilizing implementation strategies to accelerate improvements in quality of care. Developing and maintaining multi-level partnerships are crucial non-research activities that require significant effort and are essential for successful implementation. Maintaining productive research-operations partnerships, including the importance of forming a coalition to insulate against organizational changes (e.g., leadership and provider turnover), embracing flexibility to meet operations partner needs and timelines (e.g., sharing interim data), and making connections to operations partner priorities and health system performance measures, is essential to ensure buy-in and maintain strong partnerships. Information collected on impacts is used to engage health system leadership, respond to Congressional requests for information, inform funding requests for applications, develop resources that support QUERI investigators, and evaluate opportunities for collaborations and outreach. Describing recent impacts has been important for continued involvement of health system leadership, whose timeline is often shorter than a typical research project’s. The continuous focus on impact from the point of application through the funding period also allows for rapid learning and translation.

Like other funders, QUERI grapples with identifying common metrics for implementation and quality improvement initiatives, which encompass a range of clinical areas, and in linking distal outcomes to implementation efforts given the dynamic interconnected nature of a health system and the extensive body of evidence that leads to the development of an EBP and implementation strategy. In response to QUERI investigator feedback gathered during the strategic planning process, the range of QUERI impact metrics was narrowed to 10 categories and focused on measuring the contribution of QUERI initiatives, not attribution.

Systematically capturing the impacts of scientific investments is crucial for ensuring funds are directed toward activities that yield the maximum impact based on organizational priorities. The QUERI Impact Framework proposes an approach to understand the impact of implementation and quality improvement sciences in improving quality and optimizing value in a resource-constrained health system. The Framework can be applied in other health systems and research organizations to align multiple diverse stakeholders around cross-cutting goals and metrics.