The U.S. health care system increasingly embraces the measurement of value for both reimbursement and quality improvement, but a universal definition and research agenda to improve value measurement remains elusive. We convened an expert panel on value for a Department of Veterans Affairs (VA) State of the Art (SOTA) conference on performance measurement. This article highlights findings from the value committee and its recommendations for future research. The committee agreed that value is defined as the incremental outcomes gained per dollar spent, highlighted ways that the term "value" is frequently misused, and identified and prioritized seven themes for future research. Research on clinician and patient incentives to promote value was recognized as a high priority, despite regulatory constraints that often place practical limits on the size and duration of incentives within the VA. Two other critical themes involved the measurement and presentation of value metrics, which we call data-driven management. The remaining themes were as follows: reconciling divergent value perspectives, methods for managing value data in real time, the relationship of value to cost-effectiveness analysis, and minimizing unintended adverse consequences of value measurement. Finally, the committee discussed how organizations such as the VA can use coordination and competition to reinforce the delivery of high-value care.

Waste and inefficiencies within the US health care system are so profound that an estimated 30 % of costs could be cut with few adverse effects on patient outcomes.1 To better contain health care spending, many have called for providing more efficient care and finding ways to discourage low-value care.2

The challenge is to identify health services that have no benefit for specific patient populations or, more likely, that have benefits that do not justify their costs, and then to orient the delivery system to emphasize high-value products and services. But what is value? Value has been defined numerous ways, creating confusion among researchers and policymakers.3 Often, low-value care has been defined as health care services or products that provide little benefit to patients, as exemplified by theChoosing Wisely campaign, in which specialty groups highlight low value procedures.4 , 5

In 2014, the VA Health Services Research and Development Service sponsored a national State of the Art (SOTA) conference with four separate committees to address the issue of performance metrics. Our committee was tasked with defining value and prioritizing research that would improve the measurement and delivery of high-value care. We used a structured brainstorming format whereby we synthesized the literature, shared ideas, and integrated these ideas into a research agenda. Through moderated meetings, the committees were charged with formulating recommendations for future funding priorities. All committee members are co-authors of this article, which describes our methods, findings, and conclusions.

METHODS

The committee was composed of 11 nationally recognized health services researchers with expertise in economics and medicine, and familiarity with the VA. Committee members were nominated by the committee chairs (Drs. Wagner and Asch) and approved by the SOTA co-chairs, with input from VA health services research leadership. The committee met on three occasions using Think Tank, a web-based platform designed for intensive remote interactive collaboration.6 Think Tank allows simultaneous audio and text exchange and is structured to share and focus discussions. One moderator facilitated the 90-min sessions and assisted the chairs in managing the discussion and reviewing reports for completeness and accuracy, while another moderator managed the platform and helped create reports. Participants were encouraged to type comments into the Think Tank application, but were also free to raise issues verbally. Think Tank displayed the written comments anonymously, although participants often self-identified their contributions. This process, which followed a nominal group technique for achieving ranked priorities,7 allowed for open discussion without forcing consensus, the creation of themes through rapid synthesis, and real-time ranking of themes.

The Think Tank sessions represent a departure from prior SOTA meetings, which were conducted in person. Although face-to-face meetings may have advantages over virtual meetings in terms of limiting distractions, they are prone to verbal traffic jams and the desire to maintain agreement (i.e., groupthink).8 Think Tank’s use of typed comments prevented many verbal traffic jams, and the technology interface, including anonymous posts, enabled participants to disagree and provide concrete explanations in real time.

During the first committee meeting, on February 7, 2014, we discussed the definition of value, based in part on literature that was distributed prior to the first meeting.9 – 14 In the second meeting, four days later, the committee discussed how value changes depending on who bears the costs and who reaps the benefits (e.g., patient, provider, society), and how value measurement relates to cost-effectiveness analysis (CEA). The discussion then shifted to how health care systems learn in real time and how managers can feed value information back to providers and patients. This led to a discussion of provider performance and the role of incentives. We closed the second session with a discussion about the unintended adverse consequences of measuring value.

In the third session, a week later, the committee revisited the issue of incentives for promoting value, the types of incentives (financial and non-financial), and incentive valence (carrots or sticks). We also briefly discussed how large organizations can function to make achieving higher value easier. Much of the third session was used to synthesize the issues raised during prior meetings and to create themes for future research. The third session closed with a blind priority ranking of the themes identified during the preceding discussion.

RESULTS

What Is Value and What Is It Not?

In economics, value to a consumer is defined as the difference between the benefits received and the price paid. Consumers measure the value of a purchase when considering an alternative use of the money, whether that is an investment or an alternative purchase. This economic definition of value does not easily translate to health care, because consumers have little information on benefits or costs. Even when health care costs are known, this definition requires that we measure benefits in dollars; alternative approaches to assigning monetary values to health outcomes exist,15 but using clinical outcomes is frequently preferred.16 Consequently, we agreed to define value as the incremental gain in outcome per incremental cost, as discussed by Porter.12 If outcomes are measured in quality-adjusted life years (QALY), this definition follows the well-known incremental cost-effectiveness ratio (ICER) from cost-effectiveness analysis,17 which the American College of Physicians has endorsed as the preferred method for assessing value.11 Other organizations have shied away from using the ICER as the primary definition of value. For example, the Patient-Centered Outcomes Research Institute has focused more on patient preferences and de-emphasized costs. The Institute of Medicine and National Quality Forum also disassociates value from the ICER by incorporating patient or other stakeholder preference weights into the numerator, which may or may not duplicate quality adjustment of life years gained.18

Another problem with relying solely on QALYs for the value numerator is their relative insensitivity to changes that patients deem important, in part because researchers often rely on indirect measures of health-related quality of life. In a recent multi-site surgical trial, patients in both treatment arms experienced large and significant improvements in angina symptoms after surgery, while the Health Utilities Index—a means of assessing utility or quality of life—showed no changes in quality of life from baseline to follow-up.19

Consequently, the committee agreed that the definition of value should not rely on QALYs as the sole outcome but should also consider other outcomes that are more relevant to patients or institutions. Validated process measures or patient-reported factors would also be acceptable if strongly related to outcomes based on the underlying evidence, although it remains unclear when surrogate outcomes are appropriate and should trump health outcomes.

Dividing total outcomes by total costs for a population is not a valid measure of value, because this ratio lacks a comparator. Finally, value of care should not be confused with a patient’s ethical or moral values. Although one’s values can affect his or her preferences and decision-making processes, assessing value requires a relative comparison of benefits and costs of health care services.

Priorities for Future Research

During the committee’s three sessions, different themes emerged for future research on value. We identified and prioritized seven themes (Table 1), then grouped them into the following categories.

Table 1 Committee Ranking of Themes for Further Research
Full size table

Can We Create Incentives to Provide High-Value Care?

Emerging as the top priority was a theme focused on incentives for high-value care. Five of the eight voting committee members ranked it as 5 (most important), while no one ranked it less than 3 in importance (see Table 1). Creating incentives, when done correctly, rewards clinicians (or consumers) for behaviors that are consistent with the goals of the organization or society. These incentives need not be exclusively monetary, as research in behavioral economics has shown. For example, making high-value generic medications the default and easiest choice in electronic ordering could incentivize physicians to choose them.20

Of particular concern is the use of performance metrics that may encourage clinicians to provide low-value care. Considerable attention in Congress and elsewhere has been devoted to understanding specialist productivity in the VA.21 However, productivity is not synonymous with value. Tracking clinician output is likely to induce increased output, but not necessarily increased value. Focusing on output per clinic visit could result in patients receiving low-value care (or worse) if the clinically appropriate care pathways (e.g., watchful waiting) are viewed as less productive.

There was enthusiasm for studies that would determine whether it is more effective to incentivize individuals, teams, or systems to improve health system performance. The role of teams was of specific interest to several committee members, given recent work in surgical care,22 patient safety,23 and primary care.24 Creating incentives for individual clinicians might produce suboptimal outcomes, especially when care coordination is important. While much of the discussion focused on provider incentives, we were eager to see more research on incentives that encourage healthful behaviors among patients.25

Data-Driven Management

The VA has a well-established electronic medical record, and utilizes these data to generate dashboards of quality metrics, such as hospital-acquired infections, risk-adjusted mortality, or efficiency.26 , 27 None of the dashboards provide explicit value metrics (i.e., measures that synthesize information about [incremental] quality/outcomes and costs). Metrics that identify value for specific types of care (e.g., high-value diabetes care) are needed. Metrics are also needed for understanding the value of care provided by a medical center, but measuring value at the macro level is particularly thorny. One approach could emerge from understanding why dual-eligible veterans choose to seek care in VA or non-VA facilities. A scenario in which veterans choose a higher proportion of services from outside systems reveals the patients’ preference and implicitly conveys information about the value of VA care compared to alternatives.28 , 29

For data-driven management to work, more research is needed on the informational infrastructure supporting value measurement. We discussed existing efforts, including those of the Comparative Effectiveness Public Advisory Council (http://cepac.icer-review.org/) and Deliberative Democracy Consortium (http://www.deliberative-democracy.net/). These platforms might be particularly useful ways to stimulate research and efforts aimed at guiding reimbursement decisions for expensive pharmaceutical and biologic medications.30

We recognized the need for adaptable analytical platforms, as this is necessary to build high-value, learning health care systems.10 This work could be led by informaticians and health system engineers, who could modify the electronic medical record to collect the necessary parameters to make value decisions. Embedding value parameters into clinical decision support could also steer clinicians away from offering low-value options (e.g., magnetic resonance imaging for patients with low back pain).31 – 33 Although the committee saw these issues as very important, it also recognized that some platforms (e.g., automated satisfaction surveys) may not offer enough information or flexibility, and may lead to more downstream problems.

Is One Perspective More Important than Another?

Patients may value a new treatment differently from providers, health care organizations, or society. When the perceived value of care differs by patient, provider, and societal perspective, miscommunication and competing incentives can result in suboptimal care. For example, recent studies have raised questions about the value of using second-generation antipsychotics and off-pump cardiac bypass surgery; these questionable treatments are frequently chosen by providers, but may result in more expensive care or worse outcomes.34 , 35 The Patient-Centered Outcomes Research Institute has worked hard to engage patients in research. Patient engagement may be one way of reconciling differing perspectives about value, and might also improve understanding of how value changes with disease severity or near the end of life. Research is needed to understand why patient, provider, and societal perspectives about the value of costly or prevalent services may differ.

Another area for research is the value of investing in programs that benefit the organization but have little discernible impact on clinical outcomes. Some investments, such as robotic-assisted surgery, might be deemed highly valuable to the organization for marketing purposes or physician recruitment but offer little value to patients.36 , 37 We note at this time that these issues highlight internal inconsistencies with Porter’s12 definition of value, which focuses on patient-targeted outcomes. Thus, it is not clear how to compare one program that provides value to an organization but not a patient, to another program that provides value to a patient but not the organization.

Avoiding Unintended Consequences

Finally, there was interest in preventing unintended adverse consequences of value measurement. New policies or programs based on value considerations could have unintended downstream effects. Participants unanimously viewed this as an important consideration, although it was ranked lowest among the themes raised in committee discussion.

Coupling new programs with strong research evaluations represents one way of preventing or minimizing unintended effects. For example, as new performance dashboards are produced, we can test display changes in random samples, as is done by Facebook, to optimize the user’s experience. Larger programs may need to employ staged rollout to facilitate implementation research. These efforts can help minimize unintended consequences.

Finding Value Opportunities in Large Organizations

The VHA is one of a growing number of large integrated health care organizations striving to achieve high-value care. Consolidation among health care organizations over the past 15 years reflects efforts toward achieving contractual efficiencies and economies of scale.38 , 39 Although larger organizations can create efficiencies through purchasing power and negotiated contracts, size can also create inefficiencies. Analyses among other industries, including Google, have noted the challenges of communicating new information across a large workforce.40

The committee discussed whether value in a large health care system like the Veterans Health Administration can best be realized through centralization or regionalization. Decision-makers in such organizations are often left with few levers. One option is to create regions, such as the VA’s integrated service networks (VISNs), in which managers are more sensitive to local context and can implement a wider variety of policies customized for local conditions. The committee lamented the lack of evaluation data to inform centralized or decentralized programs. There were, however, some exceptions. The VA substance use disorder treatment programs have benefitted when minimum requirements were established by centralized management, perhaps because local leaders do not always share national priorities or implement programs consistently among sites.41 More research is needed to understand the context in which centralized decisions succeed or fail.

The committee also considered whether value would be best fostered by collaboration or competition within the organization.42 For many conditions and illnesses, the possible care pathways among generalists and specialists are often not linked or coordinated. A patient with back pain might be routed to neurology, a pain service, physical medicine and rehabilitation, or surgery. In these situations, patients are often responsible for synthesizing their own potentially disparate or conflicting management plans or care recommendations.43 Care coordination is one possible solution, and a growing number of studies have shown mixed results depending on the patient population.44 Sharing best practices for achieving high-value care through online collaborations is another possible solution.45

Competition on value metrics could be helpful if agreed-upon value metrics can be established. The challenge is to establish true value metrics that foster competition on both outcomes and cost, rather than just on price. A possible venue for competition within large organizations like the VA is administrative functions that worsen the denominator of value through increased overhead costs, such as human resources, information technology, and contracting. For example, there is no requirement for all 150+ VA medical centers to staff their own human resources department, but all do, as a result of historical inertia. One strategy is to create a payment structure and to then encourage competition for provision of such services across facilities. Facilities that provide high-value human resources services may be more likely to flourish, whereas those that do not could be forced to improve or cease operation.

CONCLUSIONS

Value is a hot topic in health care, and yet the term has created much confusion.3 The committee agreed that value is a relative assessment of costs and outcomes between a treatment and a comparator. We endorsed the definition of outcome improvement per dollar spent, which explicitly focuses on the incremental change in outcomes relative to the incremental change in costs between alternatives.

When outcomes are measured in QALYs, value is synonymous with an incremental cost-effectiveness ratio, as calculated in CEA. However, value measurement should not focus solely on QALYs, which might not always be meaningful to patients (or consumers) and can be difficult to measure. Confusion may occur if a new innovation provides high value when measured with one outcome or perspective but no value when measured using a different outcome or perspective.46

The committee was particularly eager to see more research on incentives to improve value and to leverage past research47 while navigating the governmental limitations on human resources, contracting, and information technology. The committee also agreed that we must enlist the help of patients in driving policy agenda and clinical care towards defining and improving value. More research is also needed on the data structures for populating value performance indicators that are salient for patients and providers alike. Defining the areas in which inter- and intra-organizational collaboration and competition might best foster value is another fruitful area for future investigation.

The ranked research priorities were a culmination of three virtual brainstorming sessions that enabled experts from around the country to participate in an online discussion. Think Tank works to minimize groupthink and verbal traffic jams by encouraging experts to type ideas and responses concurrently into the web-hosted application. The rate-limiting step is the speed at which people read and type, and there is an organic flow of ideas that is easily manipulated into themes. Verbal discussions were possible, but often the phone line was invoked to ask for clarification. Although members reported high levels of satisfaction with this virtual real-time format,43 the results are limited by the participants and their willingness to share their ideas.

In summary, few now doubt that the US health care system wastes too many resources on care that is unlikely to improve public health. Improving the measurement and management of value creates opportunities to redistribute substantial resources within the health care system such that patients can live longer, higher-quality lives in accordance with their preferences. It may also slow the overall growth of health care costs and allow societal resource redistribution to other worthy sectors of the economy.