Background

The unsustainable growth of health expenditures observed in developed countries over the last decades has sparked a variety of initiatives worldwide envisioning adequate allocations of the scarce resources available. Economic evaluations of care interventions, experiments involving payment schemes, campaigns for appropriateness of care targeting clinicians, novel purchasing strategies, among so many others, there seem to exist initiatives targeting virtually every aspect of the health care system. In recent years, the term value has occupied a central spot in this constellation of efforts toward wiser spending of money.

Assessing the value (often referred to as the ‘value for money’) of health care technologies, in particular the novel ones, has received an enormous attention from researchers, policy-makers and the industry. In countries with single-payer systems, the economic evaluation of technologies acquired a paradigmatic status, although its practice has been often incomplete or inconsistent. In the United States, where co-exist a market-oriented system of private insurers/providers and public programs (distinct among themselves, such as Medicare, Medicaid and the VA system), the notion of value has been yet incorporated but it has gained more and more attention in the latest years, despite the unclear lieu of value-based strategies in general.

Notwithstanding the disseminated use of the term ‘value’, it has not always been employed with the same meaning. The most commonly accepted definition of value in health care is provided by Michael Porter as “health outcomes achieved per dollar spent” [1]. This description is conceptually situated within the realm of technical efficiency, concerning about the maximization of objective gains in health in relation to a given amount of financial resources. In defining value at the ISPOR Special Task Force Report, in turn, Garrison et al. point out that “from an economic perspective: the ‘gross value’ can be thought of what someone would be willing to pay for an economic good or intervention, whereas the ‘net value’ subtracts the opportunity cost incurred to obtain that gross value” [2], p124). In the latter definition, on contrary, value evocates a connotation of allocative efficiency.

Although Garrison et al. argue that the underpinning notion of value does not depend on the nature of the health care system (whether a market-based, a social insurance or a single-payer system), that might be true from a normative perspective, but it does not seem the case considering the diversity of ways the term value is employed and the distinct approaches it motivates. The corollary of this observation is that, apart from the unsurprising idea that the value of health care technologies is contextual for many reasons (the perspectives included in the assessment, the societal values, the relative price of labor and technology, etc.), the proper concept of value seems to be contextually sensitive.

The aim of the present study was to understand what strategies of value assessment have been developed and implemented worldwide. In addition, we sought to comprehend the contexts in which these initiatives emerged and what definition of value underpinned it. In order to address these research questions, we performed a rapid literature review and a gray literature search focusing on existing value assessment frameworks of health care technologies within the context of developed countries.

Methods

Two-stage screening scientific literature review

A comprehensive search of the peer-reviewed literature published between 2007 and 2017 was conducted using Ovid MEDLINE, an extensive database of public health journals with a powerful platform for building searching strategies. The specific search strategy, presented in Additional file 1, resulted in the identification of 1176 references. Following execution of this search, we used a two-stage screening process.

The first stage consisted of selecting articles for full-review and the grouping of similar approaches in preliminary heuristic categories. Titles and abstracts were screened with the following two inclusion criteria: 1) does the paper describe a way to assess value of a treatment, service, intervention or technology? And 2) does the paper describe an applied framework (i.e., was the value assessment approach actually implemented)? In addition, we excluded papers that: 1) deal with animal health; 2) describe means to assess the value of management systems, administrative procedures, payment schemes, data management solutions, training programs, human resources schemes, or solutions for health care supply chains and purchasing contracts; or 3) that are not full articles (i.e., editorial, interview, commentary).

All 1176 titles/abstracts were reviewed by one reviewer and 335/1176 (28%) were also reviewed by a second reviewer. The agreement rate between the two reviewers on the first screening was over 95%, with discrepancies settled by discussion. There were no cases where a third reviewer was required. In total, 157/ 1176 papers were initially screened ‘in’ for further review. In order to make sense of the data and help in the understanding of the big picture, each of these 157 references were placed into one of six initial descriptive categories: a) CEA/CUA; b) simple consideration of costs and outcomes; c) unspecified value approach; d) multiple criteria approach; e) net economic/social value; and f) comparative effectiveness.

The second stage of screening was meant to further refine and hone the initial six categories listed above. In this stage, we were working to saturation, so we sought to include papers that described empirical activity of approaches to value assessment while excluding papers there were simply ‘more of the same’. In addition, we sought to identify any particularly innovative or novel aspects in the implementation of a given approach. This second screening resulted in the selection of 38 articles to which we applied our data extraction tool (see Additional file 2) for classification and evaluation of value assessment approaches.

Gray literature search

To complement our peer review literature search, we also conducted a gray literature search. Two main strategies of environmental scan were used. First, we relied on previous systematic reviews of initiatives around disinvestment and/or reassessment of low-value and/or potentially obsolete health technologies to ensure that we are getting at value in the context of resource management [3,4,5,6,7,8]. A search was conducted on the websites of institutions found in these reviews (see Additional file 3). The terms used for the search on each of the identified websites were: low value, disinvestment, reassessment, de-adoption, decommissioning and delisting. The second search strategy consisted of exploring the websites of highly reputable HTA agencies and other relevant professional organizations for presentations, guidelines, working papers or any other pertinent piece of gray literature. For this purpose, we looked at the following organizations: Health Technology Assessment International (HTAi); International Network of Agencies for Health Technology Assessment (INAHTA); International Society for Pharmacoeconomics and Outcomes Research (ISPOR); European Network for Health Technology Assessment (EUnetHTA); International Health Economics Association (IHEA); Agency for Health Research and Quality (AHRQ); Canadian Agency for Drugs and Technologies in Health (CADTH); Kaiser International Health Group; and Blue Cross Blue Shield Association. A total of 1390 documents were identified, of which 52 qualified for full review.

Results

Figure 1 presents the two-stage screening process. We found 22 distinct approaches to assess the value of health care interventions, which were reported in the 38 papers from the peer-reviewed literature search. The gray literature search returned only methodologies and discussions on processes to identify and prioritize low value or obsolete technologies for further evaluation but no particular approach to assess their value per se was identified.

Fig. 1
figure 1

Literature search process and results

Full size image

No gold standard value assessment framework was reported despite the supposedly theoretical supremacy of cost-effectiveness analysis. The existing strategies found here are diverse and vary according to their context and specific objectives. Given the large and heterogenous number of frameworks, our narrative synthesis process involved the development of a classification system to categorize the methodologies found. Our analytical scheme is based on the key structuring questions observed across the existing frameworks of value assessment.

The classification we propose is built around four questions: 1) Is the method based on one consideration/criterion or multiple considerations/criteria? 2) Are the considerations/criteria disease-specific or more generic? 3) Are the considerations/criteria process-oriented or are they directly based on patients’ perspective? 4) What input is primarily used to measure performance and what is the acceptable ‘evidence’?

These four questions expand on the two fundamental issues that must be addressed in any value assessment exercise: what is the entity to be measured and how is it to be measured.

Question #1: one consideration or multiple considerations?

Based on the nature of the health care service to be assessed, one has to determine the relevant criteria to guide decisions. Most scenarios, if not all, are complex and involve several components and objectives. However, it is not always feasible and practical to take into consideration all relevant aspects pertaining to an intervention. In these cases, the use of ‘one consideration’ seems the most common choice, having cost-effectiveness/cost-utility analysis (CEA/CUA) as the most prevalent framework. There is a broad evidence base supporting the use of CEA in assessing value. It should be noted that with CEA/CUA, outcomes can be defined in many different ways, from life years gained to symptoms avoided to units of clinical benefit. On the other hand, if multiple considerations are deemed relevant, then a myriad of approaches may be employed. Although the majority of the value assessment frameworks falling within the multiple consideration category draw some elements from the set of tools broadly known as multi-criteria decision analysis (MCDA), it is rare to see a consistent, structured and explicit discussion of these theoretical elements of decision sciences.

It is important to explain the choice of using the term ‘consideration’ instead of ‘criterion’ or ‘criteria’. Even though ‘criteria’ is indeed an appropriate term for most of the value assessment frameworks identified here, which basically consist of judging an intervention according to a principle, we found that the term is not entirely suitable for some approaches. Thus, the word ‘consideration’ was chosen as a way to avoid semantic inadequacies in some cases, but these terms can be used interchangeably.

Question #2: disease-specific or more generic consideration?

The key in addressing this question is determining whether or not there is a need for generalizability across a range of interventions. If the need for a particular assessment of value is for the purpose of informing choices between similar interventions, then disease-specific criteria are preferable because they will by nature be more directly relevant and the results will typically be more precise. If the decision-making applies to a broader context across different areas of care and levels of organization, then more generic criteria are more appropriate. From an economic perspective, this question relates to whether the decision at hand is one of technical efficiency (i.e., given a decision to treat a certain patient group, what is the best treatment option) or of allocative efficiency (i.e., how to allocate resources among interventions aiming different patient groups).

Question #3: process-based or patient-oriented outcomes?

This question relates to the point in the health care system where value is assessed. If one is looking at the entire service continuum, then a patient outcomes perspective seems intuitive. On the other hand, if the focus is on a specific element within that continuum, such as rehabilitation, or post-surgical care, then adherence to clinical guidelines might be a more appropriate way to assess value as it attempts to isolate the given intervention from the influence of other parts of care. Patient reported outcomes have been around for many years but recently have received greater attention across all health systems.

Question #4: what input and what ‘evidence’?

This question refers to the nature of information used to address the previous questions and the types of evidence acceptable to base these answers. The answer to this fourth question depends in part on the choices made in questions two and three. For example, process-oriented considerations are more likely going to be data-driven whereas certain health outcomes might be possibly best assessed by expert opinion, including, where possible, patients’ feedback. In addition, disease specific contexts are more likely to have established measurement tools. In contrast, when looking at value assessment across disease areas there is likely only lower level evidence available as there are going to be fewer ‘head to head’ studies of disparate treatment areas. In these cases, expert opinion may probably come to play a role in the assessment.

Classification summary

In order to illustrate how this classification might work, take for example the well-known ASCO framework. In this case, for the first question, benefit gains are measured against more than one consideration; for the second question, the focus is on cancer so the criteria lean more towards disease-specific than generic, without being so specific as to apply to only one cancer; for the third question, the criteria focus on health outcomes; and finally, for the fourth question, input is from clinical trial results with the addition of expert opinion, as necessary. Table 1 presents the 22 approaches organized as per our classification system. It must be noted that some of the approaches are, by their nature, very specific in their content while others are made to adjust to the context where they are applied. For example, the ASCO framework contains specific criteria with specific weights while ‘MCDA’ – as a more generic approach - covers any set of criteria developed for any given implementation.

Table 1 Classification of existing value assessment frameworks
Full size table

Our classification is not making any normative statements, rather it is meant as a guide to help in understanding the myriad of value assessment approaches that are currently reported in the literature. Using the four questions to classify the 22 approaches that were identified in the peer-reviewed literature we found that 18/22 use multiple criteria, 12/22 use criteria that are more disease-specific in nature, 17/22 focus on outcomes-oriented criteria and 12/22 have performance measurements that are mostly data-driven.

In addition, explicit process evaluation took place in about one third of the 38 papers reporting on the various approaches. Key evaluation findings across these studies included the need for transparency in both criteria and methods, the relevance of expert panels to support and contextualize data, and in many (but not all) cases, the application of the value assessment approach was found to lead to changes either in resource use or clinical outcomes.

Discussion

All studies found in the literature review identify the depicted methodologies as strategies or frameworks for assessing value of health care technologies. However, only a few explicitly state their underlying concept of value. Some articles that report cost-utility analysis agree with Porter defining value broadly as “health outcomes obtained per dollar spent”. Govaert et al. [39] provide a similar but more specific definition of value in a paper on surgical auditing, understanding value as “the health outcomes achieved that matter to patients, relative to costs of achieving those outcomes”. In the DCE experiment conducted by Green and Gerard [26], they sought to estimate the ‘social value’ of health care technologies, in which construct the notion of ‘value for money expected from the treatment’ is a component. It can be said that it represents an attempt to obtain a measure of allocative efficiency that rely on individuals’ judgement on the evidence around technical efficiency. ASCO defines value “as a combination of clinical benefit, side effects, and improvement in patient symptoms or quality of life in the context of cost”.

Despite the lack of a consensual definition of value and the diversity of frameworks as well as contexts where these emerge, we are able to identify two main constructs across the existing initiatives that represent the key challenges in measuring what is achieved by an intervention: what are the outcomes that are to be considered and how is the level of each outcome measured. We expect that each condition will have a unique set of relevant outcomes and, for any condition, the relative importance of each outcome may vary across patient groups. For example, some patient groups may be more risk averse and for those, the rate of adverse events is a more important outcome than to less risk adverse patients. A further complication in the assessment of the value of an intervention is that, when the focus is on outcomes, the contribution of any specific intervention depends in part on the effectiveness of other interventions related to the condition being addressed [1]. These challenges mean that, as we investigate ways to assess the value of health care, we do not expect that there is only one correct ‘value’ for any intervention, even for a specific condition, i.e. ‘value’ is contextual. Our review reveals the inexistence of a gold standard approach, as the right answer is predicated on the particular context and decision that needs to be addressed.

Seeking to understand the current state of affairs in the realm value assessment, we examined both the peer-reviewed literature and gray literature. These reviews focused on actual cases of value measurement as opposed to theoretical frameworks. It is possible that powerful insights might be obtained from theoretical strategies not covered here. Another limitation of our study is that the search was constrained to the developed world and we acknowledge that interesting initiatives may be taking place elsewhere. And it is also possible that existing value assessment frameworks that are currently used have not been published. In order to address this weakness of a sole focus on the peer-reviewed literature, we also included a gray literature search, seeking to capture broader sources of information.

The fact that we were able to identify 22 different approaches in the peer reviewed literature supports the Sorenson et al. [47] conclusion that there is no single approach that is likely to apply in most situations: “At this time, it is clear that there is no one perfect model or framework for value assessment, or even one that will garner consensus across all stakeholders.” So, what can we learn from these 22 approaches found in our review? A few clear messages emerge.

First, there seems to be an awareness across developed countries, and very much so in the US, of the need to improve the measurement of the value of health care services and there obviously have been serious efforts made to do so. That being said, while the convergence in the results of the different approaches seems to be growing [30], these approaches all seem to have some weaknesses. For example, as Cohen et al. [27] state, “all of the frameworks suffer from varying degrees of arbitrariness, namely, subjectively determined end-points and arbitrary ways of combining scores from multiple dimensions to arrive at a composite health outcome measure. Owing to this arbitrariness, it is unknown the degree to which these frameworks capture value accurately.” The awareness of the need to measure value is related to the financial pressures present in health care. This was confirmed in our gray literature search where we did not find any new approach to value measurement but where we found many instances of efforts to apply value measurement to the search for interventions that should be avoided because they provide low value for their cost. Furthermore, this perhaps speaks to the notion that the practice of value assessment is less developed than the writing and publishing on this subject.

The second message is that while the most prevalent approach cited in the literature is the single criterion CEA/CUA, the vast majority of the 22 approaches found utilize multiple criteria, perhaps reflecting the reality that decision makers typically face a broad set of objectives when assessing value.

Third, the accepted forms of evidence are expanding from the traditional gold standard level of evidence (i.e., RCTs). In fact, about a third of the approaches considered explicitly patient input as part of the process for determining value. This certainly fits with broader trends towards patient-oriented research and the inclusion of patient input in health care decision making [37, 48].

And finally, the focus of most approaches seems to be directly on health outcomes rather than on process measures. The fact that some approaches include disease specific criteria while others are more generic in nature likely reflects decision making practice, i.e., in some cases the focus of the question being addressed is narrower while in other cases the question is broader.

While a comprehensive view of existing approaches for assessing value may be useful to provide insights on further action, the question of what approach should be used in a given context remains unanswered. There has been important work on assessing existing approaches and providing guidance for improvements [30, 47, 49, 50], but our searches did not identify any direct guidance on how to select the best approach for a given situation. Although our classification system was developed as a strategy to scrutinize the diversity of methodologies identified in the literature as value assessment frameworks, one can also rely on our 4-question analytical tool in order to develop novel ways to assess value in contexts hitherto unexplored.

Conclusions

Three main lessons emerge from our literature review on value assessment approaches in health care. First, there is widespread interest amongst health care providers and funders in measuring value due to growing financial pressures. Second, the incremental cost-effectiveness ratio, the traditional value measurement tool, does not seem to meet the needs of decision-makers as demonstrated by the number of approaches being developed. And third, the definition of what is included in value and the methods and inputs involved in assessing performance have expanded significantly and continue to expand. Overall, this paper adds to the understanding of value assessment and should be a helpful guide as academics and decision makers seek to not only unpack the notion of value in health care but also move forward with initiatives to assess value in a given context.

Furthermore, our work reveals several important pathways of future research. First, the inconsistency found in the use of the notion of “value” shows the importance of a careful and robust epistemological reflection about this concept and its practical implications. Originally born within the realm of neoclassical economics with an intrinsic association with the notion of utility and its manifestation in price, value has gained other contours in the health economics literature. The widely cited work [1] by Michael Porter defines value as “health outcomes achieved per dollar spent”, which seems to place value as a maximand that fits well in a welfarist or extra-welfarist paradigm. More recently, the ISPOR’s task force [51] on value assessment presented a complex composite nature of value, although according to them only a few elements have been consistently addressed in the literature. Thus, not only a further reflection on the notion of value must be pursued by the health economics community, but also explicit statements about the underpinning theoretical notions of value should be encouraged by those who delve into empirical initiatives in this realm. Second, as many of the self-described value assessment frameworks did not present any explicit consideration of cost, it would be very important to understand how we can develop more appropriate tools that take into account the costs of health care interventions within each framework. That is related to the fact that the traditional cost-effectiveness approach does not seem to be currently fulfilling the needs of decision-makers and new manners to adequately consider cost may be also necessary. Lastly, just as our interpretive classification scheme, novel approaches to guide decision-makers and managers on how to choose and/or develop adequate practices to assess value of health care technologies need to be further investigated.