Are Current Cost-Effectiveness Thresholds for Low- and Middle-Income Countries Useful? Examples from the World of Vaccines
The World Health Organization’s CHOosing Interventions that are Cost Effective (WHO-CHOICE) thresholds for averting a disability-adjusted life-year of one to three times per capita income have been widely cited and used as a measure of cost effectiveness in evaluations of vaccination for low- and middle-income countries (LMICs). These thresholds were based upon criteria set out by the WHO Commission on Macroeconomics and Health, which reflected the potential economic returns of interventions. The CHOICE project sought to evaluate a variety of health interventions at a subregional level and classify them into broad categories to help assist decision makers, but the utility of the thresholds for within-country decision making for individual interventions (given budgetary constraints) has not been adequately explored. To examine whether the ‘WHO-CHOICE thresholds’ reflect funding decisions, we examined the results of two recent reviews of cost-effectiveness analyses of human papillomavirus and rotavirus vaccination in LMICs, and we assessed whether the results of these studies were reflected in funding decisions for these vaccination programmes. We found that in many cases, programmes that were deemed cost effective were not subsequently implemented in the country. We consider the implications of this finding, the advantages and disadvantages of alternative methods to estimate thresholds, and how cost perspectives and the funders of healthcare may impact on these choices.
Key Points for Decision Makers
We found that an analysis estimating a vaccination programme to be ‘very cost effective’ in a particular country was not sufficient to lead to funding.
It is likely that other factors beyond cost effectiveness, including the overall budgetary impact, are particularly important for decision making in low- and middle-income countries.
For local decision makers, the criterion for understanding cost effectiveness should have some relation to the budget available for allocation.
1 The Need for Cost-Effectiveness Thresholds
The reporting of incremental cost-effectiveness ratios (ICERs) in terms of the cost per unit of outcomes (such as a disability-adjusted life-year [DALY]) has limited value to decision makers without a reference for what value is considered too high . In this way, the establishment of a threshold beyond which an intervention is not considered cost effective creates a criterion to better understand the results of cost-effectiveness analyses for decision making. However, the way in which a decision maker should interpret and potentially use thresholds to help set policy depends upon several factors, including the different ways in which thresholds can be estimated (see Box 1). These distinctions are often not well articulated in the cost-effectiveness literature, which may have limited the utility of thresholds to help appropriately inform policy decision making, particularly in low- and middle-income countries (LMICs), which have less experience in using cost-effectiveness criteria for decision making.
2 The History of Threshold Creation in LMICs
Although they have been discussed for some time , the widespread reference to thresholds for cost effectiveness in LMICs emerged out of the World Health Organization’s CHOosing Interventions that are Cost Effective (WHO-CHOICE). This initiative aimed to build global and regional databases of the relative costs and effects of interventions for a wide range of diseases and risk factors, to identify the optimal mix of interventions to address issues of allocative efficiency in health [3, 4]. The WHO-CHOICE initiative built league tables containing an ‘expansion path’ of interventions, starting from those that were found to be most cost effective. The tables of interventions were then subdivided into those deemed ‘very cost effective’ because they had a cost-effectiveness ratio below the gross domestic product (GDP) per capita, those deemed ‘cost effective’ with cost-effectiveness ratios below three times the per capita GDP, and the remainder that were not cost effective. In the absence of locally established cost-effectiveness thresholds in many LMICs, the original purpose of the ‘WHO-CHOICE thresholds’ was to help classify interventions into these three broad categories to assist with the policy debate on allocative efficiency by providing “decision-makers with information on which interventions are low-cost ways of improving population health and which improve health at a much higher cost”, with an explicit warning to avoid formulaic use [3, 4]. However, since their initiation, the thresholds have been widely adapted by researchers to help advise on within-country decision making for individual interventions.
The thresholds chosen were based upon criteria set out in a report by the WHO Commission on Macroeconomics and Health . The report argued that the value of saving a life-year (or preventing a DALY) should be at least equal to the per capita income or the extra market income created. It was also suggested that the real value of such gains may be several times (e.g. up to three times) the per capita figure to reflect the value of other factors beyond market income, such as changes in longevity, and pain and suffering. The figures mentioned were based not on specific empirical evidence but, rather, on a general assessment of the economic literature, which found that the value of an extra year of healthy life is “worth considerably more than the extra market income earned in the year” and that “According to some estimates, each life-year is valued at around three times the annual earnings” .
Box 1: Three Methods of Estimating Cost-Effectiveness Thresholds (Adapted from )
• ‘Human capital approaches’: in this method, thresholds for human life (and other outcomes, e.g. DALYs) are established, based around the average income of individuals within the society. This approach may be interpreted as reflecting an implied human capital method  or that individuals are entitled to their ‘fair share’ of a nation’s wealth .
• ‘Preference approaches’: in this method, thresholds are established either by stated or by revealed preferences of individuals within the society for the outcome of interest (e.g. life-years saved or DALYs)—for example, a representative survey of the willingness of individuals to pay for the prevention of the outcome .
• ‘League table approaches’: the threshold created by this method is the cost-effectiveness ratio of the last intervention (from a table of most to least cost effective) to be approved as funds are exhausted . If a new intervention falls below this threshold, then (theoretically) greater benefits could be created by having it replace this marginal intervention in a fixed budget.
3 How Have the WHO-CHOICE Thresholds Been Used for Vaccines in LMICs?
The ‘WHO-CHOICE threshold’ values of one to three times per capita GDP have been widely cited and used as a measure of (incremental) cost effectiveness in evaluations of vaccination for LMICs [10, 11]. The growth in their use in the literature may reflect the lack of alternative thresholds in the majority of LMICs. However, the use of these thresholds has often come with an explicit caveat that the thresholds may not reflect the budget affordability of the programme being evaluated. For example, Vanni et al. , who evaluated human papillomavirus (HPV) vaccination in Brazil, used these thresholds but acknowledged that they may not be appropriate in their setting. They also reported results for a threshold many times lower than the per capita income value, which they suggested may reflect a more realistic measure of the cost effectiveness of the other interventions competing for the same investment (i.e. the opportunity cost).
Thailand is one of the only LMICs to have a locally established threshold (100,000 Baht in 2008 values) as a guide for decision making. This local threshold is below the thresholds put forward by the WHO and has been used as a reference value in some vaccine evaluations for Thailand . However, other studies evaluating HPV and rotavirus vaccination in Thailand have used the ‘WHO-CHOICE thresholds’ instead of the local threshold [14, 15, 16]. Again, these studies have often made a distinction between interventions that represent value for money (i.e. are ‘cost effective’ when compared with the ‘WHO-CHOICE thresholds’) and those that are affordable in the context of the local budget and other competing healthcare priorities.
4 Do WHO-CHOICE Thresholds Inform Funding Decisions for Vaccines in LMICs?
We compared the results of these cost-effectiveness analyses with implemented vaccination programmes (including partial introductions and those suspended after introduction) as recorded in the WHO Immunization, Vaccines and Biologicals (IVB) Database as of 29 May 2013 [17, 18]. The majority of analyses identified in the two reviews [10, 11] reported base-case ICER estimates as either the cost per DALY prevented, or life-years or quality-adjusted life-years (QALYs) gained. For simplicity, we did not distinguish between these measures when making comparisons with thresholds, but we excluded those analyses that did not report a base-case ICER estimate or that reported ICERs for intermediate outcomes (e.g. the cost per case prevented). Studies applied either a societal or a healthcare perspective; where both perspectives were reported, we chose the results from the societal perspective to be consistent with the reference case recommended by the WHO . The HPV review (Fesenfeld et al. ) reported results in 2011 US dollars, and we used the same methodology to update the figures reported in the rotavirus review  to this cost year, using the US Consumer Price Index for all urban consumers .
Figures 1 and 2 show that there have been studies in many countries where an evaluation found a cost-effectiveness estimate below the per capita GDP but where there is still no vaccination programme funded. For example, in Fig. 2, we see that all of the rotavirus studies reviewed reported an ICER below 50 % of the per capita GDP for the country of analysis; however, many of the countries have not yet implemented vaccination. In Fig. 1, for HPV there does appear to be some relationship between the decision to introduce a vaccination programme and the cost-effectiveness ratio estimated (as a proportion of per capita income) for that programme, with none of three countries that estimated a cost-effectiveness ratio greater than double the per capita income having implemented vaccination. However, only 5 (of the 13) HPV analyses that found an ICER below 50 % of the per capita GDP have implemented vaccination (to date). In general, rotavirus vaccination appears to have lower ICERs than HPV vaccination, possibly because the benefits occur sooner after vaccination and hence are less heavily discounted.
The evidence from these studies suggests that a cost-effectiveness analysis that concludes that an intervention is ‘very cost effective’ is not sufficient to lead to (enable and/or allow) funding of a vaccination programme. This could reflect a number of possibilities. First, many LMICs implicitly (or explicitly) use a different threshold from the ‘WHO-CHOICE thresholds’ when making vaccine introduction decisions. Second, LMICs may not consider cost-effectiveness results, because they lack the expertise to interpret them. Third, other criteria are more important than cost effectiveness for such decisions, as suggested by qualitative research on this topic . For instance, while most of the analyses reviewed for both HPV and rotavirus make reference to the ‘WHO-CHOICE thresholds’ in helping to interpret the results of the cost-effectiveness analysis [10, 11], they often make clear that budgetary constraints are a primary consideration in determining whether a ‘cost-effective’ intervention is implemented. Fourth, LMICs are interested in cost-effectiveness considerations but place different values on the costs and benefits included in the analyses reviewed. For instance, they may place less weight on productivity losses or quality-of-life impact not resulting in death.
Nevertheless, the limited data we reviewed do suggest that cost-effective analyses may be having some impact on vaccine funding decisions by reducing the likelihood that programmes at the higher end of the threshold (above double the per capita GDP) are implemented (Fig. 2).
5 What Could Be a Way Forward?
While the ‘WHO thresholds’ have been useful for sectoral cost-effectiveness analysis to address allocative efficiency questions, they may not capture the affordability and sustainability of funding an intervention, or the value of other interventions that are likely to be displaced by such spending . They have also been acknowledged as often being too high in the context of local healthcare budgets , leading to a situation where programmes are deemed ‘cost effective’ yet are unable to be implemented. It may be that they help to place an upper bound on what is deemed ‘cost effective’, helping decision makers to avoid allocation of resources to interventions that exceed these relatively high thresholds.
Cost-effectiveness considerations form part of the guidelines for health technology introduction in many developed countries . Their role ranges from being advisory only to being a necessary condition for funding, but often no ‘hard’ or explicit thresholds are stated. Instead, there may be an informal range that decision makers use to understand what the approximate value is in their context. In this approach, interventions falling at the lower end of the range are more likely to be funded, and interventions at the higher end (or above) are less likely to be funded unless on the basis of other criteria (e.g. equity) beyond cost effectiveness. The flexibility in this approach can provide a bargaining tool to negotiate with suppliers over the prices charged for interventions without fixed costs . It has been suggested that thresholds in middle-income countries may also help in drug price negotiations .
Ideally, thresholds for decision making should help to inform decision makers what represents approximate value for money, given the local budget constraints, local consensus about the value of improving health and the alternative ways available to allocate the resources in the setting being considered. In addition to affordability and cost effectiveness, other criteria that are beyond the scope of traditional cost-effectiveness analyses may also need to be considered. For example, discrete choice experiments conducted among national policymakers in five countries suggested that equity is a key consideration for multi-criteria decision analysis in one of the countries (Cuba) .
6 Alternative Methods of Estimating Thresholds
Both ‘human capital approaches’ and ‘preference approaches’ have the disadvantage that the theoretically optimum threshold may not actually reflect the actual availability of healthcare resources, i.e. the affordability of an intervention (Box 2). While the thresholds may predict a theoretical benefit to society (with costs outweighed by the value placed on the outcomes achieved), they may also result in lesser health gains than implementation of another health spending option available to the decision maker at the time. They may also result in net health losses if their implementation displaces existing health programmes that have greater health benefits. For instance, one study has suggested that rotavirus vaccination in India is highly cost effective but would cost about 12 % of the national health budget to implement .
The league table approach offers scope to explicitly account for budget constraints but is challenging to implement because of the difficulty in estimating the cost effectiveness of all possible healthcare interventions. Such exercises have been conducted on a global level (e.g. WHO-CHOICE, Disease Control Priorities Network) but may be difficult on a country level. One potential alternative is to obtain a rough but usable approximation of the maximum cost-effectiveness ratio of an intervention beyond which more cost-effective health investments are likely to be displaced, keeping in mind that displacement is likely to be suboptimal . One example is a recent analysis relating spending by primary care trusts (which commission local health services) in England to levels of health and mortality in different disease areas within their local area .
In LMICs, the data on local spending decisions needed for such an analysis may not exist. An even rougher approximation may simply be to use the average ICER for any recent health spending decisions for which economic evaluations exist. For instance, a recent analysis compared the financial resources needed and cost effectiveness of introducing either rotavirus or HPV vaccination in Global Alliance for Vaccines and Immunization (GAVI)-eligible countries . Since most health systems are not perfectly efficient (and hence a new investment may not necessarily displace the least cost-effective current investment), it may be better to use the incremental cost effectiveness of a typical investment rather than that of the least cost-effective investment.
Box 2. Advantages and Disadvantages of Threshold Methods for LMICs
• ‘Human capital approaches’: the threshold values estimated under this approach are easy to determine; however they may not reflect the budget available or the preferences of society. The approach may also be theoretically problematic with the per capita GDP threshold implying a society may be willing to devote it's total GDP to healthcare if appropriate interventions were available .
• ‘Preference approaches’: the threshold values estimated under this approach may help inform on questions of how much to spend on health but may be of limited use where budgets are very constrained. As the thresholds are not linked to the available budget, their use can lead to continual growth in healthcare spending as more interventions emerge that meet the cost-effectiveness criteria .
• ‘League table approaches’: to estimate threshold values under this approach, there is a requirement to evaluate all potential programmes to determine the last intervention to be approved as funds are exhausted. While this theoretical ideal is unachievable, it may still be informative at a coarse-grain level (e.g. the Oregon ‘experiment’  or WHO-CHOICE [3, 4]) or by donors having a fixed amount to spend on a specific intervention type (e.g. vaccines).
7 Who Pays and What Is the Cost Perspective?
The existence of different funders of healthcare interventions in LMICs may also impact on the interpretation and application of thresholds. For example, donors and government decision makers are likely to have different budgetary constraints and also different (and perhaps restricted) ways in which the available resources can be allocated. Alongside the alternative priorities and non-economic criteria that these decision makers have, these discrepancies mean that, depending on the underlying methodological assumptions applied (Box 1), estimation of a single cost-effectiveness threshold for a given country or region may not be possible. A threshold based on human capital or global equity considerations (such as the ‘WHO-CHOICE threshold’) may conceivably be useful when there is an expectation that funds will be raised by external donors who may transfer funds from other spending towards international aid budgets.
Consistency in the methodological approach is vital to allow comparison of results from different economic evaluations. The reference case recommended by the WHO is that of a societal perspective , which typically incorporates productivity costs. This choice in the cost perspective has implications for the estimation of thresholds. The threshold value from the WHO Commission on Macroeconomics and Health  essentially values life-years in dollars, incorporating both productivity changes and the intrinsic utility of life and health. When combined with a societal perspective, this approach means that productivity changes may be double-counted, as they are valued in both the numerator (in any specific productivity changes used in the analysis) and the denominator (in the threshold value) of the cost-effectiveness ratio.
Another important consideration for low-income countries and immunization programmes is the role of the GAVI Alliance. The GAVI Alliance offers financial support and low vaccine procurement prices to 53 GAVI-eligible countries for vaccines, including rotavirus vaccine (since 2011) and HPV vaccine (since 2012). Around 35 and 20 countries have been approved to introduce rotavirus and HPV vaccination, respectively, with GAVI Alliance support (http://www.gavialliance.org). In eligible countries, the price paid for the vaccine in the initial years of vaccine introduction is heavily subsided by the GAVI Alliance, with the cost of the programmes being transitioned to the countries themselves in later years. It has been suggested that the availability of this external support can be a major driver in vaccine adoption decisions . Furthermore, the choice of payer (whether the GAVI Alliance, the national government or individual households) can substantially affect key outcomes of vaccine evaluations, including vaccine costs (whether GAVI-negotiated prices, country co-payment levels or market prices) and benefits (which depend on which budget they accrue to and what outcomes are valued by the payer).
8 Some Limitations of this Article
The use of alterative outcome measures beyond DALYs, such as life-years or QALYs, with different underlying assumptions, further complicates the use of thresholds . However, as the issues we explore apply equally to all, an exploration of these issues was beyond the scope of this article. It is also important to acknowledge the importance of appropriately considering uncertainty when considering cost-effectiveness thresholds. The ranking and probability of different programmes being found to be cost effective can change when the high and low values from sensitivity analyses are considered , such as traditional one-way or probabilistic sensitivity analyses. Additionally, understanding how decisions are made and comparing them with international recommendations is an important area of study in order to achieve evidence-based and socially optimal decision making in health.
Despite the debate around how to estimate thresholds and the technical complexities involved, it is important to remember that some form of criterion is necessary to interpret the results of cost-effectiveness analyses. We suggest that for local decision makers in LMICs, it is important that the criterion for understanding cost effectiveness should have some relation to the budget available for allocation. A major aim of the threshold should be to ensure that local decision makers do not allocate resources to interventions that offer poor value compared with other existing or potential health spending choices. To achieve this, the threshold for national decision making must be estimated relative to other (implementable) health programme alternatives, rather than by approaches that seek to place an external value on the outcomes being achieved (e.g. DALYs).
A.T.N. holds an NHMRC Training Fellowship (630724—Australian Based Public Health Fellowship).
R.H. is a staff member of the World Health Organization. The views expressed are those of the author and do not necessarily represent the views of the World Health Organization.
A.T.N. has, in the past, received research funding from a vaccine manufacturer for other previous projects.
All authors contributed to the intellectual content of the article. A.T.N. initiated the project and led the writing of the manuscript. M.J. and R.M. assisted in the writing of the manuscript. M.J. designed and produced the figures for the article.
- 1.Drummond M, Sculpher MJ, Torrance GW, et al. Methods for the economic evaluation of health care programmes. 3rd ed. Oxford: Oxford University Press; 2005.Google Scholar
- 3.Tan-Torres Edejer T, Baltussen RM, Adam T, et al. Making choices in health: WHO guide to cost-effectiveness analysis. Geneva: World Health Organization; 2003.Google Scholar
- 5.WHO Commission on Macroeconomics and Health. Investing in health for economic development: report of the Commission on Macroeconomics and Health. Geneva: World Health Organization; 2001.Google Scholar
- 7.Williams A. What could be nicer than NICE?. London: Office of Health Economics; 2004.Google Scholar
- 17.Decision making and implementation of rotavirus vaccines. 2013; http://www.who.int/nuvi/rotavirus/decision_implementation/en/index1.html.
- 18.Decision making and implementation of human papillomavirus vaccines. 2013; http://www.who.int/nuvi/hpv/decision_implementation/en/.
- 19.US CPI inflation calculator. http://www.bls.gov/data/inflation_calculator.htm.
- 26.Rose J, Hawthorn RL, Watts B, et al. Public health impact and cost effectiveness of mass vaccination with live attenuated human rotavirus vaccine (RIX4414) in India: model based analysis. BMJ (Online). 2009;339(7724):787–91.Google Scholar
- 28.Claxton KM, Soares M, Rice N, Spackman E, Hinde S, Devlin NS, Smith PC, Sculpher M. Methods for the estimation of the NICE cost effectiveness threshold. York: University of York; 2013.Google Scholar