Capturing Budget Impact Considerations Within Economic Evaluations: A Systematic Review of Economic Evaluations of Rotavirus Vaccine in Low- and Middle-Income Countries and a Proposed Assessment Framework

Cost effectiveness is one of several key gateway criteria that inform new technology adoption decisions. However, interventions found to be cost effective or very cost effective are often not implemented [1]. While current uncertainties surrounding appropriate cost-effectiveness thresholds may explain some of this [1, 2, 3], overall healthcare budget implications and issues of sustainability also play an important role [4]. Decision makers require estimates of the real financial consequences of introducing a new intervention within a defined budget and budgetary period rather than relying on anticipated savings in economic costs alone [1, 5, 6]. The importance of high-quality budget impact analyses (BIAs) for accurate budgeting and resource allocation is increasingly recognised, especially for severely resource-constrained environments [7, 8]. Many high-income countries require budget impact information alongside cost-effectiveness estimates when making decisions to adopt new interventions [7, 9, 10]. In low- and middle-income countries (LMICs), understanding the short- and long-term impact of intervention adoption on national budgets is critical for ensuring programme sustainability [11, 12], especially when international funding is uncertain or of a temporary nature [13].

Unfortunately, the majority of published economic evaluations are cost-effectiveness analyses (CEAs), with very few BIAs, especially focused on LMICs [14]. Budget impact analyses may be conducted as often but less commonly published compared with CEAs for a variety of reasons, including secrecy on behalf of industry not wanting to disclose tender prices, or journal editors believing BIAs to be more of an administrative or financial task rather than research. Another possible explanation is that appropriate methodology for a BIA remains less well researched and less widely understood than for a CEA [9]. For instance, a previous review appears to classify nearly all (57/68) economic evaluations of rotavirus vaccination as BIAs if they provide an estimate of the net costs of vaccination as part of a CEA, despite these estimated economic (and often discounted) costs being very different from the true financial stream of costs that would be estimated within a BIA [15].

The first BIA analytic framework was published in 1998 [16]. Since then, additional publications have offered methodological guidance for BIAs [7, 10, 17, 18, 19], most recently, the International Society for Pharmacoeconomics and Outcomes Research (ISPOR) Task Force’s best-practice recommendations on conducting BIAs [18]. The recent ISPOR guidelines, however, do not include a checklist that summarises the methodological recommendations and supports a critical appraisal of the quality of BIAs. Such checklists are useful in ensuring the quality of research outputs. For example, CEAs are often appraised using Drummond’s checklist for assessing economic evaluations [20] or the Consolidated Health Economic Evaluation Reporting Standards checklist [21].

In this study, we develop a framework that incorporates the latest ISPOR recommendations into a BIA quality assessment checklist. This framework can be turned into a scoring system to critically assess the quality of a BIA. While there have been previous quality appraisal checklists in this area [14, 22], none have been explicit about how to use them to grade individual studies in particular intervention areas.

As an illustrative example, we develop a scoring system for vaccines. Newer vaccines are more costly than traditional childhood vaccines, representing increasingly large investments [23]. Delivery costs can be as great as vaccine costs, creating sustainability challenges over the long term [24]. Although development assistance for vaccination has increased over the last two decades, this is still a small proportion of total health expenditures for vaccination in LMICs [25]. In particular, for countries ineligible for assistance through Gavi, the Vaccine Alliance or those graduating out of Gavi support, the long-term budget implications from new vaccine adoptions are very important to consider. In fact, government expenditure on vaccines in countries that have adopted new vaccines was recently found to be on average double that of countries that have not [3].

As CEAs are more common than BIAs, particularly in LMICs [14], it would be helpful if they could be adapted to provide budget impact information without requiring much additional data or analysis. Such pragmatic adaptations could be immediately useful to policy makers seeking to fill the gap in stand-alone BIAs. This idea has been proposed by others in relation to vaccine CEAs, based on how population estimates are taken into account [26]. However, to the best of our knowledge, the feasibility of such adaptation has never been systematically assessed, and there is no existing guidance on preparing CEAs with such an objective in mind. We therefore also modify the BIA checklist and scoring system for CEAs, to evaluate whether, and if so, the extent to which, a CEA can be used to provide sufficient information for a BIA.

We first systematically review the literature to identify existing economic evaluations in LMICs. To keep the review manageable, we focus on rotavirus vaccination, which is recommended by the World Health Organization for all countries, particularly those with high rotavirus gastroenteritis case-fatality risks [27]. Most of the estimated 215,000 deaths in children under 5 years of age from rotavirus gastroenteritis occur in LMICs [28], and while there exist many studies demonstrating the cost effectiveness of rotavirus vaccination [15, 29], it has seen only partial uptake in LMICs. We then apply the checklists and scoring systems to examine the extent to which existing economic evaluations provide sufficient evidence about budget impact to enable decision making, and identify key areas where BIA and CEA reporting could be improved for this purpose.

The majority of economic evaluations of rotavirus vaccine in LMICs identified in this review were CEAs. Only six of 66 articles were classified as a BIA, and all but one were conducted alongside a CEA. The BIAs adhered well to the majority of ISPOR’s best-practice recommendations. However, our review identified three areas where BIAs fell short from current best-practice recommendations: not discounting future costs, providing annual or budget relevant financial streams of costs and model validation. Most BIAs were conducted alongside a CEA, and intervention costs were often handled in a manner more appropriate to CEA rather than BIA in terms of discounting and reporting. Previous reviews of BIAs have determined that good methods are not used consistently, with discounting, reporting and sensitivity analyses also reported as key issues in need of improvement, among others [14, 19, 43, 44].

The majority of existing rotavirus CEAs did not provide adequate detail to be useful for budget impact assessment. Only three of 60 CEAs passed the ‘minimum requirements’ to be assessed using the full checklist. Similar to the BIAs, CEAs rarely present undiscounted annual costs, or estimate financial streams during the first few years of programme scale-up. One pragmatic recommendation for CEAs (and those who commission them) could therefore be to explicitly require the reporting of yearly undiscounted costs in addition to the standard reporting of costs within a CEA (possibly in an appendix if not directly relevant for the cost-effectiveness results), and estimates of real financial streams during the first few years of programme scale-up.

There are important limitations of this study that merit mentioning. First, in constructing a checklist and scoring criteria, we have simplified the recommendations laid out by the ISPOR Task Force. While we attempted to capture the most important considerations addressed in the best-practice recommendations, the categories included in the checklist are not an exhaustive list. For example, our checklist does not capture whether the model allows users to input or vary parameter values, an important consideration for BIAs according to the guidelines. Additionally, some categories included in the checklist can be subjective, for example, stating appropriate conclusions and limitations from a budgetary perspective. As such, in developing the scoring system, we do not consider the quality and completeness of each of the items in the checklist. Instead, we have developed the scoring system to be as explicit as possible for scoring vaccine-specific articles.

We focused only on full economic evaluations in this review. Thus, we do not include vaccine costing studies that may have tried to provide an estimate of budget impact but did not take into account cost savings from reduced burden on the healthcare system. Only a small number of BIAs was included in our review, limiting the discriminatory power of our checklist and the generalisability of our findings to BIAs conducted in areas outside of rotavirus vaccine. Finally, it is important to reiterate that the majority of papers reviewed were not intended to be formal BIAs. As such, the ‘feasibility’ scores for CEAs do not reflect an assessment of their quality as an economic evaluation per se, but rather the extent to which they provide useful budget impact information. This is especially the case for articles scored only on Essential items. For these items in particular, scores do not reflect how well, or how appropriate for the budget holder this item was handled in the analysis. Rather, they reflect whether or not the item was taken into account or specified at all. For example, CEAs specifying the size of the eligible population score full marks for the essential component of this item, regardless of whether the population size was appropriate or justified.

Despite these limitations, our study provides a simple and easily implementable quality assessment checklist and illustrates how it can be converted into a scoring system, in this case for vaccine-specific BIAs. It is to the best of our knowledge, the first framework for systematically evaluating the feasibility of adapting a CEA to a BIA. We identified several areas where existing BIAs fell short from best-practice recommendations. We also identified relatively straightforward changes to the presentation of CEA results that could facilitate the adaptation of most cost-effectiveness studies into BIAs, potentially increasing their utility and application. Although we have used produced a vaccine-specific scoring system, the checklist could similarly be adapted to other types of interventions.

To assist countries in making informed decisions about adding a vaccine to a national immunisation programme, the World Health Organization emphasises that programmatic, economic and financial feasibility are important criteria among others to consider [45]. This is especially true in the case of LMICs not eligible for financial support or price negotiations through Gavi and those graduating out of Gavi support [4]. For these countries, the real fiscal costs of introducing and sustaining new vaccines within immunisation schedules will have enormous implications on a country’s immunisation budget and overall healthcare budget. The World Health Organization’s global vaccine and immunisation policy recommendations also acknowledge budget impact considerations as being highly relevant alongside cost effectiveness [46].

There is potential for improved decision making in the area of vaccines through programmes such as the ProVac Initiative, which provides economic evaluation tools including a CEA model (TRIVAC) and BIA model (COSTVAC) along with country-level support. [47]. Beyond vaccines, the ‘Gates Reference Case, recently developed through funding by the Bill and Melinda Gates Foundation with the aim to improve the quality of economic evaluations conducted in LMICs, recommends reporting on budget impact as one of 11 key principles [48].

The limited number of published rotavirus vaccine BIAs in LMIC settings compared with CEAs may reflect the need for more guidance on best-practice methodologies for BIAs of preventive interventions in these settings. In the context of the current difficulties in determining appropriate threshold values for cost-effectiveness ratios in LMICs, there is potentially greater value for budget impact estimates for reflecting opportunity costs for these countries. Additional use and validation of the BIA quality assessment framework proposed in this review is necessary and would be useful for further work. For example, these checklists and a vaccine-specific scoring system could be applied to economic evaluations of other vaccines, and results compared with the findings in this review. Further research is needed to investigate how this framework could be adapted to other public health or medical interventions.

Both budget impact and cost effectiveness are key criteria, among others, for policy makers deciding how to allocate limited resources. In our study, we propose and demonstrate the use of a generalised BIA checklist and vaccine-specific scoring system for guiding and assessing the quality of BIA as a stand-alone analysis or alongside CEA, and highlight current gaps in standard practices. The proposed framework could facilitate the uptake and improvement of the quality of BIAs useful for decision making, in particular in LMICs.