FormalPara Key Points for Decision Makers

The COVID-19 pandemic shows that the impact of effective vaccines can extend well beyond vaccinated individuals and healthcare systems.

This study analysed to what extent the broader value of vaccines is considered in HTA and found a substantial gap between value generation and recognition.

Five priority areas for broader value recognition include considering (1) more comprehensive cost offsets within the health care system, (2) carer quality of life, (3) transmission value, (4) prevention of antimicrobial resistance and (5) macroeconomic effects.

1 Introduction

The COVID-19 pandemic has revealed the broad and devastating health, economic and societal impact of a highly infectious and deadly disease for which no effective vaccine was readily available. While initially the human suffering of COVID-19 patients made the headlines, this soon proved to be proverbial ‘tip of the iceberg’. Consider the toll on patients’ families, friends, colleagues, and other social networks; on healthcare staff working around the clock. And on the capacity of health systems, buckling under the pandemic pressures with consequences that may last for years. Then, add to that the wider economic impact, affecting economies worldwide.

The above indicates that an effective vaccine against COVID-19 will have a broad value to society. Value that extends well beyond ‘just’ preserving the health of the vaccinated individuals and avoiding the costs of treating patients with this disease. Yet, the ability to generate some amount of broader value to society is by no means unique to a COVID-19 vaccine. In fact, health economists have long recognised the broad value attributes of vaccines and called for the recognition of those in the value assessment of vaccines [1,2,3,4,5,6,7,8,9].

From an economic perspective, optimising the allocation of scarce resources is the fundamental aim of decision makers responsible for health care budgets, and for tax revenue more widely. In many health systems, decisions about which health technologies to fund—and the level at which they are reimbursed—are informed by Health Technology Assessment (HTA), which evaluates the clinical and/or cost effectiveness of a health technology [10].

Traditionally, HTA considers “only benefits in terms of improved health, reduced health care costs and resource use (and improved quality of care) and short-term productivity increases to patients and their caregivers” [9]. This decision-making approach is consistent with the ‘health maximisation’ objective of health systems that is advocated by the ‘extra-welfarist’ school of thought. However, health technologies such as, but not limited to, vaccines can also generate substantial ‘externalities’ (i.e. indirect effects on third parties) [8].

Externalities are defined as spillover benefits and/or costs of a product’s activity, beyond the effects on the immediate consumer, to other consumers, which are not accounted for in market transactions [11]. In the context of health care, these are benefits and costs to the health system, beyond those attributable to the treated patient, and to the broader society. In the value of vaccines literature, these effects have been termed ‘broader’ benefits of vaccinations, to underline that such effects fall outside the scope of traditional health technology assessments [9]. While some of the ‘broader’ benefits have been shown to be particularly large in the case of vaccines [8], consideration of ‘broader’ benefits, where they are relevant, should be applied to all interventions funded by the same budget, to assure consistent decision making [7, 9]. If major 'broader' effects of vaccines and other health technologies are neglected in HTA, however, their true cost effectiveness may be underestimated. As a result, the allocation of health care resources will be sub-optimal, and the objective of allocative efficiency undermined.

To understand the magnitude and nature of this potential discrepancy, this study, part of the Broader Value of Vaccines (BRAVE) Initiative, sought to describe the gap between the broader value elements identified in value frameworks for vaccines and the value elements considered in HTA of vaccines in nine high income countries, and develop expert-informed, consensus-based recommendations as to how key hurdles for broader value recognition could be overcome.

2 Methods

Our study followed a modified Delphi method to develop consensus on the broader value of vaccines paradigm and, based on the state of play of vaccines value assessment in nine target high-income markets (Belgium, Canada, France, Germany, Italy, Japan, Sweden, UK and US), develop a roadmap for advancing the consideration of the broader value of vaccines. The Delphi method is a consensus-based strategy that combines existing evidence with expert opinion collected iteratively to generate balanced agreement. In recent years it has been used successfully to identify gaps and priorities for health economics research on decision making and evidence generation methods [12, 13].

We employed a four-step modified Delphi method consisting of a literature search (phase I), two consecutive phases of expert elicitation (phase II and III) and synthesis of results into recommendations (phase IV). The expert panel was composed of ten experts with specific expertise on vaccine value assessments and in-depth knowledge of HTA methods in one or more of our nine target markets, and a background working in or with HTA decision-making bodies and academia. These eligibility criteria were assessed based on the experts’ track record of academic publications on vaccines value assessment and HTA methods, and membership to HTA decision-making bodies in one of our nine target markets or employment in academia.

2.1 Phase I: Literature Research

The purpose of the literature research was twofold: (1) to synthesise current evidence to characterise the broader value generated by vaccines; (2) to understand the current ‘state of play’ for assessments of the value of vaccines in the nine target markets included in this study.

First, a targeted literature review was performed in January 2020 to identify existing frameworks for the broader value generated by vaccines and recommendations for considering such value elements in HTA and related decision-making processes. We used the ‘pearl-growing’ approach [14] to identify peer-reviewed and grey literature. We limited the search to PubMed, Web of Science and Google (for grey literature) and papers published from the year 2000 onwards. The paper by Jit et al. [15] served as the foundation for our literature search, as it is one of the first to report a systematic review of the broader economic impact of vaccination, considering and appraising the strength of evidence for this. We then searched for papers cited by this paper and papers citing this paper to identify both older and newer relevant papers. Based on the findings of the literature review, we developed a synthesising framework depicting the various value elements to be considered in vaccines assessment.

We then reviewed published HTA guidelines and relevant grey literature to understand which of these value elements are currently considered in HTA processes in each of the nine target markets included in this study. In cases where there was no formal reference to a value element in the HTA guideline, we undertook a rapid literature review and asked for written feedback from a recognised expert in that country to determine whether it might be informally considered within HTA and the wider decision-making process, and the frequency with which this takes place. This allowed us to identify a list of gaps between value elements currently considered in vaccines assessments and value elements that have been recommended in the (health economic) literature and by authoritative international bodies.

2.2 Phase II: Expert Panel Survey

In the second phase of the modified Delphi method, the experts were invited to take part in an email survey (April–May 2020, see Appendix 1 in the electronic supplementary material [ESM]) in order to (1) validate the conceptual appropriateness of the vaccines value framework, (2) provide insights on how the value assessment of vaccines in their country of expertise is conducted in practice, compared with the description in the published guidelines and (3) perform a first round of prioritisation, from the list of gaps in vaccines value assessment identified in our literature review, of the value elements for inclusion in HTA.

Each expert was provided with a description of the value framework synthesised from the categories of value of vaccines that were identified in the literature review, and its underpinning evidence. The experts were asked to provide qualitative feedback on the clarity, relevance, and soundness of each value element from a general HTA perspective, and comment on the extent to which this value element was or could be considered in the current HTA process for vaccines in their country (and if not, why not).

Next, they were asked to each prioritise for discussion three value elements that are currently not consistently considered, using the following criteria: (1) the feasibility of addressing this value element, and (2) the potential impact of including this value element on the outcomes of an HTA. Participants were also asked to state whether they felt each prioritised value element was a short- or medium-term goal.

2.3 Phase III: Expert Panel Round Table

The third phase of the modified Delphi method consisted of a 2-day virtual round table meeting (May 2020). The objectives of the round table meeting were to (1) re-iterate the prioritisation exercise of the gaps towards recognition of the value elements, (2) discuss current barriers to their full recognition and (3) develop recommendations for overcoming these barriers.

At the round table, the results of the first round of the prioritisation exercise were presented, showing the experts’ top three value elements that are currently not consistently considered. The experts were then asked to reflect on those and elaborate on the underlying causes of variations. Following clarifications and a moderated discussion, participants were invited to repeat the prioritisation process, using online polling. Based on the results of this second round, experts were subsequently asked to choose a maximum of five priority gaps for further discussion from the list of priority gaps (where gaps received the same number of votes, they were ranked from high to low by the number of countries in which they occurred).

Five moderated sessions were then conducted to discuss each of the prioritised value elements in turn. First, participants were asked to locate their country on a stylised roadmap (see Appendix 2 in the ESM) representing whether they had already surpassed, or had yet to overcome, hurdles towards the full HTA recognition of the prioritised value element. Hurdles were conceptualised in terms of three factors required for HTA policymakers and practitioners to fully recognise any aspect of value created by vaccines:

  • Evidence: high-quality data demonstrating the value accrued for each relevant value element.

  • Ability: technical/analytic tools and approaches to incorporate this evidence into HTA and broader decision-making processes.

  • Willingness: motivation to incorporate this evidence into HTA and broader decision-making processes.

This was a qualitative exercise facilitated by a virtual whiteboard that was intended to generate discussion and provide an intuitive visual of the country’s relative position compared with a full HTA recognition for each prioritised value element. Further, participants were asked to collectively discuss and recommend ideas or real-world examples of how the existing hurdles (in any of the target markets) could be overcome. The input provided was recorded on the virtual whiteboard and qualitative discussions were audio recorded.

2.4 Phase IV: Synthesis of Recommendations

Finally, the input received during the round table was analysed thematically. For each prioritised value element, we identified the most common barriers across countries in order to produce a summary position of the most common hurdles towards full HTA recognition. Further, we evaluated the relevance of the proposed solutions and synthesised recommendations based on the need for and availability of tools to advance change across countries. All experts were asked to confirm the resulting consensus view and the recommendations as the final deliverable of the round table.

3 Results

3.1 Framework for Broader Value of Vaccines

The literature research resulted in the identification of four frameworks that describe the broad value generated by vaccinations [3,4,5,6,7]. Recognising the many overlaps between these frameworks (See Appendix 3 in the ESM), we developed a synthesising framework (Fig. 1) that aims to provide a comprehensive overview of the categories of value that might result from vaccines.Footnote 1 There is considerable interest and activity in the research field of value frameworks at present, for vaccines and other types of health technologies. The framework aims to capture all of the categories of value so far attributed to vaccines in the four vaccine-specific frameworks identified in our literature review. However, it is also intended to be flexible to incorporate new dimensions as research on the broader value of vaccines and health technologies more generally evolves, whilst trying to minimise the risk of ‘double-counting’.

Fig. 1
figure 1

OHE vaccine value framework. AMR antimicrobial resistance, OHE Office of Health Economics, QoL quality of life

Full size image

The framework distinguishes four categories of effects: (1) ‘narrow’ health effects, concerning the impact of vaccines on the health of vaccinated individuals; (2) ‘broad’ health effects, concerning the impact of vaccines on the health of the unvaccinated population; (3) health system economic effects, concerning the costs of vaccination and its cost offsets to the healthcare budget and (4) societal economic effects, concerning the economic impact of vaccines outside of the health system, for example on productivity or macroeconomic growth. Within these we identified multiple distinct value elements. This structure aligns with the perspectives commonly adopted by HTAs, which typically consider narrow health effects and economic effects within the health system, but not necessarily effects external to these.

We note that these value elements are not necessarily exclusive to vaccines, yet extending the discussion on broader value to other types of health technologies is considered beyond the scope of this paper. Also, while acknowledging that the relevance of value elements might vary by vaccine and pathogen, for each value element the majority of experts considered this to be conceptually appropriate for the assessments of the value of vaccines. This opinion was elicited in two rounds of anonymous voting, during the two consecutive phases of the Delphi process. Table 1 describes each element and the underpinning evidence as well as the experts’ comments on its appropriateness for the purposes of HTA, and the results of the voting exercises.

Table 1 Description of value elements, underpinning evidence and expert comments
Full size table

3.2 Broader Value Recognition in Current HTA-Processes

Six of the 12 value elements included in the broader value of vaccines framework are not considered or uncommonly considered in several countries’ HTA processesFootnote 2 (see Table 2). These include carer quality of life, value to other interventions, impact on preventing antimicrobial resistance (AMR), productivity of patients, productivity of carers and macroeconomic effects. Furthermore, the value elements of transmission value, social equity and cost offset to the healthcare system are not formally considered in some countries. While these gaps result to some extent from the perspective used in HTA (i.e. a health system’s perspective versus a societal perspective), this does not explain all gaps. Transmission value and impact on AMR, for example, are also relevant in a health systems’ perspective and so are cost offsets to the healthcare system. Appendix 4 gives a more detailed description of the discrepancies observed (see ESM).

Table 2 Value elements considered by country
Full size table

Based on the literature and confirmed by the experts, between-country variation in the discrepancies observed is, in part, due to the following:

  • The existence of specialist technical groups for assessing vaccines. In Canada, France, Germany, Sweden, the UK and the US, HTA of vaccines is carried out separately from that of other interventions, by specialist technical groups or committees. No such provision is made in Belgium, Italy and Japan. In countries where specialist technical groups exist, there is implicit willingness to at least consider including the broader value of vaccines beyond what is captured in other HTA processes. These groups also provide greater technical capacity than is likely to exist when HTA is carried out by non-specialists.

  • The use of modelling to extrapolate from evidence on the value of vaccines. HTA bodies in some countries have greater ability to use models to extrapolate from quantitative data, and greater willingness to tolerate the uncertainty associated with this. For example, experts in the US, UK and Sweden perceived the existing modelling ability to be advanced enough to tolerate the uncertainty associated with this type of modelling, whereas representatives for Japan, Germany and Canada were less confident about the existing modelling ability. Differences also exist in the willingness to use data from other countries as inputs, and the ability and willingness to estimate long-term (future) effects on real-world evidence.

  • The use of qualitative decision-making processes. HTA bodies in some countries (e.g. Canada) have greater willingness and ability to consider value elements for which there exists limited quantitative evidence by incorporating them into qualitative or deliberative decision-making processes (as opposed to directly into an evaluation of cost or clinical effectiveness).

  • The decision makers’ perspective. HTA in Belgium and the UK is carried out from the perspective of the healthcare payer, meaning that productivity effects of vaccines are not considered. This creates a ‘hurdle’ in the sense that, without willingness to change the perspective (which in most countries would affect all healthcare technologies, not only vaccines), this hurdle cannot be overcome. To a lesser extent, a similar hurdle may apply to the consideration of carers’ health effects. We note that a change in the evaluation perspective should be supported by country-specific evidence of societal preferences for outcomes other than health improvements. Investigating this was beyond the scope of this study.

  • The existence of a separate budget for vaccines. In Belgium, vaccines are funded from earmarked vaccines budgets at the regional level, and therefore do not have to ‘compete’ with other health technologies. In Canada and Italy, they are funded from budgets for prevention and public health interventions. Although there is no central budget for healthcare technologies in the US, vaccines for vulnerable children may be funded on the advice of the Advisory Committee on Immunisation Practices without the need for Congressional approval [18]. There is no such dedicated budget in the other countries in our study.

3.3 Prioritisation of Value Elements for Broader Value Assessment

The ranking of value elements and specific gaps are shown in Table 3. The expert views are summarised as follows:

  • ‘Broad’ cost offsets at the community level (Priority 1, P1) are not comprehensively considered. Overall, HTA bodies in all countries studied are willing and able to consider cost offsets. However, evidence of broad cost offsets should be improved to ensure that value is consistently recognised.

  • Effects on carer’s health (Priority 2, P2) are not, or not consistently, considered. The ability to include effects on carer’s health is generally available, but the willingness to do so has not been established in all countries, and the evidentiary standards could be improved.

  • Transmission value (Priority 3, P3) is not consistently considered in all countries. Willingness to model transmission value is generally available. While ability may improve through an effort to standardise methods for advanced transmission modelling approaches, the availability of good quality evidence is currently the main hurdle.

  • Effects on AMR (Priority 4, P4) are rarely considered. Many countries have explicitly expressed their willingness to consider AMR effects given the related public health risks. Research on methods and evidence to quantify AMR effects is ongoing.

  • Macroeconomic effects (Priority 5, P5) are rarely considered. Consideration of macroeconomic effects requires rethinking many aspects of the value assessment approach of vaccines, as well as researching suitable evidence. So far (i.e. prior to the Covid-19 pandemic), and in the higher-income countries under study, these efforts have been limited by the lack of recent experience with infectious diseases having substantial macroeconomic impact.

Table 3 Prioritisation of value elements for including in broader value assessments
Full size table

3.4 Expert-Informed Recommendations to Broader Value Recognition of Vaccines

Based on the challenges identified for each prioritised value element, experts recommended the following approaches for achieving a broader value assessment of vaccines. While these recommendations were developed in consideration of the current gaps in the value assessment of vaccines, they note these should be pursued with the objective of improving HTA of all health technologies potentially generating broader value on these dimensions.

3.4.1 Willingness

Stimulate decision makers’ and the public’s awareness of the significant impact on carers’ health (P2) and macroeconomic effects (P5) that vaccines could help prevent by leveraging the global experience with COVID-19, and further develop metrics and models to quantify this. In addition, and to illustrate that macroeconomic impact is not unique to COVID-19 vaccine-preventable disease, develop ‘case studies’ demonstrating the impact of other vaccine-preventable diseases on carer’s health and macroeconomic effects.

Effects on AMR are not unique to vaccines and can accrue from various health technologies such as antibiotics. In the countries where willingness to consider this value element is currently missing, the issue should be addressed by promoting a broader discussion around the role of HTA in rewarding the incremental impact of preventing or mitigating AMR (P4) in all technologies expected to do so.

3.4.2 Ability

Short- and long-term adaptations of the approaches for assessing vaccines should be considered when willingness to consider AMR (P4) and macroeconomic effects (P5) exists. In the short term, where the quality of available evidence may not be sufficient to quantify the impact of vaccines on AMR and macroeconomic effects, decision makers may consider aiding resource allocation decisions with qualitative methods/judgements (e.g. multi-criteria decision marking). In the long term, consideration of non-health effects (e.g. macroeconomic effects) may require a permanent change of the approaches to assess vaccines, either through an adaptation of traditional methods (e.g. differential cost-effectiveness thresholds) or adoption of new ones (e.g. macroeconomic models).

3.4.3 Evidence

Experts recommend targeting the collection of evidence of broad cost offsets (P1) and carer’s health (P2) based on the disease characteristics (e.g. high infectiousness) and the vaccine target population. Also, more effort is needed to generate and maintain high-quality evidence of transmission value and effects on AMR. This requires (1) continuation of research that aims to generate evidence on infection dynamics to estimate the impact of vaccines on the development of herd immunity (P3) and of AMR (P4); and (2) strengthening national surveillance systems of infection transmissions (P3) and of resistant infection spread (P4).

4 Discussion

This study presents a literature-based framework for the broader value assessment of vaccines, identifies the gap between the broader value elements relevant for vaccines and those recognised in HTA of vaccines in nine developed markets, and generates expert-informed recommendations as to how key hurdles for broader value recognition could be overcome.

Key findings are that about half of the broader value elements relevant to vaccines are not (consistently) considered in the HTA processes of multiple higher-income countries, and that five priority areas for broader value recognition are (1) more comprehensive cost offsets within the health care system, (2) impact on carer quality of life, (3) transmission value, (4) prevention of antimicrobial resistance and (5) macroeconomic effects.

Based on the key challenges identified to appropriately consider these value elements, experts recommend a three-pronged approach to achieve a broader recognition of the value of vaccines, focusing on

  1. 1.

    Evidence: proactively steer the generation of high-quality evidence to quantify the broader value of vaccines to society;

  2. 2.

    Ability: leverage and further develop the available methodological and analytic expertise to appropriately recognise the broad value of vaccines within HTA processes;

  3. 3.

    Willingness: stimulate stakeholder engagement and buy-in to change the status quo and move towards more transparent and comprehensive value assessment processes for vaccines globally.

Experts also commented that particularly where both willingness and evidence hurdles exist, these may be most effectively overcome simultaneously, rather than sequentially, as efforts to improve the available evidence base around the impact of vaccines may also generate willingness on the decision maker’s side. For example, an explicit statement of willingness and commitment by the decision maker to consider such evidence, and an open dialogue with manufacturers on what the evidence should look like, may incentivise the development of further technical/analytic expertise (ability) as well as the evidence collection itself.

This study built on the increasing academic consensus that vaccines generate value beyond the dimensions typically covered within HTA and the wider decision-making processes they support. Both the International Society for Pharmacoeconomics and Outcomes Research (ISPOR) and the WHO have previously published guidelines on HTA of vaccines that explicitly advise incorporating broader value [8, 9]. Before that, in 2016, a convening of experts from the European vaccine economics community organised by the Robert Koch Institute developed a similar consensus framework intended to support the development of national guidelines in Europe [19]. The results of our study are in line with these three publications, arguing for comprehensive consideration of the narrow and broad effects of vaccines on both health and economic outcomes.

Whereas the previous studies recognise that the choice of whether to incorporate value elements such as social equity, productivity costs and macroeconomic effects ultimately depends on the objective function of the decision maker, our study analysed specific gaps in various countries and aimed to generate expert-informed recommendations that not only focus on current HTA guidelines and methods but also on the evidence, ability and willingness required from stakeholders to effectively move towards such broader value recognition.

As with all studies, ours has its limitations. First, rather than aiming to be fully comprehensive, the study was based on a targeted literature research and a selection of ten experts to cover nine countries. A broader stakeholder consultation was beyond the scope of this research and may provide further insight into stakeholders’ opinions and suggestions for achieving broader value recognition of vaccines. While our analysis of vaccine HTA practices in the nine target markets was based on an in-depth review of HTA guidelines, and their use in practice was validated by local experts, we did not review specific vaccine assessments to validate the practical implementation of these guidelines. Second, the expert round table happened to fall in the first wave of the COVID-19 pandemic, and as a result, this meeting was transformed from an in-person to a virtual 2-day meeting and the discussion therefore in part reflects participants’ experiences and thinking on the broader value of vaccines amidst a pandemic. Finally, the study results are reflective of higher-income countries with relatively developed HTA processes and cannot necessarily be generalised beyond such countries.

5 Conclusion

Broader value elements relevant to vaccines are not (consistently) considered in the HTA processes of multiple higher-income countries. Priority areas for HTA improvement are more consistent and comprehensive consideration of (1) broader cost offsets within the health care system, (2) impact on carer quality of life, (3) transmission value, (4) prevention of AMR, and (5) macroeconomic effects. The BRAVE way forward is to take a three-pronged approach including the collection of high-quality evidence, improvement of technical and analytical ability within HTA and infectious disease modelling, and engagement with all stakeholders involved to generate willingness to change.