, Volume 34, Issue 3, pp 227–244 | Cite as

Methods for Health Economic Evaluation of Vaccines and Immunization Decision Frameworks: A Consensus Framework from a European Vaccine Economics Community

  • Bernhard UltschEmail author
  • Oliver Damm
  • Philippe Beutels
  • Joke Bilcke
  • Bernd Brüggenjürgen
  • Andreas Gerber-Grote
  • Wolfgang Greiner
  • Germaine Hanquet
  • Raymond Hutubessy
  • Mark Jit
  • Mirjam Knol
  • Rüdiger von Kries
  • Alexander Kuhlmann
  • Daniel Levy-Bruhl
  • Matthias Perleth
  • Maarten Postma
  • Heini Salo
  • Uwe Siebert
  • Jürgen Wasem
  • Ole Wichmann
Open Access
Consensus Statement



Incremental cost-effectiveness and cost-utility analyses [health economic evaluations (HEEs)] of vaccines are routinely considered in decision making on immunization in various industrialized countries. While guidelines advocating more standardization of such HEEs (mainly for curative drugs) exist, several immunization-specific aspects (e.g. indirect effects or discounting approach) are still a subject of debate within the scientific community.


The objective of this study was to develop a consensus framework for HEEs of vaccines to support the development of national guidelines in Europe.


A systematic literature review was conducted to identify prevailing issues related to HEEs of vaccines. Furthermore, European experts in the field of health economics and immunization decision making were nominated and asked to select relevant aspects for discussion. Based on this, a workshop was held with these experts. Aspects on ‘mathematical modelling’, ‘health economics’ and ‘decision making’ were debated in group-work sessions (GWS) to formulate recommendations and/or—if applicable—to state ‘pros’ and ‘contras’.


A total of 13 different aspects were identified for modelling and HEE: model selection, time horizon of models, natural disease history, measures of vaccine-induced protection, duration of vaccine-induced protection, indirect effects apart from herd protection, target population, model calibration and validation, handling uncertainty, discounting, health-related quality of life, cost components, and perspectives. For decision making, there were four aspects regarding the purpose and the integration of HEEs of vaccines in decision making as well as the variation of parameters within uncertainty analyses and the reporting of results from HEEs. For each aspect, background information and an expert consensus were formulated.


There was consensus that when HEEs are used to prioritize healthcare funding, this should be done in a consistent way across all interventions, including vaccines. However, proper evaluation of vaccines implies using tools that are not commonly used for therapeutic drugs. Due to the complexity of and uncertainties around vaccination, transparency in the documentation of HEEs and during subsequent decision making is essential.


Herpes Zoster Vaccine Efficacy Expert Consensus Probabilistic Sensitivity Analysis Contact Pattern 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Key Points for Decision Makers

Health economic evaluations (HEEs) on vaccines and vaccination programmes should always be considered by decision-making bodies when considering inclusion of a new vaccine into the national programme to avoid suboptimal allocation of resources.

Proper evaluation of vaccines implies using tools that are not commonly used for therapeutic drugs in HEEs. However, vaccines should only be treated differently where they really are different (e.g. indirect effects).

Funders and decision-makers should recognize that proper and valid HEEs (of vaccines) demand time and resources.

1 Introduction

Health economic evaluations (HEEs) are widely used to evaluate new health technologies, and several guidelines of the World Health Organization (WHO) and other authorities exist and provide guidance on how to properly conduct such analyses [1, 2, 3, 4, 5, 6, 7, 8]. Furthermore, many national immunization technical advisory groups (NITAGs) and/or respective national institutions in high-income countries routinely consider results from HEEs for the evaluation of vaccines (Fig. 1 [9, 10, 11, 12, 13, 14]). However, comprehensive guidelines or frameworks on (1) how to best conduct HEEs of vaccines and (2) how to implement their findings into immunization decision making are scarce. Members of a European vaccine economics community developed a framework on how to deal with several relevant aspects in the field of HEEs of vaccines. The aim of this paper was to provide a consensus framework on how to apply HEEs to vaccines and to identify areas where further work is needed to reach harmonization. In this paper, we focused on vaccines for the prevention of infectious diseases, on aspects relevant to high-income countries, and on incremental cost-effectiveness and cost-utility analyses that serve as reference case in most European healthcare reimbursement systems.
Fig. 1

Map of Europe indicating whether results from economic evaluations of vaccines are routinely considered in recommendations, based on [9, 11, 12, 13, 14] (asterisk no response from country, country not considered, Malta and Cypress not shown)

2 Methods

Experts (authors of this manuscript) with expertise in mathematical modelling, health economics and/or immunization decision making from Europe were invited by the Robert Koch Institute (RKI, Germany) to develop a framework on methods for HEE of vaccines and immunization decision making. European experts from academia, national public health or health technology assessment bodies as well as from relevant national and international health authorities were selected by following criteria: (1) authorship of key publications and/or (2) (former) membership of a NITAG and/or (3) (former) employee of a (inter-) national decision-making body and/or (4) (former) employee of a health technology assessment body. To minimize potential and perceived conflicts of interest, experts employed by a pharmaceutical company were not invited to this meeting. However, since several of the invited experts had experience in conducting HEEs commissioned by industry or interacting with industry, e.g. during scientific debates, a comprehensive view on relevant aspects was ensured.

2.1 Systematic Literature Review and Preparation of a Workshop

After obtaining feedback from the expert group, a systematic literature review on methodologies and guidelines related to HEEs of vaccines was conducted. The databases MEDLINE and EMBASE were searched with a time horizon from 1 January 1990 to 21 February 2014 targeting five topics: ‘Vaccines and infectious diseases’, ‘economic evaluation’, ‘guidelines’, ‘methods’ and ‘decision making’. Search terms identified in study title and/or abstract within each topic were connected with an ‘OR’ (see Table 1). Three search branches were developed. In each search, branch topics, ‘Vaccines and infectious diseases’ and ‘economic evaluation’ were connected with an ‘AND’ and further connected (‘AND’) with the following:
Table 1

Systematic literature research


Vaccines and infectious diseases

Economic evaluation



Decision making

Search terms (in Title OR Abstract)

Vaccine OR Vaccination OR Vaccinate OR Vaccinating OR Vaccinated OR Immunization OR Immunisation OR Immunize OR Immunise OR Infectious disease OR Communicable disease OR Preventable disease

Cost OR Cost effectiveness OR Cost utility OR Cost benefit OR Benefit cost OR Cost saving OR Economic OR Pharmacoeconomic OR Pharmacoeconomics OR Budget impact OR Efficiency OR Efficient OR Monetary OR Financial OR ICER OR QALY

Guideline OR Guide OR Good-practice OR Good OR practice OR Good research practice OR Standards OR Standard OR Recommendation OR Recommendations OR Framework OR Frameworks OR Primer OR Consensus

Methoda OR Methodsa OR Methodological OR Decision analytic OR Decision analysis OR Decision analyses OR Modela OR Modelsa OR Modellinga OR Modelinga OR Model based OR Simulation OR Simulation OR Mathematical OR Transmission OR Dynamic OR Discounting OR Interaction OR Herd immunity OR Herd protection OR Herd effects OR Indirect effects OR Population-wide benefits OR Waning

Decision making OR Reimbursement OR Fourth hurdle OR Payer OR Pricing OR Funding OR Willingness to pay OR Threshold OR Value for money OR Social value OR Social preferences OR Public health

Search branch 1




Search branch 2





Search branch ‘methods’ (1 + 2)





Search branch ‘decision making’ (3)




aSearched in title only

  • ‘Guidelines’ (search branch 1) OR

  • ‘Methods’ (search branch 2) OR

  • ‘Decision making’ (search branch 3).

Studies were excluded on the following criteria:
  • they did not have a methodological purpose, or

  • they did not have a vaccine context, or

  • they considered exclusively non-industrialized countries.

Titles and abstracts of studies were screened independently by two reviewers. Potentially relevant studies were retrieved and assessed according to the three exclusion criteria by reviewers. Disagreements between both reviewers on exclusion of particular studies were resolved by consensus. Search branches 1 and 2 were further combined with an AND. Additionally, studies found by hand search (e.g. by the ‘snowballing technique’) were included. Finally, the identified literature was analysed for prevailing opinions and/or remaining questions.

Based on the included studies, pertinent aspects were identified and experts were asked to select the most relevant aspects in which currently required forms of HEE [mainly incremental cost-effectiveness ratio (ICER) calculations] of vaccines preventing infectious diseases differ from HEE of curative drugs. According to the experts’ feedback, aspects were selected for discussion in a 2-day workshop. With this approach of combining systematic reviews and subsequent expert consultation, a coverage of most relevant aspects was achieved, including differences between HEE of curative drugs and HEE of preventive vaccines.

2.2 Conduct of a Workshop

After an introduction and key note lectures, aspects in the area of (1) modelling methods, (2) health economics, and (3) decision making were discussed in group-work sessions (GWS). Findings from the literature review were used to guide discussions during GWS. Based on a workshop reader (available online [15]) the groups on ‘modelling methods’ and ‘health economics’ were asked to provide—where possible—concrete ‘recommendations/suggestions’ for use of a specific item per aspect and/or were asked—if applicable—to list ‘pros and contras’ as well as ‘future challenges’ of certain items within an aspect. For the ‘decision making’ GWS, a more explorative approach was chosen. Findings from GWS were then presented and discussed in plenary. Besides the identification of relevant aspects and remaining issues, the experts were asked to formulate consensus on how to handle certain aspects. Consensuses and compromises were achieved through expert discussions and voting. Where no consensus could be achieved, experts were asked to present different options or points of view.

3 Results

3.1 Identified Studies

The flow chart in Fig. 2 describes the identification process of the relevant studies. Search branch ‘methods’ (branch 1 and 2) resulted in the identification of 42 [16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57] studies. Search branch ‘decision making’ (branch 3) identified 26 studies [9, 21, 44, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80].
Fig. 2

Flow chart of the systematic literature review

3.2 Identified Aspects for Discussion

Based on the identified studies and experts’ feedback, 17 aspects were selected for discussion in the workshop. The results of expert discussion on each aspect are presented in Sects. and

3.3 Modelling Methods and Health Economics

3.3.1 Model Choice

Background [17, 18, 20, 22, 23, 24, 25, 27, 28, 30, 32, 34, 37, 42, 44, 46, 47, 54]:

There are several types of models:
  1. i.

    Static cohort models;

    1. 1.

      Decision tree,

    2. 2.

      Markov model.

  2. ii.

    Population models;

    1. a.

      Static model,

    2. b.

      Dynamic model:

      1. 1.

        Compartmental transmission dynamic models,

      2. 2.

        Agent-based models,

      3. 3.

        Discrete-event models.


Models from the category (ii-b) can intrinsically account for transmission of pathogens between individuals or population fractions that influence diseases transmission by indirect effects—dynamic models. Models from categories (i) and most of (ii) are often of a deterministic nature. Input parameters are set deterministically, and base-case results and uncertainty analyses are fully replicable. Models from category (ii-b-2 and ii-b-3) usually use a stochastic approach. For example, a study from Spain analysed the health economic effects of a seasonal influenza vaccination by using both a static and a dynamic model approach [46]. In contrast to the dynamic model’s results, the static model, neglecting indirect effects (such as herd protection), could not show that the influenza vaccination is cost saving. Hence, using a static model might underestimate a vaccine’s value [46].

Expert consensus:
  • For infectious disease modelling, the sole use of static models should always be justified. Static models can be used as a conservative estimate when there is no evidence for harm (e.g. age shifts with adverse effects) if indirect effects are ignored [4, 30, 34, 81, 82]. WHO has developed a flow chart that provides assistance when choosing an adequate type of model (Fig. 3) [4].
    Fig. 3

    Flow chart for model choice adopted from the World Health Organization [4]

  • A challenge is to handle realistic demographic predictions in models (with a long time horizon) because of migration, demographic changes and scarcity of contact studies in (special) populations.

  • Stochastic models;

  • Simulate a more realistic world.

  • Can follow an individual’s life course, which is easier for decision-makers to understand.

  • In stochastic models, the randomness is of first-order uncertainty; therefore, they provide an alternative to account for heterogeneity (if events are not rare) in subgroups as it is done in a deterministic model [81].

  • Model calibration and the probabilistic sensitivity analysis (PSA) become more challenging and computationally intensive, hence, transparency might suffer.

  • Data sources may not be accessible, as more fine-grained (non-aggregated) data are needed.

  • A remaining challenge is to find adequate ways to conduct efficient uncertainty analyses on stochastic models [83].

3.3.2 Time Horizon of Models

Background [18, 27, 34, 37]:

Outcomes of static cohort models are typically estimated over a single cohort’s lifetime or a specific time horizon in which the disease typically occurs that is set as the time horizon of the model. In dynamic population-based models, the entire population is modelled, and the time horizon of those models has three phases:
  1. 1.

    Run-in phase Dynamic models need a run-in phase to model the epidemiological conditions in the pre-vaccination era. This is important for a realistic implementation of the respective vaccination. The duration of the run-in phase impacts the model’s results.

  2. 2.

    Evaluation phase This phase starts with the implementation of the vaccination. The duration of the phase should be set in the model as long as necessary to cover indirect (positive or negative) effects caused by the vaccination in the population.

  3. 3.

    Steady-state After a certain duration of the evaluation, an epidemiological equilibrium is reached—the steady state—where the epidemiological variation terminates.

Combining model choice with time horizon, Mauskopf et al. [37] identified four categories of evaluation strategies (see Table 2). For example, a cost-effectiveness analysis of human papillomavirus (HPV) vaccination for Austria showed in sensitivity analyses that an extension of the model’s time horizon by almost three decades decreased the ICER from €50,000/life-years gained to <€0/life-years gained [84].
Table 2

Evaluation strategies of health economic models [37]

Evaluation strategy

Target population

Time horizon

Start of evaluation

Category 1

Entire population

Fixed TH (several years or decades)

From implementation of vaccination

Category 2

Entire population

1 year

From steady state

Category 3


Cohort’s lifetime or fixed TH

From implementation of vaccination

Category 4


Cohort’s lifetime or fixed TH

From steady state

TH time horizon

aOne or more cohorts

Expert consensus:
  • The time horizon of a dynamic model should last until the steady state is achieved in order to deliver valuable results. Hence, the model’s time horizon should not be defined prior to the analysis.

  • Dynamic population models should be evaluated by category 1 (Table 2). ICERs should be projected over time until their cumulative value stabilizes. However, if this takes more than two decades, cumulative ICERs should be presented for different time horizons (e.g. Kim and Goldie [34]).

  • Future research must analyse the impact of these different strategies.

  • A model usually should be run to epidemiological equilibrium, but it ideally should reproduce historical epidemiological (and demographic) values that may not be in equilibrium. There is a need for validation of pre-vaccination as well as post-vaccination epidemiology (whatever is available/applicable). However, this is often technically challenging and may suffer from lack of data.

  • A surveillance system should be implemented to monitor the impact of a vaccination programme. The results can be used to compare the real impact with the impact predicted in the model that was developed before the vaccine was implemented and to evaluate the model.

3.3.3 Natural History of Disease

Background [30, 52]:

Pathogen-specific naturally acquired immunity (including its waning) is usually an important feature in dynamic models. The way this is modelled has an influence on the results [exemplary model structures: ‘susceptible-infectious-susceptible’ (SIS), ‘susceptible-infectious-recovered’ (SIR), ‘susceptible-infectious-recovered-susceptible’ (SIRS), ‘susceptible-exposed-infectious-recovered-susceptible’ (SEIRS)]. In principle, the structure of a model should be developed on the base of the characteristics of the specific target disease, the vaccine of interest and the respective research question. However, for the same vaccine-preventable disease different compartment structures have been used in published studies. For example, a study analysing the health economic effects of seasonal influenza vaccination in Spain used an SIR approach, whereas a study considering the same disease in England used an SEIR structure [46, 85]. Particularly in health economic models for HPV vaccination, the use of an SIR and the use of an SIS structure are common, and the use of both structures can be justified [52, 86].

Expert consensus:
  • The model’s structure should represent the state of knowledge on the specific disease.

  • If uncertainty exists, structural uncertainty analysis should be performed [40, 87].

  • Better (sero-) epidemiological and immunological data are needed to inform such models.

3.3.4 Measures of Vaccine-Induced Protection

Background [16, 29, 42, 45, 54]:

Different vaccine efficacy (VE) measures can be considered in HEEs to account for protection against different outcomes such as infection, symptomatic illness/complication, and/or infectiousness.

If an infection does not cause any symptoms at all at any stage, and thereby does not require treatment and/or prophylaxis costs, there is no need to consider this in an economic assessment. For example, based on data from influenza challenge studies, Basta et al. [16] estimated absolute VEs for different endpoints. For example, vaccine efficacy for seasonal live-attenuated influenza vaccines in homologous seasons ranged between 40 and 90 % (VE for susceptibility 40 %; VE for infectiousness 50 %; VE for illness given infection 83 %; VE for infection-confirmed influenza illness 90 %).

When developing a model, the hierarchy of disease states/endpoints must be incorporated to consider the relevant type of VE.
  • There is a ‘sequential’ [targeting the first endpoint only, e.g. VE protecting against herpes zoster (HZ) only]

  • A ‘non-sequential’ [targeting all VE-relevant endpoints independently, e.g. VE protecting against HZ and also against HZ complication post-herpetic neuralgia (PHN)] approach

A study evaluating the cost effectiveness of a vaccine preventing HZ in the USA used, for instance, a sequential approach [88]. Hence, only VE against HZ was considered, and VE against the complication PHN was neglected. However, this approach was criticized by Brisson and colleagues [89, 90], who used a non-sequential approach when evaluating the cost effectiveness of an HZ vaccine in Canada. In this study, the model was informed with both VE against HZ and VE against PHN, and delivered results more in favour for the vaccine [90].

In clinical trials, two different approaches of analysis are often used to measure VE:
  • per protocol (PP) and

  • intention to treat (ITT).

PP usually produces more favourable VE results for the intervention/vaccine than ITT.

Furthermore, in models, the degree of protection and take can be distinguished.
  • The degree of protection (or leaky protection) is the percentage of (partial) protection in successfully vaccinated individuals (e.g. 100 % of vaccinated individuals have a protection of 50 %).

  • Take (or ‘all or nothing’) describes the percentage of successfully vaccinated individuals with full protection (e.g. 50 % of vaccinated individuals have a protection of 100 %).

A mathematical model, for instance, calculated the impacts of fictional HIV vaccines on seroprevalence, depending on whether the vaccine leads to a take or a degree protection. The seroprevalence proportion was, after several decades, lower when using a take protection than a degree protection [91]. This difference can impact the overall results of a health economic model [92].

Finally, a major challenge is a lack of clinical endpoints in clinical trials, especially when immunogenicity (or other surrogate of protection) is the outcome considered for licensure. The preferences, whether to use efficacy or effectiveness data in HEEs, are rather diverse in European guidelines [14].

Expert consensus:
  • The model structure should account for the type of VE measure incorporated. VE in terms of reducing susceptibility to infection is fundamentally different to VE reducing infectiousness. These different aspects of VE have a differential impact on the results. Modelling can be used to estimate unknown parameters including VE estimates by using, for example, a Bayesian framework utilizing Markov Chain Monte Carlo inference [93]. More studies assessing VE against infectiousness are warranted (e.g. challenge studies).

  • Model structure and the decision-maker’s research question determine the use of a sequential or non-sequential approach.

  • ITT data, when available, should be taken for base-case analyses and PP data for uncertainty analyses. However, the use of PP data for the base-case is sufficient when the difference between ITT and PP data is completely explained by the different proportions of susceptible individuals in the study population, since this is ideally incorporated in a model. PP data should be preferably chosen if a specific result on vaccine-dose compliance and/or completion of a vaccine course is of relevance.

  • The choice of representing VE with degree of protection versus take depends on the type of protection conferred by the vaccine of interest. When there is no evidence on whether the vaccine confers a leaky or an all-or-nothing protection, different approaches to account for vaccine efficacy (leaky or all-or-nothing or combination) should be used.

  • The quantitative relationship between immune response and the degree of vaccine-induced protection against clinical disease is often unclear. Validated surrogates can be considered if no clinical endpoints are available.

  • The impact of negative vaccine effects, both at an individual level (i.e. adverse events) and at a population level (i.e. replacement or age shift) needs to be considered. Cases of vaccine-preventable diseases and cases of adverse events are equally relevant outcomes.

  • Vaccine manufacturers currently have little incentive to collect some specific clinical data (e.g. head-to-head comparisons of different vaccine products); however, these are relevant for modelling and public health. For comparison of different studies, standardized case definitions for clinical outcomes are needed, and methods for the implementation of such indirect comparisons into models should be developed/standardized.

3.3.5 Duration of Vaccine-Induced Protection

Background [30, 52]:

The waning of vaccine-induced protection plays a major role when modelling vaccine effects. Clinical trials are often too short to generate robust data on the duration of vaccine protection. In a model, either a lifetime duration (neglecting waning) is assumed or waning is incorporated. Waning can be designed to start right after vaccination or after a delay period, and to decay in different ways (e.g. exponential, stepwise). With the example of HZ vaccination, it can be shown that different assumptions on the VE waning rate can impact the health economic results: in a study from Germany, vaccination at the age of 60 years resulted in an ICER of €21,565/quality-adjusted life-year (QALY) gained or €34,606/QALY gained assuming an annual waning rate of 1 or 20 %, respectively [94]. In this model, when vaccinating individuals at the age of 50 years, the same waning rates led to €18,486 and €43,701/QALY gained, respectively. Especially in health economic models on HPV vaccination, the waning of vaccine-induced protection is an influential determinant of overall results [52, 86, 95].

Expert consensus:
  • If vaccine waning is not well understood, then different waning scenarios should be considered in an uncertainty analysis and their impact compared. Immunological memory can be integrated.

  • The availability of detailed trial data on vaccine-induced protection and waning at the patient level would enable more rapid and less uncertain economic evaluations following the marketing of a new vaccine.

3.3.6 Indirect Effects Apart from Herd Protection

Background [18, 20, 27, 30, 37]:

Vaccination-specific negative externalities such as age shift of peak incidence, serotype replacement or impact on antibiotic resistance might be relevant. The cost effectiveness of pneumococcal vaccination, for example, is strongly influenced by the degree of serotype replacement that is expected after vaccine introduction. This influence was shown by van Hoek et al. [96] in a study on the cost effectiveness of a 13-valent pneumococcal conjugate vaccination for infants in England. Brisson and Edmunds [97, 98] illustrated that the routine infant vaccination against varicella is expected to increase the average age at infection. This age shift is caused by two effects: the cohort effect and the herd protection effect. If the disease severity increases with age at infection, this age shift negatively impacts the overall health-economic results.

Expert consensus:
  • Ecological effects such as intra-population immune boosting following exposure to a pathogen (cf. varicella-zoster virus), replacement of pathogen strains covered by a vaccine by strains not covered (e.g. pneumococcal serotypes) and antibiotic resistance should be part of uncertainty analyses whenever they are possibilities.

  • Ideally, pathogen replacement, eradication, genetic selection in host, changes in behaviour (e.g. screening uptake, risk behaviour such as unprotected sex or social mixing), weakening of maternal immunity, and using vaccination as a platform for adding other interventions should also be considered wherever relevant.

3.3.7 Target Population

Background [20, 23, 29, 30, 42, 43]:

Besides the question of whom to vaccinate (e.g. total population at a certain age versus risk groups) it is necessary to consider how populations/groups mix with each other. In dynamic models, contact patterns and the mixing matrix influence the model outcome. The literature offers data gathered by different approaches, e.g. survey-based (e.g. POLYMOD) and synthetic (social demographic data) contact matrices. For example, to understand contact patterns of infants, a study group sent contact diaries to a representative sample of mothers in the UK. Thereby, the average number, the type, and the duration of daily contacts of infants were documented. These data can serve as important input parameters for dynamic models evaluating the impact of vaccines [99]. Since observational data on contact patterns in the form of age-specific contact matrices are difficult to gather and are currently available only for few countries, Fumanelli et al. [100] presented a computational approach. Based on the simulation of a virtual society of agents based on data collected in eight European countries, contact patterns by age can be estimated for 26 European countries. This approach might serve as a valuable alternative to observational data [100]. Such synthetic contact matrices have been used by Poletti et al. [101] in a varicella-zoster virus study.

Expert consensus:
  • Survey-based mixing data are considered most adequate and should be used wherever possible. Synthetic methods need to be validated against survey data where available.

  • Future research should better measure contact patterns for children and parents, and evaluate what kind of contact is relevant. Knowledge is still lacking about how well contact patterns represent occasions for transmission for specific infections. It is therefore important to assess which subset of mixing data (e.g. contacts involving touching) provides the best fit to relevant observational data for the disease in question (e.g. seroprevalence data). Research funders need to understand that contact matrices are important for dynamic model projections, and that these differ between countries or regions.

3.3.8 Model Calibration and Validation

Background [18, 23, 34, 40, 42, 50, 51, 53, 55]:

Calibration means estimating and adjusting a model’s parameters to generate the expected outcomes observed in real life. The literature reports several approaches such as manual, random (e.g. Monte Carlo) or optimized (e.g. Nelder-Mead [102]) approaches. External validation compares the results of the model with the best available evidence. For example, Kim et al. [103] present calibration methods on how to use real-world data to develop a comprehensive natural history model of HPV. Hence, a calibrated model that fits to real-world data tends to generate more reliable results [103]. However, Basu and Galvani [104] claim that a Bayesian approach provides several advantages compared with the approaches presented by Kim et al. [103].

Expert consensus:
  • The manual calibration approach should be based on a structured process, and the algorithm should be reported. However, the random or optimized approach is considered to be more adequate [53, 105, 106, 107]. A random calibration approach may have an identifiability issue. Hence, researchers should make sure that the shape and range of the posterior distribution is plausible.

  • A plain visual validation is not considered sufficient. Instead, the use of goodness-of-fit criteria is recommended.

  • The dataset used for validation should be independent from that used for calibration (ideally even with different endpoints). An alternative is to hold back a portion of the calibration data (e.g. test/training datasets). An alternative option is a cross-model validation approach in which the same data are used on different models. Lack of data points might be a limitation.

3.3.9 Handling Uncertainty

Background [18, 22, 26, 30, 34, 37, 42, 45, 48, 54]:

Structural (or model) uncertainty can be handled by scenario analysis (i.e. presenting results for different models), model averaging or parameterizing the structural uncertainty [108, 109, 110]. Parameter uncertainty is quantified with PSA. However, PSAs are often not performed in dynamic models because they are computationally difficult (especially to include the uncertainty of parameters affecting transmission). Alternatively, parameters affecting transmission are excluded from PSA, or two models are developed: an economic sub-model including PSA and a dynamic sub-model focusing on transmission-specific issues. Furthermore, there are many vaccination-specific key aspects for uncertainty analysis, such as duration of vaccine protection, vaccination coverage in dynamic models, time horizon, boosting, contact patterns and targeted age/risk groups. For example, to consider both indirect effects caused by vaccination and PSA, Christensen et al. [111] developed two models (static and dynamic) to evaluate the potential impact of introducing vaccination against serogroup B meningococcal disease in the UK. The static model without parameters affecting the transmission was used to perform PSAs. The dynamic model including parameters affecting the transmission was used to account for indirect effects [111].

Expert consensus:
  • All identifiable sources of uncertainty should be accounted for, if not by PSA then by other analyses. The parameter distributions used in PSA need to be justified. Transparency is important because dynamic models can have many ‘deep parameters’ (i.e. parameters that are not directly observable, such as the probability of infection transmission per contact event). Calibrated parameters also need to capture information about uncertainty [87, 112].

  • Decision-makers need to understand the relevance of uncertainty analyses. PSA can help to identify future research priorities. However, uncertainty measures in calibrated parameters remain challenging.

  • Structural uncertainty should be parameterized where possible, but uncertainty in normative aspects such as perspective, vaccine price and discounting should not be analysed in PSA [87, 112]. Vaccine coverage should be varied between desirable and undesirable levels. Uncertainty in contact patterns should be parameterized wherever possible (see Table 3).
    Table 3

    Definition of uncertainty analyses based on expert consensus and literature [22, 30, 42, 55, 87, 112]

    Type of uncertainty

    Sensitivity analysis

    Scenario analysis

    Parameter uncertainty

    Methodological/normative uncertainty

    Structural/model uncertainty

    Deterministic sensitivity analysis











    • Efficacy

    • Costs

    • Transmission dynamic vs. discrete-event simulations

    • Discount rate

    • Presence of a immune state (SIS vs. SIR)

    • Pathogen strain competition and replacement

    • Coverage

    • Target age/risk group

    • Vaccine price

    NA not applicable, PSA probabilistic sensitivity analysis, SIR susceptible-infectious-recovered, SIS susceptible-infectious-susceptible

3.3.10 Discounting

Background [1, 18, 19, 25, 35, 38, 39, 41, 42, 44, 45, 49, 54, 56, 57, 113]:

Uniform discounting (using the same rates for costs and health effects) is most commonly applied in HEEs. However, this approach might considerably influence the long-term ICERs of vaccination in particular, since costs and benefits usually occur at different points of time. Differential discounting (with lower rates for health effects) is used, for example, in Belgium, the Netherlands, Poland and, until 2004, the UK, for the assessment of all health technologies, including vaccines. Differential discounting seems to be technically feasible and fairer from an ‘inter-generational’ perspective [114]. Different recommendations on discounting approaches, for example, drugs and vaccines within one country, do not exist. If lower discount rates for health effects are considered, the level of discount rate for health effects needs to be set separately. Another issue in this context is whether a constant or a changing discount rate over time is used. Constant rates are more widely used, possibly because of pragmatism and ease. In France, the first 30 years are discounted with a uniform rate of 4 %, and years thereafter at 2 %. This ‘slow’ discounting procedure is also recommended by WHO if the effects of vaccination begin only long after the intervention (e.g. vaccination against HPV) [4]. The type of decline can vary (e.g. stepwise, linear, or exponential). For example, Westra et al. [57] used nine different discounting approaches and rates in a model evaluating the cost effectiveness of the HPV vaccination in the Netherlands. Ceteris paribus, the ICERs ranged between €7,600/QALY gained and €165,400/QALY gained.

Expert opinion1:
  • The majority of, but not all, experts recommended differential discount rates for costs and effects in HHEs (exclusively in cost-utility and cost-effectiveness analysis) if the model’s time horizon is long (e.g. >20 years).

  • The discount rate of health effects could be around 50 % of the discount rate for costs. However, for a more evidence-based recommendation, empirical research has to be conducted [39].

  • Constant discount rates over time should not be applied in models with a long time horizon (e.g. >20 years) according to a majority of experts.

  • Further research on the, to date rarely applied, approach called time-shifted discounting approach is needed [39, 57, 115].

  • Since discount rates and discount approaches usually have a major impact on results of HEE of vaccines, the variation of these aspects need to be analysed (see Sect. 3.3.9) and explained to decision-makers.

3.3.11 Health-Related Quality of Life and Quality-Adjusted Life-Years as Outcome Measures

Background [20, 25, 32]:

QALYs are a common outcome measure used in cost-utility analyses and are accepted in several countries in Europe [14]. Since vaccine-preventable diseases often affect children, the impact on health-related quality of life (HR-QOL) of carers measured as QALYs is important in HEEs of vaccines. Another prevention-specific issue is utility in anticipation, measured as QALYs. Vaccinated individuals might experience a higher HR-QOL, because they feel ‘protected’ after vaccination. As counterpart, fear of adverse events measured as QALYs also needs to be mentioned. That might lead to an HR-QOL decrease, but on a rather short time horizon. For example, Weinke et al. [116] used a survey to assess the impact of HZ and PHN on the patients’ life but also on life of the family members who cared for them. Most family members (69 % children; 80 % life partners) of patients with HZ or PHN reported that caring for the patient caused a moderate to severe impact on their life. Hence, these impacts might have an effect on overall health economics results [116].

Expert consensus:
  • HR-QOL of carers should routinely be considered in uncertainty analysis in both payers’ and societal perspectives. However, input data might be scarce.

  • Utility in anticipation and fear of adverse events are also difficult to consider due to limited data. When appropriate data are available, they might be considered in uncertainty analyses.

3.3.12 Cost Components

Background [18, 32, 36]:

Even though the literature and national guidelines on (direct or indirect) cost components seem very comprehensive, some factors are vaccine specific and require attention. Traditionally, vaccines mainly protect against diseases occurring in childhood. Since sick children need care, indirect costs of carers (e.g. productivity loss) occur. Set-up costs (e.g. of vaccine campaigns) might also play an important role in immunization programmes. Depending on the disease and the target group, campaigns are needed to reach as many individuals as possible. Respective national guidelines often neglect such cost components. For example, in the model by Hornberger and Robertus [117], that evaluated the cost effectiveness of HZ vaccination in the USA, the price for the vaccine included the unit vaccine cost, a public awareness campaign, administration costs, patient travel time and time receiving vaccine as well as the cost of treating adverse events. Implementing $US500 per vaccine dose caused high ICERs of between $US280,000 and $US560,000/QALY gained [117].

Expert consensus:
  • Indirect costs of carers should be considered for both perspectives. From a payers’ perspective, they are considered as sick pay (if the payer has to cover these costs) and from a societal perspective as productivity loss.

  • If set-up costs (e.g. for campaigns) are not included in the vaccine price (i.e. promotion and distribution of a vaccine is not done by its manufacturer), they should be considered in the perspective that covers these costs.

3.3.13 Perspectives

Background [18, 19, 20, 32, 54]:

Some prevailing methodological primers recommend using the societal perspective when assessing vaccination programmes designed to improve public health. A recent systematic literature review on varicella and HZ vaccination concluded that the varicella vaccination (neglecting the impact on HZ) becomes cost saving when switching from a payer to a societal perspective [118]. Hence, indirect costs especially of carers when considering infant vaccination, tend to have a major impact on overall health economic results [118]. However, for example, if costs for vaccination are higher from a societal perspective due to co-payments, the overall results from the societal perspective can become less cost effective when compared with those from the payer perspective [119, 120].

Expert consensus:
  • A societal perspective should ideally be taken for the base-case analysis when considering infectious diseases (i.e. not for vaccines exclusively), unless this contradicts national guidelines.

3.4 Decision Making

Background [9, 24, 41, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 72, 73, 74, 75, 76, 77, 78, 79]:

3.4.1 Purposes of Health Economic Evaluations in Decision Making

From the decision-makers’ perspective, results from HEE of vaccines can be utilized to identify the most efficient vaccination strategies (e.g. targeting the total population or specific age or risk groups—technical efficiency), to support yes/no decisions for vaccine introduction, and/or to support price negotiations with manufacturers (allocative efficiency). Furthermore, the introduction of a new vaccine usually substitutes for curative treatment and/or screening measures in a healthcare system. Those substitutions are also of relevance for decision-makers and are usually considered in HEEs. Furthermore, the experts clearly stated that additional budget impact analyses (BIA) can be useful, especially when a costs of a prevention/intervention measure is extremely high and affordability is unclear.

3.4.2 Integration of Health Economic Results in Decision-Making Processes

The introduction of a new vaccine into a healthcare system affects several aspects of the system. Hence, many questions, such as acceptance, practicability or equity in access, must be addressed. Erickson et al. [62] provide a comprehensive view on all factors that could be considered in immunization decision making. A broad micro- and macro-economic view might be useful. Many but not all questions can be addressed within a health economic model, but should also be considered within a broader appraisal. However, details of a broader appraisal were not subject to this framework. Furthermore, other health economic approaches, such as cost-benefit analyses or BIAs, may also serve as a basis for decision making. However, which or how many approach(es) are considered for decision making is a rather normative question that needs to be address by relevant stakeholders at the national level. Therefore, experts concentrated in this framework on incremental cost-effectiveness and cost-utility analyses, that are the most commonly used forms of HEEs for national decision making in Europe [14]. Yet, the experts acknowledged the usefulness of broader and alternate approaches in the economic analysis of vaccines.

In general, the experts cautioned that lack of transparency and high complexity of evaluations leads to results from HEEs appearing like a black box to decision-makers. Based on a decision making continuum, experts stated pros and contras of three decision making approaches: pure threshold, multi-criteria decision analysis (MCDA), and informal judgment (Fig. 4). Decision making based on a pure threshold is transparent. Especially if the budget is very constrained, a fair use of public resources is possible and it is a useful tool for price negotiations with manufacturers. However, pure threshold does not consider clinical/epidemiological severity or budget impact. Furthermore, the choice of the level or range of the threshold sometimes seems rather arbitrary, and in some countries such willingness-to-pay thresholds are not accepted. MCDA is also transparent and applicable without a threshold. However, MCDA is very complex, needs a well-developed design and requires proportional weights of each criterion. Clearly, in terms of MCDA, more research is needed. In terms of informal judgment, experts recognized advantages such as applicability without a threshold and comprehensive summary of many parameters related to the respective vaccine and disease, including issues that are difficult to quantify (e.g. implementation issues or acceptability of a vaccine).
Fig. 4

Decision making continuum, adapted from expert-group discussion. HEE health economic evaluation, MCDA multi-criteria decision analysis

3.4.3 Key Parameters that Should be Varied in Uncertainty Analyses

The experts made clear that parameter uncertainty should be considered in PSA (see Table 3; Sect. 3.3.9). Additionally, the experts recommended including the variation of age and risk group, vaccination schedule, herd protection, booster or catch-up vaccination, delivery strategy and vaccination coverage in scenario/uncertainty analyses.

3.4.4 Vaccination-Specific Aspects of Reporting Results

Experts listed aspects that are essential to be reported in HEEs:
  • Discounted and also undiscounted results should be presented.

  • Cumulative results should be reported at various time points over a model’s construed decision horizon, including a longitudinal view up to the end of the defined time horizon of the model.

  • Results from various relevant perspectives.

  • Cost-effectiveness acceptability curves (CEACs).

  • Best- and worst-case scenarios.

  • Absolute values and ICERs for all disease-specific outcomes.

  • A report of an HEE should describe the validation/calibration process, the strength of evidence behind the input data, and should discuss the potential variation of results in uncertainty analyses.

  • The most recent questionnaire [71] assessing the credibility of a modelling study needs at least one infectious disease-specific addition: “If applicable, why was a dynamic model not used?”

4 Summary

As described in the Introduction, the aim of this paper was to provide a consensus framework on how to apply HEEs to vaccines and to identify areas where further work is needed to reach harmonization. The manuscript provides a comprehensive overview on many important aspects related to the economic evaluation of vaccines.

Experts from the field of modelling, health economics, and immunization decision making discussed 17 vaccine-specific aspects and reached consensus between modellers and decision-makers. Steered by a systematic literature review and expert opinion, this framework suggests the following:
  • In general, international standards as laid down in established guidelines should be applied and adopted to specific problems where necessary.

  • HEEs on vaccines and vaccination programmes should be considered by decision-making bodies such as NITAGs when considering inclusion of a new vaccine into the national programme to avoid inferior allocation of resources.

  • A mechanical use of a threshold without considering other criteria may not be necessary. However, information about incremental costs and incremental outcomes of relevant vaccination strategies and ICERs (with adequate comparator(s)) should be delivered to decision-making bodies.

  • Other interventions (e.g. drugs in preventive medicine) often share similar characteristics as vaccines. Vaccines should only be treated differently where they really are different (e.g. indirect effects).

  • HEEs must be objective, systematic and transparent.

  • HEEs should be as complex as necessary but as simple as possible.

  • Ideally, infectious disease models should be dynamic.

  • Models should focus on patient-relevant clinical endpoints wherever possible. But surrogates may be used if no clinical endpoints are available, preferably if they are validated. Both relying on surrogates and disregarding them is risky. The uncertainty concerning surrogates should be made clear to decision-makers.

  • Future costs and outcomes should be discounted. There are arguments for differential and over time decreasing discounting (not only for vaccines).

  • Information about costs and outcomes from the societal perspective is relevant. This perspective should ideally be reported in addition to the payer perspective.

  • A broad set of utility-generating characteristics (such as carer quality of life, utility in anticipation) may be adequate. However, more research is needed.

  • All uncertainties should be accounted for. Uncertainty analysis plays an important role.

  • Methodological problems need to be solved. However, it is not necessary to reject HEE per se because of the methodological challenges.

  • Funders and decision-makers should recognize that HEEs (of vaccines) demand time and resources.


  1. 1.

    Since an expert consensus was not reached, we used the term ‘expert opinion’.



The authors would like to thank the German Federal Ministry of Health for funding this study, and Thomas Harder, Felix Weidemann and Eva Wetzel (all RKI).


The consensus of the framework does not necessarily reflect the views of all single authors and all single authors’ institutions. Hence, the consensus cannot be considered as binding for all contributing parties.

Author contributions

BU and OD prepared the first draft of the manuscript. MPe, JW and OW added further background details to the draft. JB, MJ, GH, AGG, PB, AK, US, as well as OD, BB, HS, MP, WG, JW and BU finalized the background of the modelling method and health economic section of the manuscript and added further evidence. MK, DLB, RH, RvK, OW and MPe added further background details to the decision-making section. All authors contributed to the ‘expert consensus’ sections. We confirm that the manuscript has been approved by all named authors and that there are no other individuals who satisfied the criteria for authorship but are not listed. We further confirm that the order of authors listed in the manuscript has been approved by all of us.

Compliance with Ethical Standards


This study is part of a research project that is fully funded by the Federal Ministry of Health Germany.


All authors, Bernhard Ultsch, Oliver Damm, Philippe Beutels, Joke Bilcke, Bernd Brüggenjürgen, Andreas Gerber-Grote, Wolfgang Greiner, Germaine Hanquet, Raymond Hutubessy, Mark Jit, Mirjam Knol, Rüdiger von Kries, Alexander Kuhlmann, Daniel Levy-Bruhl, Matthias Perleth, Maarten Postma, Heini Salo, Uwe Siebert, Jürgen Wasem, and Ole Wichmann confirm that there are no known conflicts of interest associated with this publication and there has been no significant financial support for this work that could have influenced its outcome.


  1. 1.
    HAS. Choice in methods for Economic Evaluation. 2012. Accessed 18 Dec 2012.
  2. 2.
    KCE. Belgian guidelines for economic evaluations and budget impact analyses: second edition. 2012. Accessed 1 Aug 2014.
  3. 3.
    IQWiG. Allgemeine Methoden zur Bewertung von Verhältnissen zwischen Nutzen und Kosten. 2009. Accessed 16 Sept 2013.
  4. 4.
    WHO. WHO guide for standardization of economic evaluations of immunization programmes. Geneva. 2008. Accessed 6 Dec 2010.
  5. 5.
  6. 6.
    PHAC. Canadian Immunization Guide. Otawa. 2007. Accessed 27 Nov 2012.
  7. 7.
    NICE. National institution for health and clinical excellence:Guide to the methods of technology appraisal. 2008. Accessed 22 Nov 2012.
  8. 8.
    ISPOR. ISPOR Good Practices for Outcomes Research Task Forces. 2014. Accessed 19 Nov 2014.
  9. 9.
    Nohynek H, Wichmann O, D Ancona F, Gatekeepers VN. National Advisory Groups and their role in immunization policy-making processes in European countries. Clin Microbiol Infect. 2013;19(12):1096–105.Google Scholar
  10. 10.
    Ricciardi GW, Toumi M, Weil-Olivier C, Ruitenberg EJ, Danko D, Duru G, et al. Comparison of NITAG policies and working processes in selected developed countries. Vaccine. 2014. doi: 10.1016/j.vaccine.2014.09.023.Google Scholar
  11. 11.
    Takla A, Wichmann O, Carrillo-Santisteve P, Cotter S, Lévy-Bruhl D, Paradowska-Stankiewicz I, et al. Characteristics and practices of National Immunisation Technical Advisory Groups in Europe and potential for collaboration, April 2014. Evaluation (GRADE). 2015;5:6.Google Scholar
  12. 12.
    Bryson M, Duclos P, Jolly A, Bryson J. A systematic review of national immunization policy making processes. Vaccine. 2010;28(Suppl 1):A6–12. doi: 10.1016/j.vaccine.2010.02.026.PubMedCrossRefGoogle Scholar
  13. 13.
    Bryson M, Duclos P, Jolly A, Cakmak N. A global look at national Immunization Technical Advisory Groups. Vaccine. 2010;28(Suppl 1):A13–7. doi: 10.1016/j.vaccine.2009.07.089.PubMedCrossRefGoogle Scholar
  14. 14.
    ISPOR. Pharmacoeconomic Guidelines Around The World. 2015. Accessed 14 Jul 2015.
  15. 15.
    RKI. International workshop on methods for health economic evaluations of vaccines—workshop reader. 2014.;jsessionid=341320D8E4169CBE30A6938F3D4A4229.2_cid372?__blob=publicationFile. Accessed 14 July 2015.
  16. 16.
    Basta NE, Halloran ME, Matrajt L, Longini IM Jr. Estimating influenza vaccine efficacy from challenge and community-based study data. Am J Epidemiol. 2008;168(12):1343–52. doi: 10.1093/aje/kwn259.PubMedCentralPubMedCrossRefGoogle Scholar
  17. 17.
    Bauch CT, Anonychuk AM, Van Effelterre T, Pham BZ, Merid MF. Incorporating herd immunity effects into cohort models of vaccine cost-effectiveness. Med Decis Making. 2009;29(5):557–69.PubMedCrossRefGoogle Scholar
  18. 18.
    Beutels P, Edmunds WJ, Antoñanzas F, De Wit GA, Evans D, Feilden R, et al. Economic evaluation of vaccination programmes: a consensus statement focusing on viral hepatitis. PharmacoEconomics. 2002;20(1):1–7.PubMedCrossRefGoogle Scholar
  19. 19.
    Beutels P, Scuffham PA, MacIntyre CR. Funding of drugs: do vaccines warrant a different approach? Lancet Infectious diseases. 2008;8(11):727–33. doi: 10.1016/S1473-3099(08)70258-5.PubMedCrossRefGoogle Scholar
  20. 20.
    Beutels P, Van Doorslaer E, Van Damme P, Hall J. Methodological issues and new developments in the economic evaluation of vaccines. Expert Rev Vaccines. 2003;2(5):649–60.PubMedCrossRefGoogle Scholar
  21. 21.
    Black S. The role of health economic analyses in vaccine decision making. Vaccine. 2013;31(51):6046–9.PubMedCrossRefGoogle Scholar
  22. 22.
    Brisson M, Edmunds WJ. Impact of model, methodological, and parameter uncertainty in the economic analysis of vaccination programs. Med Decis Making. 2006;26(5):434–46.PubMedCrossRefGoogle Scholar
  23. 23.
    de Vries R, Kretzschmar M, Schellekens JFP, Versteegh FGA, Westra TA, Roord JJ et al. Cost-effectiveness of adolescent pertussis vaccination for the netherlands: Using an individual-based dynamic model. PLoS One. 2010;5(10):1–11.Google Scholar
  24. 24.
    Debicki D, Ferko N, Demarteau N, Gallivan S, Bauch C, Anonychuk A, et al. Comparison of detailed and succinct cohort modelling approaches in a multi-regional evaluation of cervical cancer vaccination. Vaccine. 2008;26(Suppl 5):16–28.CrossRefGoogle Scholar
  25. 25.
    Drummond M, Chevat C, Lothgren M. Do we fully understand the economic value of vaccines? Vaccine. 2007;25(32):5945–57. doi: 10.1016/j.vaccine.2007.04.070.PubMedCrossRefGoogle Scholar
  26. 26.
    Duintjer Tebbens RJ, Thompson KM, Hunink MG, Mazzuchi TA, Lewandowski D, Kurowicka D, et al. Uncertainty and sensitivity analyses of a dynamic economic evaluation model for vaccination programs. Med Decis Making. 2008;28(2):182–200.PubMedCrossRefGoogle Scholar
  27. 27.
    Edmunds WJ, Medley GF, Nokes DJ. Evaluating the cost-effectiveness of vaccination programmes: a dynamic perspective. Stat Med. 1999;18(23):3263–82.PubMedCrossRefGoogle Scholar
  28. 28.
    Ethgen O, Standaert B. Population- versus cohort-based modelling approaches. PharmacoEconomics. 2012;30(3):171–81. doi: 10.2165/11593050-000000000-00000.PubMedCrossRefGoogle Scholar
  29. 29.
    Halloran ME, Struchiner CJ, Longini IM Jr. Study designs for evaluating different efficacy and effectiveness aspects of vaccines. Am J Epidemiol. 1997;146(10):789–803.PubMedCrossRefGoogle Scholar
  30. 30.
    Jit M, Brisson M. Modelling the epidemiology of infectious diseases for decision analysis: A primer. PharmacoEconomics. 2011;29(5):371–86.PubMedCrossRefGoogle Scholar
  31. 31.
    Jit M, Choi YH, Laprise J-F, Boily M-C, Drolet M, Brisson M. Two-dose strategies for human papillomavirus vaccination: How well do they need to protect? Vaccine. 2014. doi: 10.1016/j.vaccine.2014.03.098.
  32. 32.
    Kauf TL. Methodological concerns with economic evaluations of meningococcal vaccines. PharmacoEconomics. 2010;28(6):449–61.PubMedCrossRefGoogle Scholar
  33. 33.
    Kim JJ, Goldie SJ. Health and Economic Implications of HPV Vaccination in the United States. New Engl J Med. 2008;359(8):821–32. doi: 10.1056/NEJMsa0707052.PubMedCentralPubMedCrossRefGoogle Scholar
  34. 34.
    Kim SY, Goldie SJ. Cost-effectiveness analyses of vaccination programmes : a focused review of modelling approaches. PharmacoEconomics. 2008;26(3):191–215.PubMedCrossRefGoogle Scholar
  35. 35.
    Klock RM, Brouwer WBF, Annemans LJP, Bos JM, Postma MJ. Towards a healthier discount procedure. Expert Rev Pharmacoecon Outcomes Res. 2005;5(1):59–63. doi: 10.1586/14737167.5.1.59.PubMedCrossRefGoogle Scholar
  36. 36.
    Krol M, Brouwer W. How to estimate productivity costs in economic evaluations. PharmacoEconomics. 2014;32(4):335–44.PubMedCrossRefGoogle Scholar
  37. 37.
    Mauskopf J, Talbird S, Standaert B. Categorization of methods used in cost-effectiveness analyses of vaccination programs based on outcomes from dynamic transmission models. Expert Rev Pharmacoecon Outcomes Res. 2012;12(3):357–71. doi: 10.1586/erp.12.11.PubMedCrossRefGoogle Scholar
  38. 38.
    O’Mahony JF, de Kok IM, van Rosmalen J, Habbema JD, Brouwer W, van Ballegooijen M. Practical implications of differential discounting in cost-effectiveness analyses with varying numbers of cohorts. Value Health. 2011;14(4):438–42.PubMedCrossRefGoogle Scholar
  39. 39.
    Parouty M, Le H, Krooshof D, Postma M. Differential time preferences for money and quality of life. PharmacoEconomics. 2014;32(4):411–9.Google Scholar
  40. 40.
    Philips Z, Bojke L, Sculpher M, Claxton K, Golder S. Good practice guidelines for decision-analytic modelling in health technology assessment. Pharmacoeconomics. 2006;24(4):355–71.PubMedCrossRefGoogle Scholar
  41. 41.
    Phillips KA, Holtgrave DR. Using cost-effectiveness/cost-benefit analysis to allocate health resources: a level playing field for prevention? Am J Prev Med. 1997;13(1):18–25.PubMedGoogle Scholar
  42. 42.
    Pitman R, Fisman D, Zaric GS, Postma M, Kretzschmar M, Edmunds J, et al. Dynamic transmission modeling: a report of the ISPOR-SMDM Modeling Good Research Practices Task Force Working Group-5. Med Decis Making. 2012;32(5):712–21. doi: 10.1177/0272989X12454578.PubMedCrossRefGoogle Scholar
  43. 43.
    Poletti P, Melegaro A, Ajelli M, Del Fava E, Guzzetta G, Faustini L, et al. Perspectives on the impact of varicella immunization on herpes zoster. A model-based evaluation from three European countries. PLoS One. 2013;8(4):e60732. doi: 10.1371/journal.pone.0060732.PubMedCentralPubMedCrossRefGoogle Scholar
  44. 44.
    Postma MJ. Public health economics of vaccines in the Netherlands: methodological issues and applications. J Public Health. 2008;16(4):267–73.CrossRefGoogle Scholar
  45. 45.
    Postma MJ, Westra TA, Quilici S, Largeron N. Economic evaluation of vaccines: specificities and future challenges illustrated by recent European examples. Expert Rev Vaccines. 2013;12(5):555–65.PubMedCrossRefGoogle Scholar
  46. 46.
    Pradas-Velasco R, Antoñanzas-Villar F, Martínez-Zárate MP. Dynamic modelling of infectious diseases: an application to the economic evaluation of influenza vaccination. PharmacoEconomics. 2008;26(1):45–56.PubMedCrossRefGoogle Scholar
  47. 47.
    Standaert B, Demarteau N, Talbird S, Mauskopf J. Modelling the effect of conjugate vaccines in pneumococcal disease: cohort or population models? Vaccine. 2010;28(Suppl 6):G30–8. doi: 10.1016/j.vaccine.2010.06.015.PubMedCrossRefGoogle Scholar
  48. 48.
    Szucs TD. Health economic research on vaccinations and immunisation practices—an introductory primer. Vaccine. 2005;23(17–18):2095–103.PubMedCrossRefGoogle Scholar
  49. 49.
    Tasset A, Nguyen VH, Wood S, Amazian K. Discounting: technical issues in economic evaluations of vaccination. Vaccine. 1999;17(SUPPL. 3):75–80.CrossRefGoogle Scholar
  50. 50.
    Taylor DC, Pawar V, Kruzikas D, Gilmore KE, Pandya A, Iskandar R, et al. Methods of model calibration: observations from a mathematical model of cervical cancer. Pharmacoeconomics. 2010;28(11):995–1000. doi: 10.2165/11538660-000000000-00000.PubMedCrossRefGoogle Scholar
  51. 51.
    Taylor DC, Pawar V, Kruzikas DT, Gilmore KE, Sanon M, Weinstein MC. Incorporating calibrated model parameters into sensitivity analyses: deterministic and probabilistic approaches. PharmacoEconomics. 2012;30(2):119–26.PubMedCrossRefGoogle Scholar
  52. 52.
    Van de Velde N, Brisson M, Boily MC. Understanding differences in predictions of HPV vaccine effectiveness: a comparative model-based analysis. Vaccine. 2010;28(33):5473–84. doi: 10.1016/j.vaccine.2010.05.056.PubMedCrossRefGoogle Scholar
  53. 53.
    Vanni T, Karnon J, Madan J, White RG, Edmunds WJ, Foss AM, et al. Calibrating models in economic evaluation. Pharmacoeconomics. 2011;29(1):35–49.PubMedCrossRefGoogle Scholar
  54. 54.
    Walker DG, Hutubessy R, Beutels P. WHO Guide for standardisation of economic evaluations of immunization programmes. Vaccine. 2010;28(11):2356–9. doi: 10.1016/j.vaccine.2009.06.035.PubMedCrossRefGoogle Scholar
  55. 55.
    Weinstein MC. Recent developments in decision-analytic modelling for economic evaluation. PharmacoEconomics. 2006;24(11):1043–53.PubMedCrossRefGoogle Scholar
  56. 56.
    West RR. Economic rate of discount and estimating cost benefit of viral immunisation programmes. Rev Med Virol. 1999;9(1):51–5.PubMedCrossRefGoogle Scholar
  57. 57.
    Westra TA, Parouty M, Brouwer WB, Beutels PH, Rogoza RM, Rozenbaum MH, et al. On discounting of health gains from human papillomavirus vaccination: effects of different approaches. Value Health. 2012;15(3):562–7.PubMedCrossRefGoogle Scholar
  58. 58.
    Ahmed F, Temte JL, Campos-Outcalt D, Schünemann HJ, Group AEBRW. Methods for developing evidence-based recommendations by the Advisory Committee on Immunization Practices (ACIP) of the US Centers for Disease Control and Prevention (CDC). Vaccine. 2011;29(49):9171–6.PubMedCrossRefGoogle Scholar
  59. 59.
    Burchett HE, Mounier-Jack S, Griffiths UK, Mills AJ. National decision-making on adopting new vaccines: a systematic review. Health Policy Plan. 2012;27(Suppl 2):62–76.Google Scholar
  60. 60.
    Dempsey AF, Cowan AE, Stokley S, Messonnier M, Clark SJ, Davis MM. The role of economic information in decision-making by the Advisory Committee on Immunization Practices. Vaccine. 2008;26(42):5389–92.PubMedCrossRefGoogle Scholar
  61. 61.
    Drummond M. Funding processes for new vaccines: The need for greater understanding of the economic issues. J Public Health. 2008;16(4):261–5.CrossRefGoogle Scholar
  62. 62.
    Erickson LJ, De Wals P, Farand L. An analytical framework for immunization programs in Canada. Vaccine. 2005;23(19):2470–6. doi: 10.1016/j.vaccine.2004.10.029.PubMedCrossRefGoogle Scholar
  63. 63.
    Farrant A. The fair innings argument and increasing life spans. J Med Ethics. 2009;35(1):53–6. doi: 10.1136/jme.2007.023762.PubMedCrossRefGoogle Scholar
  64. 64.
    Freed GL. The structure and function of immunization advisory committees in Western Europe. Human Vaccines. 2008;4(4):292–7.PubMedCrossRefGoogle Scholar
  65. 65.
    Goetghebeur M, Wagner M, Khoury H, Levitt R, Erickson L, Rindress D. Evidence and value: impact on DEcisionMaking—the EVIDEM framework and potential applications. BMC Health Serv Res. 2008;8(1):270.PubMedCentralPubMedCrossRefGoogle Scholar
  66. 66.
    Hall AJ. The United Kingdom Joint Committee on vaccination and immunisation. Vaccine. 2010;28(Suppl 1):54–7.CrossRefGoogle Scholar
  67. 67.
    Houweling H, Verweij M, Ruitenberg EJ. National Immunisation Programme Review Committee of the Health Council of the N. Criteria for inclusion of vaccinations in public programmes. Vaccine. 2010;28(17):2924–31.PubMedCrossRefGoogle Scholar
  68. 68.
    Hutubessy R, Henao AM, Namgyal P, Moorthy V, Hombach J. Results from evaluations of models and cost-effectiveness tools to support introduction decisions for new vaccines need critical appraisal. BMC Med. 2011;9:55.Google Scholar
  69. 69.
    Ismail SJ, Langley JM, Harris TM, Warshawsky BF, Desai S, FarhangMehr M. Canada’s National Advisory Committee on Immunization (NACI): evidence-based decision-making on vaccines and immunization. Vaccine. 2010;28(Suppl 1):58–63.CrossRefGoogle Scholar
  70. 70.
    Jacobs P. Are economic evaluations an important tool in vaccine policy decisions? Expert Rev Pharmacoecon Outcomes Res. 2011;11(5):507–11.PubMedCrossRefGoogle Scholar
  71. 71.
    Caro J, Eddy DM, Kan H, Kaltz C, Patel B, Eldessouki R, et al. Questionnaire to assess relevance and credibility of modeling studies for informing health care decision making: an ISPOR-AMCP-NPC Good Practice Task Force Report. Value Health. 2014;17(2):174–82.CrossRefGoogle Scholar
  72. 72.
    Kimman TG, Boot HJ, Berbers GA, Vermeer-de Bondt PE, Ardine de Wit G, de Melker HE. Developing a vaccination evaluation model to support evidence-based decision making on national immunization programs. Vaccine. 2006;24(22):4769–78.PubMedCrossRefGoogle Scholar
  73. 73.
    Newall AT, Reyes JF, Wood JG, McIntyre P, Menzies R, Beutels P. Economic evaluations of implemented vaccination programmes: key methodological challenges in retrospective analyses. Vaccine. 2014;32(7):759–65.PubMedCrossRefGoogle Scholar
  74. 74.
    Nohynek H. The Finnish decision-making process to recommend a new vaccine: from vaccine research to vaccination policy. J Public Health. 2008;16(4):275–80.CrossRefGoogle Scholar
  75. 75.
    Piso B, Wild C. Decision support in vaccination policies. Vaccine. 2009;27(43):5923–8. doi: 10.1016/j.vaccine.2009.07.105.PubMedCrossRefGoogle Scholar
  76. 76.
    Piso B, Zechmeister I, Geiger-Gritsch S. Criteria for vaccine introduction: results of a DELPHI discussion among international immunisation experts on a stepwise decision-making procedure. J Public Health. 2011;19(1):73–80.CrossRefGoogle Scholar
  77. 77.
    Spicher VM. The Federal Vaccination Commission in Switzerland: an officially appointed independent commission ensuring evidence-based recommendations and transparent procedures. Vaccine. 2010;28(Suppl 1):48–53.CrossRefGoogle Scholar
  78. 78.
    Thokala P, Duenas A. Multiple Criteria decision analysis for health technology assessment. Value Health. 2012;15(8):1172–81. doi: 10.1016/j.jval.2012.06.015.PubMedCrossRefGoogle Scholar
  79. 79.
    Welte R, Trotter CL, Edmunds WJ, Postma MJ, Beutels P. The role of economic evaluation in vaccine decision making: focus on meningococcal group C conjugate vaccine. PharmacoEconomics. 2005;23(9):855–74.PubMedCrossRefGoogle Scholar
  80. 80.
    Williams A. Intergenerational equity: an exploration of the ‘fair innings’ argument. Health Econ. 1997;6(2):117–32. doi: 10.1002/(SICI)1099-1050(199703)6:2<117:AID-HEC256>3.0.CO;2-B.PubMedCrossRefGoogle Scholar
  81. 81.
    Siebert U, Alagoz O, Bayoumi AM, Jahn B, Owens DK, Cohen DJ, et al. State-transition modeling: a report of the ISPOR-SMDM Modeling Good Research Practices Task Force-3. Value Health. 2012;15(6):812–20. doi: 10.1016/j.jval.2012.06.014.PubMedCrossRefGoogle Scholar
  82. 82.
    Brennan A, Chick SE, Davies R. A taxonomy of model structures for economic evaluation of health technologies. Health Econ. 2006;15(12):1295–310.PubMedCrossRefGoogle Scholar
  83. 83.
    Willem L, Stijven S, Vladislavleva E, Broeckhove J, Beutels P, Hens N. Active learning to understand infectious disease models and improve policy making. PLoS Comput Biol. 2014;10(4):e1003563.PubMedCentralPubMedCrossRefGoogle Scholar
  84. 84.
    Zechmeister I, Blasio BFd, Garnett G, Neilson AR, Siebert U. Cost-effectiveness analysis of human papillomavirus-vaccination programs to prevent cervical cancer in Austria. Vaccine. 2009;27(37):5133–41. doi: 10.1016/j.vaccine.2009.06.039.
  85. 85.
    Baguelin M, Jit M, Miller E, Edmunds WJ. Health and economic impact of the seasonal influenza vaccination programme in England. Vaccine. 2012;30(23):3459–62. doi: 10.1016/j.vaccine.2012.03.019.PubMedCrossRefGoogle Scholar
  86. 86.
    V Barnabas R, Kulasingam SL. Economic evaluations of human papillomavirus vaccines. Expert Rev Pharmacoecon Outcomes Res. 2007;7(3):251–67. doi: 10.1586/14737167.7.3.251.PubMedCrossRefGoogle Scholar
  87. 87.
    Bilcke J, Beutels P, Brisson M, Jit M. Accounting for methodological, structural, and parameter uncertainty in decision-analytic models: a practical guide. Med Decis Making. 2011;31(4):675–92. doi: 10.1177/0272989x11409240.PubMedCrossRefGoogle Scholar
  88. 88.
    Rothberg MB, Virapongse A, Smith KJ. Cost-effectiveness of a vaccine to prevent herpes zoster and postherpetic neuralgia in older adults. Clin Infect Dis. 2007;44(10):1280–8. doi: 10.1086/514342.PubMedCrossRefGoogle Scholar
  89. 89.
    Brisson M, Pellissier JM, Levin MJ. Cost-effectiveness of herpes zoster vaccine: flawed assumptions regarding efficacy against postherpetic neuralgia. Clin Infect Dis. 2007;45(11):1527–9. doi: 10.1086/523011.PubMedCrossRefGoogle Scholar
  90. 90.
    Brisson M, Pellissier JM, Camden S, Quach C, De Wals P. The potential cost-effectiveness of vaccination against herpes zoster and post-herpetic neuralgia. Human Vaccines. 2008;4(3):238–45.PubMedCrossRefGoogle Scholar
  91. 91.
    McLean AR, Blower SM. Modelling HIV vaccination. Trends Microbiol. 1995;3(12):458–63. doi: 10.1016/S0966-842X(00)89010-1.PubMedCrossRefGoogle Scholar
  92. 92.
    Metcalf CJE, Andreasen V, Bjørnstad ON, Eames K, Edmunds WJ, Funk S, et al. Seven challenges in modeling vaccine preventable diseases. Epidemics. 2015;10:11–5. doi: 10.1016/j.epidem.2014.08.004.PubMedCrossRefGoogle Scholar
  93. 93.
    Weidemann F, Dehnert M, Koch J, Wichmann O, Hohle M. Modelling the epidemiological impact of rotavirus vaccination in Germany—a Bayesian approach. Vaccine. 2014;32(40):5250–7. doi: 10.1016/j.vaccine.2014.06.090.PubMedCrossRefGoogle Scholar
  94. 94.
    Ultsch B, Weidemann F, Reinhold T, Siedler A, Krause G, Wichmann O. Health economic evaluation of vaccination strategies for the prevention of herpes zoster and postherpetic neuralgia in Germany. BMC Health Serv Res. 2013;13(1):359.PubMedCentralPubMedCrossRefGoogle Scholar
  95. 95.
    Seto K, Marra F, Raymakers A, Marra CA. The cost effectiveness of human papillomavirus vaccines: a systematic review. Drugs. 2012;72(5):715–43. doi: 10.2165/11599470-000000000-00000.PubMedCrossRefGoogle Scholar
  96. 96.
    van Hoek AJ, Choi YH, Trotter C, Miller E, Jit M. The cost-effectiveness of a 13-valent pneumococcal conjugate vaccination for infants in England. Vaccine. 2012. doi: 10.1016/j.vaccine.2012.10.017.Google Scholar
  97. 97.
    Brisson M, Edmunds W. Economic evaluation of vaccination programs: the impact of herd-immunity. Med Decis Making. 2003;23(1):76–82.PubMedCrossRefGoogle Scholar
  98. 98.
    Brisson M, Edmunds WJ. The cost-effectiveness of varicella vaccination in Canada. Vaccine. 2002;20(7–8):1113–25.PubMedCrossRefGoogle Scholar
  99. 99.
    van Hoek AJ, Andrews N, Campbell H, Amirthalingam G, Edmunds WJ, Miller E. The social life of infants in the context of infectious disease transmission; social contacts and mixing patterns of the very young. PloS One. 2013;8(10):e76180. doi: 10.1371/journal.pone.0076180.PubMedCentralPubMedCrossRefGoogle Scholar
  100. 100.
    Fumanelli L, Ajelli M, Manfredi P, Vespignani A, Merler S. Inferring the structure of social contacts from demographic data in the analysis of infectious diseases spread. PLoS Comput Biol. 2012;8(9):e1002673. doi: 10.1371/journal.pcbi.1002673.PubMedCentralPubMedCrossRefGoogle Scholar
  101. 101.
    Poletti P, Melegaro A, Ajelli M, del Fava E, Guzzetta G, Faustini L et al. Perspectives on the impact of varicella immunization on herpes zoster. A model-based evaluation from three european countries. PLoS One. 2013;8(4):1–13.Google Scholar
  102. 102.
    Nelder JA, Mead R. A simplex method for function minimization. Comput J. 1965;7(4):308–13. doi: 10.1093/comjnl/7.4.308.CrossRefGoogle Scholar
  103. 103.
    Kim JJ, Kuntz KM, Stout NK, Mahmud S, Villa LL, Franco EL, et al. Multiparameter calibration of a natural history model of cervical cancer. Am J Epidemiol. 2007;166(2):137–50. doi: 10.1093/aje/kwm086.PubMedCrossRefGoogle Scholar
  104. 104.
    Basu S, Galvani AP. Re: “Multiparameter calibration of a natural history model of cervical cancer”. Am J Epidemiol. 2007;166(8):983. doi: 10.1093/aje/kwm240 (author reply-4).
  105. 105.
    Eddy DM, Hollingworth W, Caro JJ, Tsevat J, McDonald KM, Wong JB, et al. Model transparency and validation: a report of the ISPOR-SMDM Modeling Good Research Practices Task Force—7. Value Health. 2012;15(6):843–50. doi: 10.1016/j.jval.2012.04.012.PubMedCrossRefGoogle Scholar
  106. 106.
    Weinstein MC, O’Brien B, Hornberger J, Jackson J, Johannesson M, McCabe C, et al. Principles of good practice for decision analytic modeling in health-care evaluation: report of the ISPOR Task Force on Good Research Practices—modeling studies. Value Health. 2003;6(1):9–17.PubMedCrossRefGoogle Scholar
  107. 107.
    Stout NK, Knudsen AB, Kong CY, McMahon PM, Gazelle GS. Calibration methods used in cancer simulation models and suggested reporting guidelines. Pharmacoeconomics. 2009;27(7):533–45.PubMedCentralPubMedCrossRefGoogle Scholar
  108. 108.
    Jackson CH, Sharples LD, Thompson SG. Structural and parameter uncertainty in Bayesian cost-effectiveness models. J R Stat Soc Ser C Appl Stat. 2010;59(2):233–53. doi: 10.1111/j.1467-9876.2009.00684.x.PubMedCentralPubMedCrossRefGoogle Scholar
  109. 109.
    Jackson CH, Thompson SG, Sharples LD. Accounting for uncertainty in health economic decision models by using model averaging. J R Stat Soc Ser A. 2009;172(2):383–404. doi: 10.1111/j.1467-985X.2008.00573.x.CrossRefGoogle Scholar
  110. 110.
    Jackson CH, Bojke L, Thompson SG, Claxton K, Sharples LD. A framework for addressing structural uncertainty in decision models. Med Decis Making. 2011;31(4):662–74. doi: 10.1177/0272989X11406986.PubMedCrossRefGoogle Scholar
  111. 111.
    Christensen H, Hickman M, Edmunds WJ, Trotter CL. Introducing vaccination against serogroup B meningococcal disease: an economic and mathematical modelling study of potential impact. Vaccine. 2013;31(23):2638–46. doi: 10.1016/j.vaccine.2013.03.034.PubMedCentralPubMedCrossRefGoogle Scholar
  112. 112.
    Briggs AH, Weinstein MC, Fenwick EAL, Karnon J, Sculpher MJ, Paltiel AD. Model parameter estimation and uncertainty: a report of the ISPOR-SMDM Modeling Good Research Practices Task Force-6. Value Health. 2012;15(6):835–42. doi: 10.1016/j.jval.2012.04.014.PubMedCrossRefGoogle Scholar
  113. 113.
    Skoupá J, Annemans L, Hájek P. Health economic data requirements and availability in the european union: results of a survey among 10 European countries. Value Health Reg Issues. 2014;4:53–7.CrossRefGoogle Scholar
  114. 114.
    Jit M, Mibei W. Discounting in the evaluation of the cost-effectiveness of a vaccination programme: A critical review. Vaccine. 2015;33(32):3788–94. doi: 10.1016/j.vaccine.2015.06.084.PubMedCrossRefGoogle Scholar
  115. 115.
    Bos JM, Beutels P, Annemans L, Postma MJ. Valuing prevention through economic evaluation: some considerations regarding the choice of discount model for health effects with focus on infectious diseases. PharmacoEconomics. 2004;22(18):1171–9.PubMedCrossRefGoogle Scholar
  116. 116.
    Weinke T, Glogger A, Bertrand I, Lukas K. The societal impact of herpes zoster and postherpetic neuralgia on patients, life partners, and children of patients in Germany. Sci World J. 2014;2014:749698. doi: 10.1155/2014/749698.CrossRefGoogle Scholar
  117. 117.
    Hornberger J, Robertus K. Cost-effectiveness of a vaccine to prevent herpes zoster and postherpetic neuralgia in older adults. Ann Intern Med. 2006;145(5):317–25.PubMedCrossRefGoogle Scholar
  118. 118.
    Damm O, Ultsch B, Horn J, Mikolajczyk R, Greiner W, Wichmann O. Systematic review of models assessing the economic value of routine varicella and herpes zoster vaccination in high-income countries. BMC Public Health. 2015;15(1):1–19. doi: 10.1186/s12889-015-1861-8.CrossRefGoogle Scholar
  119. 119.
    Annemans L, Bresse X, Gobbo C, Papageorgiou M. Health economic evaluation of a vaccine for the prevention of herpes zoster (shingles) and post-herpetic neuralgia in adults in Belgium. J Med Econ. 2010;13(3):537–51.PubMedCrossRefGoogle Scholar
  120. 120.
    Szucs TD, Kressig RW, Papageorgiou M, Kempf W, Michel JP, Fendl A, et al. Economic evaluation of a vaccine for the prevention of herpes zoster and post-herpetic neuralgia in older adults in Switzerland. Hum Vaccines. 2011;7(7):749–56. doi: 10.4161/hv.7.7.15573.CrossRefGoogle Scholar

Copyright information

© The Author(s) 2015

Open AccessThis article is distributed under the terms of the Creative Commons Attribution-NonCommercial 4.0 International License (, which permits any noncommercial use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.

Authors and Affiliations

  • Bernhard Ultsch
    • 1
    Email author
  • Oliver Damm
    • 2
  • Philippe Beutels
    • 3
  • Joke Bilcke
    • 3
  • Bernd Brüggenjürgen
    • 4
  • Andreas Gerber-Grote
    • 5
  • Wolfgang Greiner
    • 2
  • Germaine Hanquet
    • 6
  • Raymond Hutubessy
    • 7
  • Mark Jit
    • 8
    • 9
  • Mirjam Knol
    • 10
  • Rüdiger von Kries
    • 11
  • Alexander Kuhlmann
    • 12
  • Daniel Levy-Bruhl
    • 13
  • Matthias Perleth
    • 14
  • Maarten Postma
    • 15
  • Heini Salo
    • 16
  • Uwe Siebert
    • 17
    • 18
  • Jürgen Wasem
    • 19
  • Ole Wichmann
    • 1
  1. 1.Department for Infectious Disease Epidemiology, Immunisation UnitRobert Koch Institute (RKI)BerlinGermany
  2. 2.Bielefeld UniversityBielefeldGermany
  3. 3.University of AntwerpAntwerpBelgium
  4. 4.Steinbeis University Berlin (SHB)BerlinGermany
  5. 5.Institute for Quality and Efficiency in Health Care (IQWiG)CologneGermany
  6. 6.Belgian Health Care Knowledge Centre (KCE)BrusselsBelgium
  7. 7.World Health Organization (WHO)GenevaSwitzerland
  8. 8.London School of Hygiene and Tropical Medicine (LSHTM)LondonUK
  9. 9.Public Health England (PHE)LondonUK
  10. 10.Centre for Infectious Disease Control (RIVM)BilthovenThe Netherlands
  11. 11.Ludwig Maximilians University (LMU)MunichGermany
  12. 12.University of HannoverHannoverGermany
  13. 13.Institut de Veille Sanitaire (InVS)Saint-Maurice CedexFrance
  14. 14.Federal Joint Committee (G-BA)BerlinGermany
  15. 15.University of GroningenGroningenThe Netherlands
  16. 16.National Institute for Health and Welfare (THL)HelsinkiFinland
  17. 17.University for Health Sciences, Medical Informatics and Technology (UMIT)Hall in TirolAustria
  18. 18.ONCOTYROL, Center for Personalized Cancer MedicineInnsbruckAustria
  19. 19.University of Duisburg-EssenEssenGermany

Personalised recommendations