Benefit–Risk Assessment of Vaccines. Part I: A Systematic Review to Identify and Describe Studies About Quantitative Benefit–Risk Models Applied to Vaccines



Understanding the balance between the benefits and risks of vaccination is essential to ensure informed and adequate public health decision making. Quantitative benefit–risk models (qBRm) represent useful tools to help decision makers with supporting benefit–risk assessment throughout the lifecycle of a medical product. However, few initiatives have been launched to harmonise qBRm approaches, specifically for vaccines.


The aim of this paper was to identify publications about qBRm applied to vaccines through a systematic literature review, and to describe their characteristics.


Medline, Scopus and Institute for Scientific Information Web of Knowledge databases were searched to identify articles in English, published from database inceptions up to December 2019. The search strategy included the combination of three key concepts: ‘benefit–risk’, ‘modelling’ and ‘vaccines’. Data extracted included the modelling context and the methodological approaches used.


Of 3172 publications screened, 48 original publications were included. Most of the selected studies were published over the past decade and focused on rotavirus (15), dengue (10) and influenza (6) vaccines. The majority (30) of studies reported analyses related to high-income countries. The methodology of the studies differed, particularly in modelling techniques, benefit–risk measures, and sensitivity analyses. The present work also pointed out a high level of variability in the quality of reporting across studies, with particular regard to input parameters and methodological approaches.


This review provides an extensive list of qBRm applied to vaccines. Discrepancies across studies were identified during our review. While the number of published qBRm studies is increasing, no reporting guidance for qBRm applied to vaccines is currently available. This may affect decision makers’ confidence in the results and their benefit–risk assessment(s); therefore, the development of such reporting guidance is highly needed.

FormalPara Key Points
Although quantitative benefit–risk models (qBRm) represent useful tools to support decision makers in assessing the benefits and risks throughout the lifecycle of a medical product, few initiatives have been launched to harmonise qBRm approaches applied to vaccines.
The aim of the present review is to provide a complete list of available studies about qBRm applied to vaccines and to describe their characteristics according to the modelling context and the methodological approaches used.
Discrepancies across studies in terms of quality of reporting and methodological approaches used were observed in our review. Thus, we advocate for the development of an operational checklist for improving the reporting in scientific articles.


Vaccination is one of the most successful public health achievements for disease prevention [1]. Compared with most other medicines, vaccines have some unique features; (i) they can be administered to large populations of healthy subjects, including children [2], and (ii) they can be introduced by health authorities as mandatory [3]. These specificities require vaccines to have a highly favourable benefit–risk profile [4]. In practice, this means that the benefits of vaccination need to be compared with its risks, in order to ensure informed and adequate public health decision making. Benefit–risk assessment (BRA) is the basis of regulatory decisions in both pre- and post-marketing review processes [5]. Although qualitative judgments have been used in BRA of health interventions, quantitative benefit–risk models (qBRm) may provide better consistency, transparency and predictability of decision making [6,7,8].

qBRm integrate evidence from multiple sources to quantify and put into perspective the benefits and risks of a health intervention. They can be simulation models (applying simulation techniques of various complexity degrees to estimate final outcomes) or non-simulation models (using a simple calculation to obtain final outcomes) [8,9,10]. They have been increasingly considered by decision makers (regulatory authorities, pharmaceutical companies, payers, guideline developers, etc.) to support the BRA of drugs and vaccines throughout their lifecycle [11,12,13]. However, although the advantages of using qBRm to weight benefits against risks are recognised by some regulatory authorities, formal guidelines are currently not available [14,15,16].

During the last decade, several initiatives have been undertaken to harmonise qBRm approaches applied to drugs through the identification, appraisal and classification of qBRm methods and the development of recommendations or frameworks to perform qBRm; for example, the Benefit–Risk Methodology Project driven by the European Medicines Agency (EMA) [9, 17], the Pharmacoepidemiological Research on Outcomes of Therapeutics (PROTECT) consortium in Europe [8, 10, 18,19,20,21], the Unified Methodologies for Benefit–Risk Assessment (UMBRA) [22] and the International Society for Pharmacoeconomics and Outcomes Research (ISPOR) Risk–Benefit Management Working Group [23].

Vaccine specificities may require specific modelling approaches, like those considering dynamic transmission and the occurrence of both individual and indirect protection through herd immunity. For this reason, similar initiatives aiming at streamlining qBRm approaches applied to vaccines have been launched, that is, the Accelerated Development of Vaccine benefit–risk Collaboration in Europe (ADVANCE) [24] and the Vaccine Monitoring Collaboration for Europe (VAC4EU) [25].

However, to date, a comprehensive review of the literature about qBRm applied to vaccines, comprising details on the modelling context and methodological approaches used, is lacking. To this end, we developed this first paper (Part I) aiming at systematically reviewing publications about qBRm applied to vaccines, and to describe their characteristics. Based on this mapping exercise and in order to ease the interpretation of future qBRm, we developed a second paper (Part II) that proposes standards in reporting qBRm applied to vaccines [26].

Materials and Methods

The definitions of technical terminologies (quantitative and qualitative BRA, qBRm) used in this article, and adapted from the glossary developed by PROTECT [8, 27], are provided in the Appendix Table 1 (see Electronic Supplementary Material [ESM]).

Literature Search and Study Selection

Medline, Scopus and the Institute for Scientific Information (ISI) Web of Knowledge databases were searched to identify articles published from database inceptions up to 31 December 2019. The search strategy combined three key concepts: (i) benefit–risk, (ii) modelling and (iii) vaccines. The search was conducted on 13 March 2020 and was limited to articles in English (see Appendix Table 2, ESM).

All citations were downloaded and imported in the reference management software EndNote version X7 (Thomson-Reuters Corp, New York, NY, USA). Duplicate citations were identified and excluded using the reference management software and manual title examination. Two reviewers (HA and NP) independently screened all titles and abstracts using predefined exclusion criteria (see Appendix Table 3, ESM). Subsequently, all publications retained after screening were assessed for eligibility by examining their full text. Disagreements between the two reviewers were resolved through discussion. A secondary manual search was performed through snowballing the reference lists of the eligible articles to identify additional articles. A grey literature search was performed through searching major public health organisation websites and using targeted search terms in Google.

Data Extraction

The following data were extracted by reviewers: (i) the general information including study publication date and the funding source; (ii) the modelling context including vaccine indication, the geographical location and the income level of the countries considered in the analyses according to the World Bank classification, the targeted population, the alternatives used for comparison with standard vaccination and the perspectives from which the intervention’s benefit and risk outcomes are evaluated; and (iii) the methodological approaches, that is, the modelling techniques, the model attributes, the benefit–risk measures, the sensitivity/scenario analyses and whether a standardised framework, a tabular representation summarising input parameters, discount rates and utility or preference information, were used or not. See Appendix Table 4 (ESM) for more details and definitions.

The data extraction template was developed based on similar initiatives about qBRm applied to drugs, particularly on the modelling context [9, 18, 28, 29] and the methodological approaches [9, 10, 17, 21, 23], and was adapted to consider vaccine specificities (i.e. dynamic modelling technique, herd immunity and waning effect attributes) [24, 30, 31].

Two reviewers (HA and KB) independently extracted data from the selected articles. Disagreements were resolved through discussion. The authors of the papers selected for data extraction were not contacted to provide additional information.

Data Extraction

Tables 1 and 2 describe the general information, the modelling context and the methodological approaches of each study. Table 3 summarises the distribution of methodological approaches depending on the use of a simulation model or non-simulation model.

Table 1 Characteristics of the general information and the modelling context used in quantitative benefit–risk models applied to vaccines
Table 2 Characteristics of the methodological approaches used in quantitative benefit–risk models applied to vaccines
Table 3 Distribution of the methodological approaches stratified by use of a simulation or non-simulation model

Distribution of the number of publications according to the source of funding and the modelling techniques used over time is described in Fig. 2 and Appendix Fig. 1 (see ESM).


Selection of Studies

The literature search retrieved 4917 potentially relevant publications, of which 1745 were duplicates and hence excluded. Of the remaining 3172 publications screened, 3081 were excluded due to lack of relevance, as determined by evaluation of their title and abstract. The full texts of the remaining 91 publications were extensively evaluated and discussed, leading to the exclusion of 52 articles. Reasons for exclusion were repetitive data from original studies (2), non-relevant publication type (14), non-qBRm studies (35) and no access to the full-text article (1). Nine additional publications were included through snowballing the reference lists. A total of 48 publications [32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79] were retained for data extraction (Fig. 1). Tables listing the publications and reasons for exclusion and inclusion after assessment of the eligibility of the full texts are provided in the ESM (See Appendix Tables 5 and 6).

Fig. 1

PRISMA flow diagram

General Information

A temporal trend analysis of the selected articles according to their publication date showed that the number of publications focusing on vaccine qBRm was limited before the year 2000 and increased from 2008 onwards, with an average of three publications per year (Fig. 2). Ten studies disclosed funding by pharmaceutical companies.

Fig. 2

Distribution of the number of publications according to the source of funding over time

Modelling Context of Selected Studies

Most of the vaccines included in the selected publications were targeting viruses (40). Two-third of the studies were performed for three viruses (rotavirus (15), dengue (10) and influenza (6)). By contrast, other indications were only supported by a single publication (e.g. Human Papillomavirus or Hepatitis B virus) (Table 1).

Most countries considered in the analyses were high-income countries (30) and the majority were located in the United States and in Europe (Table 1).

In most studies (45), the targeted population analysed in the selected studies was clearly specified and most referred to paediatric populations (38). Others focused on the general population without any further specification (3) or were conducted in several populations (e.g. paediatric and adult populations or general and high-risk populations) (4).

Most studies compared standard vaccination versus absence of vaccination (30), and only a few compared different vaccines of the same class (3). In some cases (15), the vaccine was compared with several alternatives (e.g. standard vaccination versus absence of vaccination and standard vaccination versus vaccination with extended indication).

A large proportion of studies described the perspectives at a population level only (40), compared with individual (3) or both levels (5).

Methodological Approaches Used in Selected Studies

Few studies (5) used a structured approach or tool that facilitates the development of the qBRm (Tables 2 and 3). Among them, the multi-criteria decision analysis (MCDA) was the main standardised framework used (3) [34, 60, 61].

Studies included in the review used non-simulation (16) or simulation models (32) to estimate final outcomes. Among the simulation models, the most common types were the dynamic model (12), followed by the decision tree (5), Markov model (4) and microsimulation (1). Of note, one-third (10) of the simulation models did not specify the modelling technique used. We did not observe a temporal trend of modelling techniques (non-simulation versus simulation) (See Appendix Fig. 1 in the ESM).

Most of the qBRm included were static (36), deterministic (31), did not consider herd immunity and waning effect (35), assumed a closed model (i.e. the model does not allow new individual entrances over time) (47) and stayed at an aggregate level (i.e. no tracking of individuals’ behaviour) (44) (Tables 2 and 3).

Almost two-thirds (29) of the studies used a tabular representation to list and summarise all input parameters used in the qBRm.

Few studies (5) used discount rates to reduce the value of benefit and risk outcomes over time.

Health state preference values (also called utilities) are used to represent the strength of individuals’ preferences for different health states. In some studies (8), preference elicitation techniques were performed to weight benefit and risk outcomes. In these studies, participants were interviewed to obtain preference scores by different methods: standard gamble (1), time trade-off (1) and the Index of Well-Being (1); the other studies (5) referred to an expert panel for eliciting or weighing data.

Regarding the benefit–risk measures, one-third of the studies (16) used measures to estimate benefits and risks separately (single indices), followed by composite measures to integrate benefits and risks (trade-off indices) (13) and the remaining ones (19) used both (Tables 2 and 3). The most commonly used single indices were the impact numbers (34) and the number needed to vaccinate/to harm (NNV/NNH) (8). Impact numbers measure the number of events prevented by and/or cases attributable to vaccination within a given population. NNV and NNH provide the number of patients who need to be vaccinated for preventing or inducing one case of the event of interest. The most commonly used trade-off indices were the benefit–risk difference (or net health benefit [NHB]) (18) and the benefit–risk ratio (BRR) (13). NHB estimates the difference between the total various benefits and the total various risks of a specified vaccination. BRR divides a single benefit by a single risk. BRR and NHB expressed all outcomes in the same metric.

Almost all included studies (47) used a visual representation of their quantitative benefit–risk results, notably tables (34), line graphics (23), bar charts (8), scatter plots (6), box plots (5) and area graphs (2).

Most studies (44) conducted sensitivity and/or scenario analyses including one-way deterministic sensitivity analyses (13), multiple-way deterministic sensitivity analyses (4), probabilistic sensitivity analyses (20) and scenario analyses (37) (Tables 2 and 3). Most of the studies using non-simulation models only reported scenario analyses (12). By contrast, most studies with simulation models reported sensitivity (deterministic and/or probabilistic) analyses (25).


To our knowledge, the present study is the first that systematically reviewed published information about qBRm applied to vaccines. This study was not intended to appraise the qBRm quality but aimed to build a comprehensive repository of the available publications about qBRm applied to vaccines. The present work is the first of two companion papers. Based on this mapping exercise (i.e. Part I), we developed a second paper that proposes standards of reporting for qBRm applied to vaccines (Part II) [26].

General Information

Since the late 1990s, the development of qBRm has been progressively encouraged by regulatory authorities [14,15,16, 80, 81]. Although the first qBRm applied to vaccines identified in our systematic literature review was published in 1976 [75], most studies were published over the last decade. The number of those developed by pharmaceutical companies has increased over time. This trend may be explained by the recent launch of international consortiums [18, 24, 25] facilitating collaboration between public and private partners to develop tools for assessing benefit–risk, particularly for vaccines.

Modelling Context

Among the 48 original publications included, the majority targeted the rotavirus, dengue and influenza vaccines. The focus on rotavirus vaccines might be explained by their association with a transient increased risk of intussusception following administration [82]. The high number of publications about the recent dengue vaccine is likely to be caused by the increased risk of ‘secondary-like’ infection in populations [83]; in this context, qBRm were performed to simulate optimal vaccine strategies according to different transmission intensity settings and serological status impacts. For influenza vaccines, the variable nature of the influenza viruses requiring the undertaking of a new vaccine version each fall is a likely explanation [84]. By contrast, qBRm were not performed for some vaccines, notably older ones, such as those for diphtheria or poliomyelitis, probably because the recent adoption of qBRm and the benefit–risk balance of older vaccines is no longer questioned.

Most qBRm targeted high-income countries. Their results may thereby not be representative for countries with lower income. Estimation of benefits and risks of vaccination may indeed vary according to the epidemiology of the diseases and the healthcare system. Those geographical and socio-economical specificities should be considered when interpreting qBRm results. Furthermore, the development of qBRm applied to vaccines in different settings should be encouraged for improving and adapting the decision-making process.

Methodological Approaches

The present work pointed out the high level of variability in the quality of reporting across studies. First, it was not straightforward to identify input parameters used for each qBRm. One-third of the studies did not use a tabular representation to list and define all input parameters. Second, information on modelling techniques and benefit–risk measures were not sufficiently outlined, which added complexity in extracting the data summarised in Tables 2 and 3. Incomplete and inadequate qBRm reporting may affect their result interpretation, their reproducibility, and potentially, any related decision-making process (e.g. the approval and/or recommendation of a new vaccine or its indications). Developing reporting guidance is therefore the key to helping ensure transparency and reproducibility of qBRm applied to vaccines, but is currently lacking.

One of the main findings of this review was the heterogeneity observed across studies in terms of their methodology. First, two-thirds of the studies used simulation models to estimate final outcomes. Among the simulation models, the number of static and dynamic models was similar. Dynamic models are particularly useful when targeting infectious diseases because they allow taking into account the evolution of certain factors that modulate the risk of infection over time, such as immunity [85]. Of note, except for two publications, all dynamic transmission models were performed for the dengue vaccine. The low number of transmission dynamic models used for other vaccines may be related to their high level of complexity, requiring more advanced programming skills [31]. Second, a few studies incorporated health state preference values, offering the advantage that it represents the strength of individuals’ preferences for different health states, thereby moving beyond the narrow biomedical model for evaluative research [86]. Various techniques were used to obtain health state preference value. Third, benefit–risk measures considered also differed from one study to another. Single, trade-off and both indices were each used in one-third of the selected studies. Single indices are valued for their intuitive interpretation by clinicians and decision makers. Conversely, by using a commensurable score, trade-off indices confer the advantage of providing a straightforward comparison of benefit and risk outcomes, thereby facilitating the BRA. Nevertheless, trade-off indices may be less intuitive and can only be performed when both benefit and risk outcomes use the same metric. Fourth, almost all studies performed uncertainty analyses by varying one or more key parameter values, using various techniques such as scenario (18), sensitivity (7) and both (18). The questions on benefit–risk decision are heterogeneous, due to differences like targeted population, benefit and risk outcomes and importance of indirect effects. Thus, there is no one-size-fits-all approach, as proved by the high level of methodological heterogeneity among the selected publications. This observation highlights the richness of methodological approaches available to perform qBRm applied to vaccines; each of them with their own strengths and weaknesses. The choice of methodological approaches depends on several considerations like the analyst’s technical skills, the required model complexity, the question at hand and the nature of the decision matter, the natural history and features of the particular infectious disease and the data available to parameterise and calibrate the model [87, 88]. Consequently, it is crucial to develop methodological guidance to help researchers use the best methodological approach suited to specific situations. Furthermore, the variety of methodological approaches available reinforces the need for specific standards in reporting.

The present review has some limitations. Even though we aimed to be comprehensive, we cannot guarantee that all relevant studies were identified, mainly because there are no specific keywords for qBRm. Furthermore, grey literature is not identifiable through conventional methods of bibliographic search, and consequently, some relevant websites or other source(s) of information might be missing. However, various search strategies were tested before selecting the one that allowed us to retrieve the largest number of relevant publications, while keeping the number of articles to be reviewed within acceptable limits.


This study is the first that systematically reviewed published information about qBRm applied to vaccines. Our findings showed (i) large differences in terms of methodology used to perform qBRm applied to vaccines and (ii) a noticeable heterogeneity across studies in terms of quality of reporting. Since the number of published qBRm studies is increasing over time and given that paucity of formal guidance for qBRm applied to vaccines may affect the confidence in these models’ results, the development of specific standards of reporting should help to ensure the transparency and reproducibility. Part II of this series of companion papers proposes an operational checklist aimed to improve the reporting of qBRm applied to vaccines in scientific articles.


  1. 1.

    WHO. Immunization. World Health Organization. 2017. Accessed 13 Mar 2020.

  2. 2.

    European Medicines Agency (EMA). Guideline on good pharmacovigilance practices (GVP). Vaccines for prophylaxis against infectious diseases. 2013. Accessed 13 Mar 2020.

  3. 3.

    Larson HJ, Cooper LZ, Eskola J, Katz SL, Ratzan S. Addressing the vaccine confidence gap. Lancet (London, England). 2011;378(9790):526–35.

    Article  Google Scholar 

  4. 4.

    Ahmed SS, Plotkin SA, Black S, Coffman RL. Assessing the safety of adjuvanted vaccines. Sci Transl Med. 2011;3(93):932.

    CAS  Article  Google Scholar 

  5. 5.

    Food and Drug Administration (FDA). Structured approach to benefit-risk assessment in drug regulatory decision-making. Draft PDUFA V Implementation Plan. 2013-2017.

  6. 6.

    Raju GK, Gurumurthi K, Domike R. Benefit-risk analysis for decision-making: an approach. Clin Pharmacol Ther. 2016;100(6):654–71.

    CAS  Article  PubMed  Google Scholar 

  7. 7.

    Luteijn JM, White BC, Gunnlaugsdottir H, Holm F, Kalogeras N, Leino O, et al. State of the art in benefit-risk analysis: medicines. Food Chem Toxicol. 2012;50(1):26–32.

    CAS  Article  PubMed  Google Scholar 

  8. 8.

    Mt-Isa S, Ouwens M, Robert V, Gebel M, Schacht A, Hirsch I. Structured Benefit-risk assessment: a review of key publications and initiatives on frameworks and methodologies. Pharm Stat. 2016;15(4):324–32.

    Article  PubMed  Google Scholar 

  9. 9.

    European Medicines Agency (EMA). Benefit-risk methodology project work package 2 report: Applicability of current tools and processes for regulatory benefit-risk assessment. London: European Medicines Agency. 2010. Accessed 31 Aug 2010.

  10. 10.

    Mt-Isa S, Hallgreen CE, Wang N, Callreus T, Genov G, Hirsch I, et al. Balancing benefit and risk of medicines: a systematic review and classification of available methodologies. Pharmacoepidemiol Drug Saf. 2014;23(7):667–78.

    Article  PubMed  Google Scholar 

  11. 11.

    Greenberg M, Simondon F, Saadatian-Elahi M. Perspectives on benefit-risk decision-making in vaccinology: conference report. Human Vacc Immunother. 2016;12(1):176–81.

    CAS  Article  Google Scholar 

  12. 12.

    Schosser R. Risk/benefit evaluation of drugs: the role of the pharmaceutical industry in Germany. Eur Surg Res. 2002;34(1–2):203–7.

    CAS  Article  PubMed  Google Scholar 

  13. 13.

    Bonhoeffer J, Black S, Izurieta H, Zuber P, Sturkenboom M. Current status and future directions of post-marketing vaccine safety monitoring with focus on USA and Europe. Biologicals. 2012;40(5):393–7.

    Article  PubMed  Google Scholar 

  14. 14.

    European Medicines Agency. ICH guideline E2C(R2) on periodic benefit-risk evaluation report (PBRER). EMA/CHMP/ICH/544553/1998. 2013.

  15. 15.

    International Conference of Harmonization (ICH) harmonised guideline. Revision of M4E guideline on enhancing the format and structure of benefit-risk information in ICH Efficacy - M4E(R2). 2016. Accessed 13 Mar 2020.

  16. 16.

    U.S. Department of Health and Human Services; Food and Drug Administration; Center for Drug Evaluation and Research (CDER); Center for Biologics Evaluation and Research (CBER). E2C(R2) Periodic Benefit-Risk Evaluation Report (PBRER). Guidance for Industry 2016. Accessed 13 Mar 2020.

  17. 17.

    European Medicines Agency (EMA). Benefit-risk methodology project work package 3 report: Field tests. London: European Medicines Agency. 2011. Accessed 13 Mar 2020.

  18. 18.

    Pharmacoepidemiological Research on Outcomes of Therapeutics by a European Consortium (PROTECT) Benefit Risk website. Accessed 13 Mar 2020.

  19. 19.

    Hallgreen CE, Mt-Isa S, Lieftucht A, Phillips LD, Hughes D, Talbot S, et al. Literature review of visual representation of the results of benefit-risk assessments of medicinal products. Pharmacoepidemiol Drug Saf. 2016;25(3):238–50.

    Article  PubMed  Google Scholar 

  20. 20.

    Hallgreen CE, van den Ham HA, Mt-Isa S, Ashworth S, Hermann R, Hobbiger S, et al. Benefit-risk assessment in a post-market setting: a case study integrating real-life experience into benefit-risk methodology. Pharmacoepidemiol Drug Saf. 2014;23(9):974–83.

    Article  PubMed  Google Scholar 

  21. 21.

    Hughes D, Waddingham E, Mt-Isa S, Goginsky A, Chan E, Downey GF, et al. Recommendations for benefit-risk assessment methodologies and visual representations. Pharmacoepidemiol Drug Saf. 2016;25(3):251–62.

    Article  PubMed  Google Scholar 

  22. 22.

    McAuslane N, Leong J, Liberti L, Walker S. The benefit-risk assessment of medicines: experience of a consortium of medium-sized regulatory authorities. Ther Innov Regul Sci. 2017;51(5):635–44.

    Article  PubMed  Google Scholar 

  23. 23.

    Guo JJ, Pandey S, Doyle J, Bian B, Lis Y, Raisch DW. A review of quantitative risk-benefit methodologies for assessing drug safety and efficacy-report of the ISPOR risk-benefit management working group. Value Health. 2010;13(5):657–66.

    Article  PubMed  Google Scholar 

  24. 24.

    ADVANCE – Accelerated development of vaccine benefit-risk collaboration in Europe.

  25. 25.

    VAC4EU–Vaccine monitoring Collaboration for Europe. Accessed 13 Mar 2020.

  26. 26.

    Arlegui H, Bollaerts K, Bauchau V, Nachbaur G, Begaud B, Praet N. Benefit-risk assessment of vaccines. Part II: proposal towards consolidated standards of reporting quantitative benefit-risk models applied to vaccines. Drug Saf. 2020.

    Article  PubMed  PubMed Central  Google Scholar 

  27. 27.

    Pharmacoepidemiological Research on Outcomes of Therapeutics (PROTECT) consortium in Europe. Glossary. Accessed 13 Mar 2020.

  28. 28.

    Coplan PM, Noel RA, Levitan BS, Ferguson J, Mussen F. Development of a framework for enhancing the transparency, reproducibility and communication of the benefit-risk balance of medicines. Clin Pharmacol Ther. 2011;89(2):312–5.

    CAS  Article  PubMed  Google Scholar 

  29. 29.

    Nixon R, Dierig C, Mt-Isa S, Stockert I, Tong T, Kuhls S, et al. A case study using the PrOACT-URL and BRAT frameworks for structured benefit risk assessment. Biom J. 2016;58(1):8–27.

    Article  PubMed  Google Scholar 

  30. 30.

    Kim SY, Goldie SJ. Cost-effectiveness analyses of vaccination programmes: a focused review of modelling approaches. Pharmacoeconomics. 2008;26(3):191–215.

    Article  Google Scholar 

  31. 31.

    Kuntz K, Sainfort F, Butler M, Taylor B, Kulasingam S, Gregory S, et al. Decision and simulation modeling in systematic reviews. Rockville: AHRQ Methods for Effective Health Care; 2013.

    Google Scholar 

  32. 32.

    Bollaerts K, De Smedt T, Donegan K, Titievsky L, Bauchau V. Benefit-Risk Monitoring of Vaccines Using an Interactive Dashboard: a Methodological Proposal from the ADVANCE Project. Drug Saf. 2018;41(8):775–86.

    Article  PubMed  PubMed Central  Google Scholar 

  33. 33.

    Bollaerts K, de Smedt T, McGee C, Emborg HD, Villa M, Alexandridou M, et al. ADVANCE: towards near real-time monitoring of vaccination coverage, benefits and risks using European electronic health record databases. Vaccine. 2019.

    Article  PubMed  Google Scholar 

  34. 34.

    Bollaerts K, Ledent E, de Smedt T, Weibel D, Emborg HD, Danieli G, et al. ADVANCE system testing: benefit-risk analysis of a marketed vaccine using multi-criteria decision analysis and individual-level state transition modelling. Vaccine. 2019.

    Article  PubMed  Google Scholar 

  35. 35.

    Bruijning-Verhagen P, van Dongen JAP, Verberk JDM, Pijnacker R, van Gaalen RD, Klinkenberg D, et al. Updated cost-effectiveness and risk-benefit analysis of two infant rotavirus vaccination strategies in a high-income, low-endemic setting. Bmc Med. 2018.

    Article  PubMed  PubMed Central  Google Scholar 

  36. 36.

    Bruun T, Watle SSV, Tveteraas IH, Flem E. Intussusception among Norwegian children: what to expect after introduction of rotavirus vaccination? Vaccine. 2019;37(38):5717–23.

    Article  PubMed  Google Scholar 

  37. 37.

    Carlin JB, Macartney KK, Lee KJ, Quinn HE, Buttery J, Lopert R, et al. Intussusception risk and disease prevention associated with rotavirus vaccines in Australia’s National Immunization Program. Clin Infect Dis. 2013;57(10):1427–34.

    Article  PubMed  Google Scholar 

  38. 38.

    Cho BH, Clark TA, Messonnier NE, Ortega-Sanchez IR, Weintraub E, Messonnier ML. MCV vaccination in the presence of vaccine-associated Guillain-Barre Syndrome risk: a decision analysis approach. Vaccine. 2010;28(3):817–22.

    Article  PubMed  Google Scholar 

  39. 39.

    Clark A, Jit M, Andrews N, Atchison C, Edmunds WJ, Sanderson C. Evaluating the potential risks and benefits of infant rotavirus vaccination in England. Vaccine. 2014;32(29):3604–10.

    Article  PubMed  Google Scholar 

  40. 40.

    Clark A, Tate J, Parashar U, Jit M, Hasso-Agopsowicz M, Henschke N, et al. Mortality reduction benefits and intussusception risks of rotavirus vaccination in 135 low-income and middle-income countries: a modelling analysis of current and alternative schedules. Lancet Glob Health. 2019;7(11):E1541–52.

    Article  PubMed  PubMed Central  Google Scholar 

  41. 41.

    Clark M, Cameron DW. The benefits and risks of bacille Calmette-Guerin vaccination among infants at high risk for both tuberculosis and severe combined immunodeficiency: assessment by Markov model. BMC Pediatr. 2006;6:5.

    Article  PubMed  PubMed Central  Google Scholar 

  42. 42.

    Coudeville L, Baurin N, Olivera-Botello G. Assessment of benefits and risks associated with dengue vaccination at the individual and population levels: a dynamic modeling approach. Expert Rev Vacc. 2018;17(8):753–63.

    CAS  Article  Google Scholar 

  43. 43.

    Coudeville L, Baurin N, Vergu E. Estimation of parameters related to vaccine efficacy and dengue transmission from two large phase III studies. Vaccine. 2016;34(50):6417–25.

    CAS  Article  PubMed  Google Scholar 

  44. 44.

    Desai R, Cortese MM, Meltzer MI, Shankar M, Tate JE, Yen C, et al. Potential intussusception risk versus benefits of rotavirus vaccination in the united states. Pediatric Infect Dis J. 2013;32(1):1–7.

    Article  Google Scholar 

  45. 45.

    Desai R, Parashar UD, Lopman B, de Oliveira LH, Clark AD, Sanderson CF, et al. Potential intussusception risk versus health benefits from rotavirus vaccination in Latin America. Clin Infect Dis. 2012;54(10):1397–405.

    Article  PubMed  Google Scholar 

  46. 46.

    Ferguson NM, Rodriguez-Barraquer I, Dorigatti I, Mier-y-Teran-Romero L, Laydon DJ, Cummings DAT. Benefits and risks of the Sanofi-Pasteur dengue vaccine: modeling optimal deployment. Science (New York, NY). 2016;353(6303):1033–6.

    CAS  Article  Google Scholar 

  47. 47.

    Flasche S, Jit M, Rodriguez-Barraquer I, Coudeville L, Recker M, Koelle K, et al. The long-term safety, public health impact, and cost-effectiveness of routine vaccination with a recombinant, live-attenuated dengue vaccine (Dengvaxia): a model comparison study. PLoS Med. 2016.

    Article  PubMed  PubMed Central  Google Scholar 

  48. 48.

    Hladish TJ, Pearson CA, Chao DL, Rojas DP, Recchia GL, Gomez-Dantes H, et al. Projected Impact of Dengue Vaccination in Yucatan, Mexico. PLoS Negl Trop Dis. 2016;10(5):e0004661.

    CAS  Article  PubMed  PubMed Central  Google Scholar 

  49. 49.

    Kelly H, Carcione D, Dowse G, Effler P. Quantifying benefits and risks of vaccinating Australian children aged six months to four years with trivalent inactivated seasonal influenza vaccine in 2010. Eurosurveillance. 2010;15(37):1–4.

    Google Scholar 

  50. 50.

    Kent DL. The basics of decision analysis. J Dent Educ. 1992;56(12):791–9.

    CAS  Article  Google Scholar 

  51. 51.

    Koplan JP, Schoenbaum SC, Weinstein MC, Fraser DW. Pertussis vaccine; an analysis of benefits, risks and costs. N Engl J Med. 1979;301(17):906–11.

    CAS  Article  Google Scholar 

  52. 52.

    Lamrani A, Tubert-Bitter P, Hill C, Escolano S. A benefit–risk analysis of rotavirus vaccination, France, 2015. Eurosurveillance. 2017;22(50):28–37.

    Article  Google Scholar 

  53. 53.

    Ledent E, Arlegui H, Buyse H, Basile P, Karkada N, Praet N, et al. benefit versus risk assessment of rotavirus vaccination in France: a simulation and modeling analysis. Biodrugs. 2018;32(2):139–52.

    Article  PubMed  PubMed Central  Google Scholar 

  54. 54.

    Ledent E, Lieftucht A, Buyse H, Sugiyama K, McKenna M, Holl K. Post-marketing benefit-risk assessment of rotavirus vaccination in japan: a simulation and modelling analysis. Drug Saf. 2016;39(3):219–30.

    Article  PubMed  PubMed Central  Google Scholar 

  55. 55.

    Ledogar RJ, Fleming J, Andersson N. Knowledge synthesis of benefits and adverse effects of measles vaccination: the Lasbela balance sheet. BMC Int health Hum Rights. 2009;9(Suppl 1):S6.

    Article  PubMed  PubMed Central  Google Scholar 

  56. 56.

    Lee GM, Lebaron C, Murphy TV, Lett S, Schauer S, Lieu TA. Pertussis in adolescents and adults: should we vaccinate? Pediatrics. 2005;115(6):1675–84.

    Article  PubMed  Google Scholar 

  57. 57.

    Lourenco J, Recker M. Dengue serotype immune-interactions and their consequences for vaccine impact predictions. Epidemics. 2016;16:40–8.

    Article  PubMed  PubMed Central  Google Scholar 

  58. 58.

    Maier SB, Huang X, Massad E, Amaku M, Burattini MN, Greenhalgh D. Analysis of the optimal vaccination age for dengue in Brazil with a tetravalent dengue vaccine. Math Biosci. 2017;294:15–32.

    Article  PubMed  Google Scholar 

  59. 59.

    Manissero D, Lopalco PL, Levy-Bruhl D, Degli Atti CML, Giesecke J. Assessing the impact of different BCG vaccination strategies on severe childhood TB in low-intermediate prevalence settings. Vaccine. 2008;26(18):2253–9.

    Article  PubMed  Google Scholar 

  60. 60.

    Marcelon L, Verstraeten T, Dominiak-Felden G, Simondon F. Quantitative benefit-risk assessment by MCDA of the quadrivalent HPV vaccine for preventing anal cancer in males. Expert Rev Vacc. 2016;15(1):139–48.

    CAS  Article  Google Scholar 

  61. 61.

    Maro JC, Fryback DG, Lieu TA, Lee GM, Martin DB. Responding to vaccine safety signals during pandemic influenza: a modeling study. PLoS One. 2014;9(12):e115553.

    CAS  Article  PubMed  PubMed Central  Google Scholar 

  62. 62.

    Meltzer MI. Risks and benefits of preexposure and postexposure smallpox vaccination. Emerg Infect Dis. 2003;9(11):1363–70.

    Article  PubMed  PubMed Central  Google Scholar 

  63. 63.

    Monath TP. Review of the risks and benefits of yellow fever vaccination including some new analyses. Expert Rev Vacc. 2012;11(4):427–48.

    CAS  Article  Google Scholar 

  64. 64.

    Nokes DJ, Anderson RM. Vaccine safety versus vaccine efficacy in mass immunisation programmes. Lancet (London, England). 1991;338(8778):1309–12.

    CAS  Article  Google Scholar 

  65. 65.

    Oleksiewicz MB, Steenhard NR, Hansen JE. Modernizing stockpiles of medical countermeasures against smallpox: benefits, risks, and knowledge gaps. Am J Disaster Med. 2015;10(2):109–20.

    Article  PubMed  Google Scholar 

  66. 66.

    Onorato IM, Orenstein WA, Hinman AR, Rogers MF, Koplan JP. Immunization of asymptomatic HIV-infected children with measles vaccine: assessment of risks and benefits. Med Decis Mak. 1989;9(2):76–83.

    CAS  Article  Google Scholar 

  67. 67.

    Oster G, Weycker D, Edelsberg J, Nichol KL, Klein JO, Belshe RB. Benefits and risks of live attenuated influenza vaccine in young children. Am J Manag Care. 2010;16(9):e235–44.

    PubMed  Google Scholar 

  68. 68.

    Patel MM, Clark AD, Glass RI, Greenberg H, Tate J, Santosham M, et al. Broadening the age restriction for initiating rotavirus vaccination in regions with high rotavirus mortality: benefits of mortality reduction versus risk of fatal intussusception. Vaccine. 2009;27(22):2916–22.

    Article  PubMed  Google Scholar 

  69. 69.

    Patel MM, Clark AD, Sanderson CF, Tate J, Parashar UD. Removing the age restrictions for rotavirus vaccination: a benefit-risk modeling analysis. PLoS Med. 2012;9(10):e1001330.

    Article  PubMed  PubMed Central  Google Scholar 

  70. 70.

    Patel MM, Lopez-Collada VR, Bulhoes MM, De Oliveira LH, Bautista Marquez A, Flannery B, et al. Intussusception risk and health benefits of rotavirus vaccination in Mexico and Brazil. N Engl J Med. 2011;364(24):2283–92.

    CAS  Article  PubMed  Google Scholar 

  71. 71.

    Perkins T, Reiner RC, Ten Bosch QA, Espana G, Verma A, Liebman KA, et al. Statistical and biological uncertainties associated with vaccine efficacy estimates and their implications for dengue vaccine impact projections. bioRxiv. 2016.

    Article  Google Scholar 

  72. 72.

    Phillips LD, Fasolo B, Zafiropoulous N, Eichler HG, Ehmann F, Jekerle V, et al. Modelling the risk-benefit impact of H1N1 influenza vaccines. Eur J Pub Health. 2013;23(4):674–8.

    Article  Google Scholar 

  73. 73.

    Rodriguez-Barraquer I, Mier-y-Teran-Romero L, Schwartz IB, Burke DS, Cummings DA. Potential opportunities and perils of imperfect dengue vaccines. Vaccine. 2014;32(4):514–20.

    Article  PubMed  Google Scholar 

  74. 74.

    Rothberg M, Bennish ML, Kao JS, Wong JB. Do the benefits of varicella vaccination outweigh the long-term risks? A decision-analytic model for policymakers and pediatricians. Clin Infect Dis. 2002;34(7):885–94.

    Article  PubMed  Google Scholar 

  75. 75.

    Rouillon A, Waaler H. BCG vaccination and epidemiological situation: a decision making approach to the use of BCG. Adv Tuberc Res. 1976;19:64–126.

    CAS  PubMed  Google Scholar 

  76. 76.

    Shim E. Optimal dengue vaccination strategies of seropositive individuals. Math Biosci Eng. 2019;16(3):1171–89.

    Article  PubMed  Google Scholar 

  77. 77.

    Tosti ME, Traversa G, Bianco E, Mele A. Multiple sclerosis and vaccination against hepatitis B: analysis of risk benefit profile. Ital J Gastroenterol Hepatol. 1999;31(5):388–91.

    CAS  PubMed  Google Scholar 

  78. 78.

    Yung CF, Chan SP, Soh S, Tan A, Thoon KC. Intussusception and monovalent rotavirus vaccination in Singapore: self-controlled case series and risk-benefit study. J Pediatr. 2015;167(1):163-8.e1.

    Article  PubMed  Google Scholar 

  79. 79.

    Zalkind DL, Shachtman RH. A decision analysis approach to the swine influenza vaccination decision for an individual. Med Care. 1980;18(1):59–72.

    CAS  Article  Google Scholar 

  80. 80.

    Council for International Organizations of Medical Sciences (CIOMS). Benefit–risk balance for marketed drugs: evaluating safety signals, Geneva. 1998. Accessed 13 Mar 2020.

  81. 81.

    European Medicines Agency (EMA). Benefit-risk methodology project. 2009. Accessed 13 Mar 2020.

  82. 82.

    Rosillon D, Buyse H, Friedland LR, Ng SP, Velazquez FR, Breuer T. Risk of intussusception after rotavirus vaccination: meta-analysis of postlicensure studies. Pediatr Infect Dis J. 2015;34(7):763–8.

    Article  PubMed  Google Scholar 

  83. 83.

    Deen J. The Dengue Vaccine Dilemma: balancing the Individual and Population Risks and Benefits. PLoS Med. 2016;13(11):e1002182.

    Article  PubMed  PubMed Central  Google Scholar 

  84. 84.

    Palache B. New vaccine approaches for seasonal and pandemic influenza. Vaccine. 2008;26(49):6232–6.

    CAS  Article  PubMed  Google Scholar 

  85. 85.

    Wilder-Smith A, Longini I, Zuber PL, Barnighausen T, Edmunds WJ, Dean N, et al. The public health value of vaccines beyond efficacy: methods, measures and outcomes. BMC Med. 2017;15(1):138.

    CAS  Article  PubMed  PubMed Central  Google Scholar 

  86. 86.

    York Health Economics Consortium (YHEC). A glossary of Health Economic Terms. 2016. Accessed 13 Mar 2020.

  87. 87.

    Barton P, Bryan S, Robinson S. Modelling in the economic evaluation of health care: selecting the appropriate approach. J Health Serv Res Policy. 2004;9(2):110–8.

    Article  PubMed  Google Scholar 

  88. 88.

    Brinsmead R, Hill S, Walker D. Are economic evaluations of vaccines useful to decision-makers? Case study of Haemophilus influenzae type b vaccines. Pediatr Infect Dis J. 2004;23(1):32–7.

    Article  PubMed  Google Scholar 

  89. 89.

    The World Bank. Updated income Classifications 2016. Accessed 13 Mar 2020.

Download references


The authors would like to thank Sophie Timmery and Salomé Murinello (Modis, Wavre, Belgium) for editorial assistance and coordination on behalf of GSK (Rixensart, Belgium).

Author information



Corresponding author

Correspondence to Hugo Arlegui.

Ethics declarations


This study is part of a PhD programme funded by GlaxoSmithKline Biologicals SA and the ANRT (Association Nationale pour la Recherche et la Technologie, Paris, France).

Conflicts of interest

Hugo Arlegui was employed by the GSK group of companies at the time of the study in the framework of a PhD programme. Vincent Bauchau, Gaëlle Nachbaur and Nicolas Praet are employed by the GSK group of companies. GSK is the producer of some of the vaccines of interest in this review. Vincent Bauchau, Gaëlle Nachbaur and Nicolas Praet also hold shares in the GSK group of companies. Kaatje Bollaerts, Francesco Salvo and Bernard Bégaud declare no financial and non-financial relationships and activities and no conflicts of interest.

Ethics approval

Not applicable.

Consent to participate

Not applicable.

Consent for publication

Not applicable.

Availability of data and material (data transparency)

All data generated and analysed during this study are included in this published article and its supplementary information files.

Code availability (software application or custom code)

Not applicable.

Authors’ contribution

All authors attest they meet the ICMJE criteria for authorship. All authors were involved in the conception of the study. Hugo Arlegui, Kaatje Bollaerts and Nicolas Praet participated in the collection and generation of the study data. Hugo Arlegui performed the study. All authors were involved in the analyses and interpretation of the data. All authors revised the manuscript critically for important intellectual content and gave final approval to submit for publication.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (DOCX 217 kb)

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License, which permits any non-commercial use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Arlegui, H., Bollaerts, K., Salvo, F. et al. Benefit–Risk Assessment of Vaccines. Part I: A Systematic Review to Identify and Describe Studies About Quantitative Benefit–Risk Models Applied to Vaccines. Drug Saf 43, 1089–1104 (2020).

Download citation