Introduction

There have been over 600 million confirmed cases of coronavirus disease (COVID-19) and over 6 million worldwide deaths by the end of 2022 since the onset of the COVID-19 pandemic [1]. The pandemic has significantly impacted healthcare and socio-economic development worldwide. The most prevalent clinical features of COVID-19 include fever, cough, and dyspnea [2]. While most cases are mild, the elderly and those with underlying diseases are at high risk of severe COVID-19. Moreover, some people also experience long-term effects after recovery. Novel oral antivirals such as molnupiravir, fluvoxamine, and paxlovid [3] are still under development, and heteropathy is believed to be the main clinical treatment. Therefore, vaccination is the first and most important step in stopping the spread of COVID-19 and reducing the social burden.

Vaccines can be divided into five categories according to their principles of antigen generation and production processes: inactivated virus vaccines, mRNA vaccines, DNA vaccines, viral vector vaccines, and protein subunit vaccines. Each type has certain advantages. Inactivated viral vaccines containing intact spike proteins and other proteins protect against viral variants by inducing a broader immune response [4]. mRNA and DNA vaccines are rapid and cost-effective platforms that can simulate natural infections by synthesizing endogenous proteins to induce a strong immune response [5]. Viral vector vaccines are characterized by robust immunogenicity, the absence of adjuvants, and long-term storage without freezing [6]. Protein subunits vaccines can produce robust and durable antibody responses and are expected to be safer because they do not utilize genetic materials [7].

Vaccine efficacy (VE) data are primarily obtained from phase III randomized controlled trials (RCTs). Previous studies have compared the efficacy and safety of vaccines using multiple post-hoc pairwise comparisons in meta-analyses [8,9,10]. In June 2021, a meta-analysis was conducted for eight Phase III RCTs encompassing four vaccine types [8]. The study indicated that all vaccine types exhibited good preventive effects against COVID-19, accompanied by an elevated risk of overall adverse events in the vaccinated groups. However, these studies did not compare multiple vaccines administered under identical conditions [8]. A network meta-analysis (NMA) provides a methodological approach to simultaneously compare vaccines through a common comparator (placebo) since there are no head-to-head clinical studies directly comparing the relative efficacy and safety of COVID-19 vaccines. In April 2021, the first published NMA of four Phase III RCTs showed that the vaccine exhibited different efficacies to prevent COVID-19: BNT162b2 ≥ mRNA-1273 > Gam-COVID-Vac > AZD1222 [11]. Subsequently, Rotshild et al. reported no statistical differences among vaccines in the preventive effect against severe COVID-19 of the elderly [12]. The latest NMA evaluation of the efficacy of 16 vaccines (October 2022) revealed that BNT126b2 conferred the highest protection against symptomatic severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) infection [13].

This study aimed to integrate the latest published data from Phase III RCTs to compare the efficacy and safety of COVID-19 vaccines in adult populations. The efficacy of COVID‑19 vaccines was also conducted to prevent symptomatic disease among the elderly. This manuscript was written following the PRISMA-NMA checklist [14].

Methods

Search strategy and selection criteria

A systematic search was performed in PubMed, EMBASE, the Cochrane Library, medRxiv, and SSRN from their inception to Sep 30, 2023 for COVID-19 vaccine studies. The search included the following keywords and subject terms: “COVID-19,” “SARS-CoV-2,” “vaccines,” “efficacy,” “safety” and “clinical trial”. Details regarding the search strings for the different databases are provided in Table S1.

The PICOS design approach was used to formulate the study eligibility criteria:

Population

Subjects who participated in clinical trials related to COVID-19 vaccines, aged > 18 years, and without a prior history of SARS-CoV-2 infection or COVID-19 vaccination.

Intervention

The intervention was to complete the COVID-19 vaccination according to the design plan. We selected the optimal administration regimen approved by the relevant agencies as the only intervention when a vaccine contained multiple regimens.

Comparison

Placebo or COVID-19 vaccines.

Outcome

The efficacy outcomes included the incidence of laboratory-confirmed (RT-PCR-positive) symptomatic SARS-CoV-2 infection. Safety outcomes included serious adverse events (SAEs).

Study design

Phase III RCTs with full-text publications were included.

Data extraction and quality assessment

Two investigators (XDW and KX) independently selected the articles and extracted data according to the title, abstract, full reports, and supplementary materials. All discrepancies were resolved by consensus between two other authors of the study (HBL and PZ). Data were extracted in three parts: study characteristics (date of publication, author, phase, sample size, trial country, and study design), baseline demographic characteristics (sex ratio and age range), vaccine characteristics (vaccine type, company, adjuvant, injection interval, and concentration), and outcomes (definition, and follow-up time). The quality of individual studies was evaluated using RoB2 (version 2 of the Cochrane tool for assessing the risk of bias in randomized trials) [15]. The five assessed sources of risk of bias were randomization process, deviations from intended intervention, missing outcome data, measurement of the outcome, and selection of the reported result.

Outcomes

The primary outcomes included type-specific efficacy and safety of COVID-19 vaccines in adults. Vaccines were divided into five categories: inactivated viral vaccines, mRNA vaccines, DNA vaccines, viral vector vaccines, and protein subunit vaccines. The secondary outcomes included the efficacy and safety of individual vaccines in adults, type-specific efficacy of COVID-19 vaccines in the elderly, and the efficacy of individual vaccines in the elderly.

VE was evaluated by comparing the difference in the number of laboratory-confirmed (RT-PCR-positive) symptomatic SARS-CoV-2 infection cases commencing 7–28 days after the last dose of the investigational product between the experimental and control groups.

Safety outcomes were evaluated as the number of participants that reported SAEs throughout the study period. Analysis of SAEs included all participants who received at least one dose. SAEs were defined in accordance with the ICH-GCP as any untoward medical contingency that resulted in death, was life-threatening, requiring hospitalization, or resulted in persistent or significant disability or incapacity at any dose, regardless of whether they were considered as associated with vaccination [16]. Safety analysis of the vaccines was limited to adults only, as no clinical research provided SAE data for the elderly.

Data synthesis and statistical analysis

An NMA only including indirect comparisons was conducted to compare and rank the COVID-19 vaccines in terms of efficacy and safety in the absence of trials directly comparing the two COVID-19 vaccines. Heterogeneity was initially assessed using the Cochrane Q test and I² statistics were calculated. A random-effects model was used when I² was greater than 50% and a fixed-effects model was used when I² was below 50%. Possible causes of heterogeneity were explored through sensitivity analysis. The transitivity underlying NMA was subjectively evaluated by comparing key clinical features. Inconsistency was not evaluated since no study directly compared the two vaccines. The risk ratio (RR) was chosen for the outcomes with a corresponding 95% confidence interval (95% CI) to determine the effect size. The model was run based on simulations of 20,000 iterations in the framework of the Bayesian theory with each of the four chains after a burn-in of 5,000 using Markov chain Monte Carlo (MCMC) techniques with Gibbs sampling. Model fit was ensured using trace plots, density plots with bandwidth, and Brooks-Gelman-Rubin diagnostic plots. Network diagrams were used to present the networks for the models, and the outcomes of pairwise comparisons were presented in the corresponding tables. The surface under the cumulative ranking curve (SUCRA) was calculated to summarize probability values and rank the interventions measured on a scale of 0 (best) to 1 (worst) [17]. Potential publication bias of the included studies was evaluated using a funnel plot and Egger’s test. All analyses were conducted using the “gemtc” package and “rjags” package that interfaces with JAGS 4.3.0 in R x64 4.0.3 (R Foundation for Statistical Computing, Vienna, Austria) [18,19,20].

Results

A total of 5606 records were identified by the search, with 24 published and one unpublished Phase III RCT [21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45] involving 22 vaccines eventually included in the NMA (Fig. 1). Two of the search results included a small number of individuals under the age of 18 years [29, 31], and another study included Phase I/II/III RCTs of AZD1222 vaccines [34]. These three studies were included in the NMA to ensure a sufficient number of samples. None of the included studies directly compared two different vaccines. In total, 915,370 participants were included, and more than 50% were randomly assigned for vaccination. Study characteristics and raw data are summarized in Table 1 and S2. A comparison of basic features, including outcome definition and participant characteristics (age, sex, and race) is presented in Table S3 and Figure S1. There was no evidence of violation of the transitivity assumption. Among these articles, studies with some concerns accounted for 36%, but there were no serious risks of bias according to the RoB2 (Figure S2).

Fig. 1
figure 1

Flowchart of study selection

Full size image
Table 1 Characteristics of the clinical studies included in the network meta-analysis
Full size table

Comparative efficacy and safety of different types of vaccines in adults

We explored the differences in efficacy and safety between different types of vaccines using NMA. Vaccines were divided into five categories: inactivated viral vaccines, mRNA vaccines, DNA vaccines, viral vector vaccines, and protein subunit vaccines. Star-shaped network diagrams of the primary outcomes are shown in Fig. 2(A) and S3.

Fig. 2
figure 2

Network diagram (A) Network diagram of type-specific efficacy for adults. (B) Network diagram of individual vaccine efficacy for adults. The thickness of the lines is proportional to the number of trials comparing every pair of treatments

Full size image

The inactivated viral, mRNA, viral vector, and protein subunit vaccines were predictably more effective than the placebo in terms of efficacy (25 RCTs involving 22 vaccines), with RRs ranging between 0.13 (95% CI [0.05, 0.31]) for mRNA vaccines and 0.28 [0.16, 0.49] for inactivated viral vaccines (Fig. 3(A)). The DNA vaccines (0.32 [0.07, 1.5]) were not statistically significant compared with the placebo. There were no significant differences between the various types of vaccines in the indirect pairwise comparisons (Table S4), although there was a trend in the mRNA vaccines for the lowest risk of symptomatic disease, with the lowest SUCRA value of 0.09 (Table S5).

Fig. 3
figure 3

Forest plot of intervention compared to the placebo in the network meta-analysis. RR, risk ratio; CI, confidence interval. (A) Forest plot of the efficacy of different vaccine types in adults. (B) Forest plot of the safety of different vaccine types in adults. (C) Forest plot of individual vaccine efficacy for adults. (D) Forest plot of individual vaccine safety for adults. (E) Forest plot of efficacy of different vaccine types in the elderly. (F) Forest plot of individual vaccine efficacy for the elderly

Full size image

In terms of safety (21 RCTs involving 19 vaccines), none of vaccines had a higher incidence of SAEs than the placebo (Fig. 3(B)). The inactivated virus vaccine ranked first, with a SUCRA value of 0.04, whereas the mRNA vaccine ranked last, with a SUCRA value of 0.98 (Table S6). There was a significant difference in the side effect rates between mRNA vaccines and other vaccine types in the indirect pairwise comparisons (Table S7). Funnel plots and Egger’s tests revealed asymmetry in VE and no asymmetry in vaccine safety (Figure S4).

Comparative efficacy and safety of individual vaccines in adults

Network diagrams are shown in Fig. 2(B) and S5. In terms of efficacy (25 RCTs involving 22 vaccines), all 22 vaccines were more effective than the placebo, with RRs ranging between 0.05 [0.02, 0.09] for BNT162b2 and 0.64 [0.52, 0.79] for SpikoGen (Fig. 3(C)). According to the outcome of pairwise comparisons (Table S8) and SUCRA value (Table S9), BNT162b2 had the highest efficacy (SUCRA value: 0.02), followed by mRNA-1273, Abdala, Gam-COVID-Vac, and NVX-CoV2373. The efficacy of SpikoGen was the lowest, with a SUCRA of 0.94.

In terms of safety (21 RCTs involving 19 vaccines), none of the vaccines had a higher incidence of SAEs than the placebo (Fig. 3(D)). BIV1-CovIran had the highest probability of being the vaccine with the lowest incidence of SAEs (SUCRA value: 0.1), followed by BBV152, Soberana, Gam-COVID-Vac, and ZF2001. In contrast, the safety of CoVLP + AS03 was the lowest, with a SUCRA value of 0.89. There were no statistically significant differences between most of the vaccines. Details of the pairwise comparisons and SUCRA values are shown in Tables S10 and S11.

Comparative efficacy of different types of vaccines in the elderly population

Data on efficacy in the elderly population were retrieved from 15 RCTs involving 14 vaccines. Vaccines are divided into four categories: inactivated virus vaccines, mRNA vaccines, viral vector vaccines, and protein subunit vaccines. The definition of the elderly population slightly differed across the included studies, ranging from 50 to 65 years. Star-shaped network diagram is shown in Figures S6.

The mRNA, viral vector, and protein subunit vaccines were predictably more effective than the placebo, with RRs ranging from 0.18 [0.05, 0.67] for mRNA vaccines and 0.23 [0.07, 0.75] for protein subunit vaccines (Fig. 3(E)). The inactivated virus vaccine (0.4 [0.1, 1.5]) was not statistically significant compared to the placebo. There were no significant differences between the various types of vaccines in the indirect pairwise comparisons (Table S12), although there was a trend in the mRNA vaccine for the lowest risk of symptomatic disease, with the lowest SUCRA value of 0.24 (Table S13). Funnel plots and Egger’s tests indicated no publication bias (Figure S7).

Comparative efficacy of individual vaccines in the elderly population

Star-shaped network diagram is shown in Figures S8. 11 of the 14 vaccines had good preventive effects against COVID-19 compared with the placebo, with RRs ranging between 0.06 [0.01, 0.16] for BNT162b2 and 0.48 [0.21, 0.99] for Ad5-nCoV (Fig. 3(F)). CVnCoV, CoVLP + AS03, and CoronaVac were interpreted as having no differences from the placebo. BNT162b2 had the lowest SUCRA value of 0.08, with the highest probability of being the most effective vaccine for the elderly, followed by Gam-COVID-Vac and mRNA-1273, whereas CVnCoV had the lowest probability, with the highest SUCRA value of 0.92. Details of the SUCRA values and pairwise comparisons are shown in Tables S14 and S15.

Additional analyses

Sensitive analyses were performed after excluded trials with a follow-up time of less than 2 months. 18 RCTs were included in analyses. The results were stable and were similar to the main analysis after excluding 7 trials (Table S16). In addition, sensitivity analyses were performed after excluded the unpublished study, and the results are robust.

Discussion

This study was based on 25 RCTs that included 915,370 patients randomly assigned to receive 22 vaccines or a placebo. This project updates and extends previous research and is the most comprehensive NMA to compare the efficacy of COVID-19 vaccines in preventing symptomatic disease and the incidence of SAEs in adults and the elderly.

In terms of safety, mRNA vaccines may increase SAEs versus the placebo, although this result was not statistically significant. Similar trends were described in an earlier meta-analysis of 11 trials [46]. Our results provided the following rankings according to RR in the indirect comparison: inactivated vaccines ≥ viral vector vaccines ≥ protein subunit vaccines > mRNA vaccines. This is unsurprising given the high safety of inactivated vaccines since no viral genetic material is involved. In addition to SAEs, inactivated vaccines have the lowest risk of local or systemic adverse events following immunization [47]. The ranking of individual vaccines was generally consistent with the vaccine type. BIV1-CovIran, an inactivated vaccine, had the lowest incidence of SAEs. Notably, most included studies did not specifically exclude patients with symptomatic COVID-19 from SAE, which may have affected the accuracy of the above ranking.

In terms of efficacy, all vaccine types versus placebo significantly prevented symptomatic SARS-CoV-2 infection, but the 95% CI for DNA vaccines indicated no effect. In the indirect comparison, our results provided the following ranking according to the RR: mRNA vaccines ≥ protein subunit vaccines ≥ viral vector vaccines ≥ inactivated vaccines ≥ DNA vaccines. The 95% CI for all vaccine types was compatible with no effect, although the RR values were significant. One possible explanation for the excellent efficacy of mRNA vaccines is the production of a fully functional protein through cellular translational machinery, which induces powerful and durable immunity against the coronavirus [48]. An earlier NMA compared nine vaccines to prevent symptomatic SARS-CoV-2 infection, based on the results of Phase III RCTs up to August 1, 2021 [12]. BNT162b2 had the highest efficacy, followed by mRNA‑1273, Gam‑COVID‑Vac, NVX‑CoV2373, CoronaVac, BBIBP-CorV, WIBP-CorV, and Ad26.COV2.S [12]. Similarly, one recent NMA reported that BNT126b2 conferred the highest protection, followed by mRNA-1273, Gam‑COVID‑Vac and NVX-CoV2373 [13]. In line with previous evidence, we ranked BNT162b2 with the highest efficacy, followed by mRNA-1273, Abdala, Gam-COVID-Vac, and NVX-CoV2373. We also found that BNT162b2 and mRNA-1273 mRNA vaccines performed best in preventing symptomatic COVID-19, while CVnCoV ranked lower. A possible explanation is that approximately 85% of COVID-19 cases in the CVnCoV trial were caused by variants that might alter VE owing to the increased transmissibility and evasion of neutralizing humoral immunity [49]. In addition, 12 µg mRNA contained in CVnCoV may be insufficient to elicit a protective immune response compared to 30 µg in BNT162b2 and 100 µg in mRNA-1273.

We found that BNT162b2 had the highest efficacy in terms of the efficacy in preventing symptomatic SARS-CoV-2 infection in the elderly population. This was consistent with the conclusion of an earlier study [47]. CVnCoV, CoVLP + AS03, and CoronaVac were interpreted as having no difference from the placebo, possibly owing to an insufficient absolute number of events in the short follow-up duration. In fact, the VE of CoronaVac in the real world has reached 66.6% in individuals aged > 60 [50]. In addition to the elderly, the impact of vaccines on children is gradually emphasized. Recently published Phase III clinical trials show that mRNA-1273 [51,52,53], BNT162b2 [54], and BBIBP-CorV [55] are safe in populations younger than 18 years and trigger an immune response no less than that in young people. There is a lack of large-scale clinical trials to support the active use of COVID-19 vaccines for other populations, such as pregnant women, immunodeficient patients, and people that were previously exposed to SARS-CoV-2.

Our review has some limitations; the above results should be cautiously interpreted since inconsistencies were not assessed in the absence of trials that directly compared the two COVID-19 vaccines. The transitivity assumption underlying the NMA was evaluated by comparing key clinical features, including participant characteristics (age, sex, and race), and outcome assessment (definition and measurement). However, there are some differences in the research background and protocols, such as vaccine dose and different SARS-CoV-2 variants, which might lead to deviations in analytical results. Furthermore, vaccines face great challenges in terms of increasing the diversity of variants, and the ranking of VE can change. Booster vaccines are necessary to prevent SARS-CoV-2 variant infections and provide durable immunity. These data suggest that homologous and heterologous booster vaccines have an acceptable safety profile and heterologous boosting may be more immunogenic than homologous boosting [56]. Our conclusion aims to provide a primary reference for vaccine selection. However, other important factors such as the prevention of severe COVID-19, long-term side effects, and economic considerations should also be considered practical scenarios.

Conclusions

Our study is the most comprehensive NMA exploring the efficacy and safety of type-specific and individual COVID-19 vaccines based on the latest data. Our analysis showed that BIV1-CovIran inactivated vaccine had the lowest incidence of SAEs in adults, and BNT162b2 mRNA vaccine had the highest efficacy in preventing symptomatic SARS-CoV-2 infections in adults and the elderly population.