Introduction

Since the middle of the 20th century, there has been a general belief that influenza vaccination is cost-effective, at least to certain risk groups, such as the over-65s or people with chronic diseases like asthma, chronic obstructive pulmonary disease (COPD) or cancer. Many developed countries offer a seasonal influenza vaccination programme for people in certain risk groups [1], and some countries, such as the United States, recommend vaccinating healthy children and adults annually from the age of 6 months [2].

Most evidence for the efficacy of influenza vaccination stems from observational studies. Although these studies are continually being developed, unknown biases and confounders render their results uncertain. They are, therefore, considered to be of lower quality than randomised controlled trials (RCTs). An example of this was the former overestimation of the effect of influenza vaccination on mortality, which is now largely attributed to a confounder known as the “healthy vaccine recipient effect”: healthy individuals are more likely to get vaccinated, and have better outcomes [3].

One meta-analysis [4] found the Cochrane Collaboration to have the highest quality of systematic reviews of influenza vaccination. Between 2006 and 2014, they have published several systematic reviews and meta-analyses on the evidence of influenza vaccination of different demographic groups, both healthy and sick. These analyses were largely based on RCTs, but also included case–control, cohort and other observational studies.

The aim of this review is to present a summary of Cochrane Reviews investigating the effect of influenza vaccination and to discuss the implications of that effect.

Method

The Cochrane Library was searched with “*flu* AND vaccin*” in “Title, Abstracts and Keywords” (last accessed 7 June 2014). The use of asterisks ensured that all forms of the search words were included, for example, “influenza”, “influenzas”, “the flu”, “vaccinations” and “vaccinate”.

Included studies were quality assessed according to AMSTAR. The results of each study were stratified for the number and type of included studies, population, intervention, primary outcomes, significant effects and the Cochrane Review’s authors’ assessment of quality of included RCTs. The results are aggregated in Table 1. Important inclusion and exclusion criteria and citations of the Cochrane authors’ main conclusions are mentioned in the main text. Durations and settings were not available for all included studies in the reviews, and their mentioning would require a detailed walkthrough of each study, which is beyond the scope of this article. Full texts with descriptions of all included studies can be downloaded at http://www.thecochranelibrary.org.

Table 1 Cochrane Reviews investigating the effect of influenza vaccination in different populations
Full size table

Different formulae to calculate the number needed to vaccinate (NNV) were used in the Cochrane Reviews. To standardise comparison, NNVs were recalculated using the formula [NNV = 1/(Control Event Rate − Interventional Event Rate)].

Results

The search yielded 43 hits, of which 12 corresponded with the aim of this article. The other 31 were excluded, as they did not address the effect of vaccination against the influenza virus, instead dealing with the bacteria Haemophilus influenzae or containing protocols for future Cochrane analyses.

All included studies had a maximum AMSTAR [5] score of 11.

Vaccines for preventing influenza in healthy children [6]

The analysis investigated the effect of all types of influenza vaccines in healthy children under 16 years of age on reducing cases of influenza, influenza symptoms, complications, and systemic and severe adverse events. The children were stratified by three available age groups (under 2s, under 6s and over 6s) and by type of outcome.

For laboratory-confirmed influenza, the vaccine efficacy (VE, reduction in cases of laboratory-confirmed influenza) for inactivated vaccines was only significant for children over 6 years of age. Live attenuated vaccines demonstrated an overall efficacy in reducing cases of influenza in children under 6 years old, but the evidence was weak for children over 6 years of age, with only one small study among 60 participants. Both inactivated and live vaccines showed similar vaccine effectiveness (VE, reduction in cases of influenza-like illness, ILI). No evidence was found for otitis media, lower respiratory tract diseases, hospitalisations or all-cause mortality.

The authors stated that data for children under 2 years old either showed no effect or were not usable. Because of insufficient information, no estimates could be given regarding the effect of matching different vaccines with circulating viral strains.

Certain vaccines have been associated with specific side effects in children:

  • “Pandemrix” was associated with cataplexy and narcolepsy in children during the 2009 seasonal influenza pandemic.

  • “2010 TIV” caused fever cramps in every 110th child. It was mostly used in Australia and was retracted from the market. Only 162 children participated in studies before “2010 TIV” was released, deeming the pre-release examination of side effects insufficient.

One study with a low risk of bias showed an increase in the rate of bronchitis between live attenuated influenza vaccines (LAIVs) and saline placebo recipients (3.1 % and 1.6 %, respectively; RR 1.9, p = 0.046), but according to the authors, the varying study design prevented a meta-analysis of safety outcome data. Moreover, “extensive evidence of reporting bias of safety outcomes from trials of live attenuated influenza vaccines (LAIVs) impeded meaningful analysis”.

The authors emphasised that the review included trials funded by industry and concluded, “influenza vaccines are efficacious in preventing cases of influenza in children older than two years of age (…) reliable evidence on influenza vaccines is thin but there is evidence of widespread manipulation of conclusions and spurious notoriety of the studies. The content and conclusions of this review should be interpreted in the light of this finding.”

Vaccines for preventing influenza in healthy adults [7]

Both randomised and non-randomised studies were included to analyse the effect of any type of influenza vaccination in healthy adults compared to placebo or no treatment in people aged 16 to 65 years. Outcomes included the effect on influenza, ILI, complications, adverse effects and the impact on pregnancy and newborns.

Vaccine effects were moderate for laboratory-confirmed influenza and small for ILI. The Cochrane Review’s authors commented on the apparent higher NNV for influenza than for ILI, and argued that this was due to different incidences in the study population: when the inactivated vaccine matched the virus antigenically, 2.4 % of unvaccinated versus 1.1 % of vaccinated participants developed laboratory-confirmed influenza, while 15.6 % versus 9.9 %, respectively, developed ILI. Regardless of matching, the efficacy and effectiveness were similar, except for the inactivated vaccine’s effectiveness against ILI. The similarity may be explained if many of the studies with unknown matching were actually performed in seasons where the virus and the vaccine matched.

No evidence was found for any effect on complications or other outcomes, e.g. visits to the GP, time off work (0.04 saved working days), hospitalisation, use of antibiotics or pneumonia. No RCTs were found which had investigated maternal, pregnancy or neonatal outcomes.

All vaccines caused local side effects. Rarely did influenza vaccines cause major harms, and they were mostly batch-related. One RCT reported mild oculo-respiratory syndrome caused by an inactivated vaccine. Regarding adverse outcomes, the authors stated, “the safety evidence base from randomised trials of inactivated vaccines is very small, probably indicating less concern with harms.”

Considering all included randomised and non-randomised trials, the authors found that the risk of bias was unclear because of insufficient information, commenting, “The main problems with influenza vaccine studies are their poor quality and discrepancies between the data presented, their conclusions and the authors’ recommendations”, arguing that industry-funded studies likely presented more favourable results.

The authors concluded that, “Influenza vaccines have a very modest effect in reducing influenza symptoms and working days lost in the general population, including pregnant women (…) the results of this review provide no evidence for the utilisation of vaccination against influenza in healthy adults as a routine public health measure.”

Vaccinations for preventing influenza in the elderly [8]

The analysis included both randomised and non-randomised studies examining the effect of any influenza vaccination on cases of influenza, ILI and its complications in the over-65s.

To avoid one case of influenza, 30 elderly people had to be vaccinated. For ILI, the NNV was 54. Included RCTs showed no effect on pneumonia or all-cause mortality. The non-randomised trials in the analysis also found no effect on complications whatsoever, including pneumonia, hospitalisation due to influenza or any respiratory disease or heart disease, total deaths or deaths related to influenza. All included studies were performed during the influenza season or other periods of high virulence, but further details regarding a match between virus and vaccine were lacking. Only one RCT assessed currently available vaccines.

The non-randomised trials had a high risk of bias. According to the authors, “Selective reporting including major inconsistencies between different parts of the text were a common feature.” It was not possible to evaluate confounding factors, since there was a lack of information about co-morbidity, social status etc. The authors stated, “In most of the trials, the quality of the text was such that we had difficulty in understanding what went on”.

The authors concluded, “The general quality of influenza vaccines studies is very low and that publication in prestigious journals is associated with partial or total industry funding. We could not explain this association with study quality, size or its status (…) it is likely that data presented in this review are so biased as to be virtually uninterpretable (…) the available evidence is of poor quality and provides no guidance regarding the safety, efficacy or effectiveness of influenza vaccines for people aged 65 years or older”.

Influenza vaccination for healthcare workers who care for people aged 60 years or older living in long-term care institutions (LTCIs)[9]

The analysis included only RCTs investigating the effect in people aged over 60 years living in LTCIs on cases of influenza, lower respiratory tract infection, admission to hospital and deaths caused by respiratory illness, when their carers were given influenza vaccine.

No effect was found on any outcome. Nevertheless, the Cochrane authors noted that the included studies, “…all reached conclusions which were not based on the data presented.” All trials had a high risk of bias, including unknown blinding and selection bias. Information was lacking regarding a match between the circulating influenza virus and the vaccine.

In conclusion, “there is an absence of high-quality evidence that vaccinating healthcare workers against influenza protects people aged 60 years or older in their care and thus there is little evidence to justify medical care and public health practitioners mandating influenza vaccination for healthcare workers who care for the elderly in long-term care institutions.”

Influenza vaccine for patients with chronic obstructive pulmonary disease (COPD) [10]

Only RCTs were included in the meta-analysis, which investigated the effect of any type of influenza vaccination on COPD exacerbations, acute respiratory illnesses, complications, lung function, adverse effects and cost-effectiveness. Included trials had to exclude persons with asthma.

Fears were that influenza vaccination could cause an exacerbation, which was why some studies examined whether the participants experienced an increase in early exacerbations during the first and second weeks after vaccination because of a side effect of the vaccine, or a decrease in late exacerbations later than three to four weeks after vaccination as a consequence of developing immunity to influenza. However, all outcome estimates were only significant for late exacerbations and influenza-related respiratory infections. These estimates were based on just two studies totalling 180 participants, and only one of these studies demonstrated significant effects. This study was from 1961 and had 55 participants. The other study was performed in 2004, had 125 participants and was not significant regarding any of the outcomes. The studies were of similar quality.

The authors concluded, “It appears, from the limited number of studies performed, that inactivated vaccine reduces exacerbations in COPD patients.”

Vaccines for preventing influenza in people with asthma [11]

RCTs in children over 2 years old and adults investigating the effect of any influenza vaccine on asthma exacerbation, complications and mortality were included in the analysis. Trials including persons with COPD were excluded.

It has been thought that influenza can cause asthma exacerbations, but, also, that influenza vaccines can cause asthma attacks. No significant reduction or increase in the number of asthma exacerbations was found in those people who were vaccinated against influenza. The only significant outcome was a barely clinically significant improvement in the asthma quality of life score. However, the proportion of patients with a minimal clinical difference was not significant.

The authors concluded that the effect of influenza vaccination against asthma exacerbations is uncertain and “there is no firm evidence from controlled clinical trials to support the adoption of universal vaccination in patients with asthma as a clinical policy.”

Vaccines for preventing influenza in persons with cystic fibrosis [12]

RCTs were included to investigate the effect of any influenza vaccine in persons of all ages with cystic fibrosis.

All studies demonstrated that persons with cystic fibrosis generated a satisfactory immune response to influenza vaccination. The studies did not investigate clinical outcomes or compare vaccination with placebo. To perform such a comparison is important because, as the Cochrane Review’s authors pointed out, “…a high antibody level, however, does not imply protective immunity, as they are not the main component of the host defense against viral infection”.

The authors concluded that there is no support for recommending influenza vaccination for people with cystic fibrosis, as “there is currently no evidence from randomised studies that influenza vaccine given to people with cystic fibrosis is of benefit to them.”

Influenza vaccine for children and adults with bronchiectasis [13]

A search for RCTs investigating the effect of all types of influenza vaccines in people with bronchiectasis yielded no studies.

Influenza vaccination in children being treated with chemotherapy for cancer [14]

RCTs and clinical controlled trials investigated the effect of any influenza vaccine on cases of influenza, ILI, complications, adverse reactions and influenza immunity in children (1–18 years old) with cancer. Only children treated with chemotherapy or who had been off chemotherapy for less than 1 month were included in the analysis.

The results demonstrated that children who received chemotherapy generated an immune response to the influenza vaccination, although the response was weaker than in those who did not receive chemotherapy. None of the included studies compared influenza vaccine with placebo, and no clinical outcomes of influenza infection were assessed. An analysis of adverse effects was only descriptive in the Cochrane Review, without comparison to placebo: those vaccinated had side effects in the form of mild, local reactions and low fever. No life-threatening or lasting side effects were reported.

The authors concluded, “These patients are able to generate an immune response to influenza vaccine, but it remains unclear whether this immune response protects them from influenza infection or its complications.”

Influenza vaccines in immunosuppressed adults with cancer [15]

The analysis included randomised and non-randomised trials assessing the effect of inactivated influenza vaccines on all-cause mortality in adults (over 16 years of age) with cancer treated with chemotherapy, haematological cancers and cancer up to 6 months after autologous transplantation or allogeneic haematopoietic stem cell transplantation.

Only one RCT was included, which did not investigate the primary outcome, but found a statistically significant reduction in ILI and hospitalisations, which was not found in the observational studies. Outcomes and results varied greatly among the included studies, and a meta-analysis was not performed.

Although the quality of evidence was low, the authors concluded that there was a possible benefit with regard to influenza, ILI, pneumonia, hospitalisations and survival: “The evidence (though weak) is in favour of vaccinating this population”.

Vaccines for prophylaxis of viral infections in patients with haematological malignancies [16]

The analysis included RCTs assessing the effect of all types of viral vaccines (including influenza vaccines) on the incidence of the viral infection concerned in patients of all ages with haematological malignancies.

No data were found on the primary outcome. Most trials focused on the immunological response instead of on clinical outcomes, which, as previously mentioned, cannot necessarily be translated to a clinical benefit. The quality of the trials was low: none explained the randomisation process, were double-blinded or contained an intention-to-treat analysis.

The authors concluded, “Inactivated influenza vaccine might reduce respiratory infections and hospitalisation in adults with multiple myeloma or children with leukemia or lymphoma. However, the quality of evidence is low.”

Influenza vaccines for preventing coronary heart disease [17]

The analysis included RCTs investigating the effect of inactivated influenza vaccines on the rates of myocardial infarction, unstable angina or cardiovascular death in people over 18 years age with or without a history of cardiovascular disease.

Two RCTs were included in the analysis, FLUVAC and FLUCAD. All participants had known cardiovascular disease, which was why no data were available to determine the effect of primary prophylaxis of influenza vaccination against coronary heart disease. FLUVAC contained vague information on randomisation and blinding. The authors could not be contacted to solve the issue, leaving the trial with a moderate risk of bias. FLUCAD was a randomised, controlled, double-blinded study with pre-registered effect outcomes. The risk of bias in this trial was low.

The only significant effect was found in one of two groups in the FLUVAC study including participants with former acute myocardial infarction, demonstrating a decreased risk for secondary cardiovascular death (RR = 0.19 (0.07, 0.53), N = 200, NNV 6). Neither the other FLUVAC group, which received percutaneous coronary intervention, nor the participants of the FLUCAD study (all treated for coronary artery disease, not further specified) showed any significant effect (N = 759) on the same outcome.

The Cochrane authors stated, “The studies were small and the methodological quality of one trial was just moderate. Therefore we conclude that it is not possible to draw conclusions on the beneficial or harmful effect of influenza vaccination in the prevention of acute heart disease.”

Future analyses

The Cochrane Library also contained protocols for future analyses evaluating the effect of influenza vaccination in patients with HIV/AIDS, infants and children with acute otitis media, multiple sclerosis and the impact on the health of pregnant women, neonates and infants.

Discussion

Many studies, both observational and RCTs, have found a positive effect of influenza vaccination. The results, however, vary according to the type of vaccine, the match between the vaccine and the circulating influenza strain, which population group is being vaccinated and which outcomes are being investigated. Vaccines administered to healthy immunocompetent recipients in years where the vaccine matches the circulating viral strain show the greatest effect.

Current guidelines recommending influenza vaccination are largely based on observational studies, which are biased and confounded in various ways, the ‘healthy vaccine recipient effect’ being one of them [18–21]. The Cochrane Reviews presented in this paper mostly included RCTs. Several of them found evidence that influenza vaccination reduces the number of cases of influenza and ILI. However, apart from a possible reduction in the number of ‘late exacerbations’ and influenza-related respiratory infections found in patients with COPD, and a slight improvement in the asthma quality of life symptom score, no significant effect was found on serious influenza complications. Many randomised and non-randomised trials had a high risk of bias. Often, the randomisation and blinding processes were inadequately described or missing. Non-specific disease definitions, such as ILI, or surrogate measures, such as a vaccine’s ability to increase antibody levels, were sometimes used as primary outcomes instead of hard clinical endpoints. Considering all the different strains of viruses that circulate during an influenza season, this creates uncertainty regarding the clinical effect. Generally, the Cochrane Reviews also found the included studies to be lacking in data on side effects.

On the other hand, the meta-analyses themselves had several shortcomings, the main being the heterogeneity of the included results: studies with different vaccines and administration routes, varying virulence and pathogenicity of different strains of influenza virus, changing definitions of influenza, ILI and its complications, and investigations performed during endemics of variable intensity are not immediately suited for a synthesis of data. The conclusions of a meta-analysis must, therefore, be critically scrutinised, and it must be realised that well-performed observational studies may give more valid results than meta-analyses with an indefensible high degree of heterogeneity.

Beyer et al. [22] challenged the conclusion of one Cochrane Review [8]. They rearranged the data regarding influenza vaccination for the elderly and used broad outcome definitions instead of sub-divided strata. During seasons where the vaccine matched the circulating virus, they found that the vaccine effect against laboratory-confirmed influenza was 49 % (95 % CI 33–62 %), against ILI 39 % (35–43 %) and against complications 28 % (26–30 %). They also found a lesser or absent effect in seasons with antigenic drift or absence of virus circulation, respectively. The relative risk for ‘all-cause mortality’ was 48 % (47–50 %), which the authors stated suggested a ‘healthy vaccine recipient bias’. Strangely, Beyer et al. did not mention how this bias might have affected the other outcomes too.

The analysis did not provide data regarding NNVs, and no comment was made on the quality of the included studies, deemed poor by the authors of the Cochrane Review. Five of the six authors of the article, including the first and last authors, declared conflicts of interest.

In comparison, RCTs included in the Cochrane Review [8] found (in conditions of high viral circulation, but with unknown matching) an inactivated vaccine effect of 58 % (95 % CI 34–73 %) against laboratory-confirmed influenza and 43 % (21–58 %) against ILI.

The results from non-RCTs included in the same review found no significantly different effect on confirmed influenza, whether there was a match or not. VE against ILI was 23 % (6–36 %) in seasons of good match between the vaccine and the circulating virus, and was not significant in seasons of bad matching. Against pneumonia, VE was 46 % (30–58 %) when the vaccine was matching, and, again, not significant when not matching the circulating virus. The effect on all-cause mortality was 60 % (23–79 %), but the estimate stemmed from only one small study.

The first author of the Cochrane Review declared conflicts of interest.

Beyer et al. [22] advocated that, when examining the effect of an influenza vaccine, a season with a match between the viral antigens and the vaccine is needed. However, when examining the effect of influenza vaccines in general, as when calculating the cost–benefit, absolute risk reduction or NNV, it is important to take into account that a vaccine may be less effective in the case of antigenic drift or may fail completely if there is no match with the circulating virus. If the number of infected individuals is low or the circulating influenza virus has a low virulence, the effect will also be diminished, even if there is a good match between the virus and the vaccine.

Another meta-analysis, by Osterholm et al. [3], focused on the effect of influenza vaccination on the number of cases of polymerase chain reaction-confirmed influenza. The vaccine efficacy was found to be 59 % (95 % CI 51–67 %, NNV 65) for the inactivated vaccine in adults aged 18–64 years, and 83 % (69–91 %, NNV 8) for the live attenuated vaccine in children aged 6 months to 7 years. The authors concluded that vaccine protection against influenza is moderate, being greatly reduced or absent in seasons of poor matching, stating, “Evidence for protection in adults aged 65 years or older is lacking”.

Two Cochrane Reviews reached similar results: in healthy adults aged 16–65 years, the overall inactivated vaccine efficacy was 62 % (95 % CI 56–67 %, NNV 82) [7], while live vaccine efficacy in children between 2 and 6 years of age was 82 % (71–89 %, NNV 7) [6].

Conclusion

Observational studies may have confounding factors and biases not yet accounted for, which is why randomised controlled trials (RCTs) are preferred. Several Cochrane Reviews on influenza vaccination, mostly including RCTs, have found some evidence of effect on influenza, but the included studies were often heavily biased. Except for a possible benefit in people with COPD or haematological malignancies or in immunosuppressed adults with cancer, all Cochrane Reviews concluded that the general recommendations for influenza vaccination are not supported by current evidence. This conclusion has been challenged by other non-Cochrane reviews, despite obtaining similar results. Additional high-quality studies are warranted in order to give an exact measure of the effects and side effects of influenza vaccination. In addition to being independent, randomised and placebo-controlled, they should contain relevant clinical outcomes and span multiple seasons. Cases of influenza should be confirmed by reverse transcription polymerase chain reaction (RT-PCR) and complications like pneumonia with X-rays and bacterial culture. Data on all-cause mortality and death from specific causes should be included.

For ethical reasons, such trials should begin with populations who rarely experience severe complications from influenza, minimising the risk of withholding an effective treatment. Many Western countries still do not have a policy of recommending the vaccination of healthy children and adults against influenza. These groups would be suitable for large, well-performed RCTs. If the vaccines turn out to be considerably less effective than currently presumed, one might consider further research in groups at risk of developing serious diseases secondary to influenza. Alternatively, studies can be performed in countries where influenza vaccination policies have not yet been decided. It is essential that these studies follow the highest standards of quality, explaining that participation is of international benefit. Moreover, in order for the results to be applicable to other populations, investigators in non-Western countries must be aware of other confounding factors, such as decreased overall physical condition, exotic infective comorbidities and vast differences in social status.