Background

Since randomized controlled trials (RCTs) assessing the effects of influenza vaccination on clinical outcomes are scarce, evidence on influenza vaccine effectiveness (VE) mainly derives from observational studies [1]. However, these studies are prone to bias and have been suspected to systematically overestimate VE, particularly against unspecific outcomes such as all-cause mortality and among the elderly [2]. Although it has been accepted that observational studies are susceptible to bias, there is an ongoing controversy whether and to what extend confounding by indication and healthy vaccinee bias affect influenza VE estimates [3–9]. Both forms of bias/confounding have been described in such studies, but it is important to note that their presence has opposing consequences for the VE estimates: “confounding by indication” is likely to be present if patients with underlying chronic diseases are more likely to be vaccinated than healthy study participant. If no adequate statistical adjustment (e.g., for comorbidities) is made, this leads to an underestimation of VE since the less healthy population is at higher risk of adverse health outcomes. The alternative scenario is called “healthy vaccinee bias” and refers to a situation when patients, who are in better health conditions, are more likely to adhere to the annually recommended influenza vaccination [10]. If not corrected for (e.g., by adjustment for comorbidities or indicators of health seeking behavior), healthy vaccinee bias leads to an overestimation of VE.

To test whether residual confounding by healthy vaccinee effects is still present in the adjusted data, it has been suggested by some authors that investigators should obtain “off-season” estimates. Off-season estimates are calculated for time periods outside influenza seasons when the virus is (virtually) not circulating and therefore no vaccine effect should be present [10, 11]. Any VE obtained during this control period would be attributable to unmeasured confounding, whereas successful adjustment would have removed the effect.

A systematic analysis of these two forms of bias/confounding and their consequences for influenza VE studies has not been published so far. We therefore addressed this issue by a systematic review.

Methods

Question framing

This study addressed the following questions: (i) How often do observational studies on influenza VE show indication of confounding by indication and/or healthy vaccinee bias? (ii) What is the impact on VE point estimates? And (iii) how many of these studies show indication of unmeasured (residual) confounding in the adjusted analyses? To define the conceptual framework of the study, we identified five indicators from the literature, which allow conclusions on the presence of the two forms of bias/confounding in the included studies (Table 1).

Table 1 Conceptual framework: Indicators and conclusions for presence of confounding by indication and healthy vaccinee bias in influenza vaccine effectiveness studies
Full size table

Study protocol

We performed the systematic review according to the Preferred Reporting Items for Systematic Reviews and Meta-analyses (PRISMA) statement [12]. The respective protocol for this review is shown in Additional file 1.

Eligibility criteria

Studies were included if they fulfilled the following criteria defined a priori: (i) observational (non-randomized) study; (ii) calculated influenza VE by comparing vaccinated and unvaccinated participants; (iii) reported baseline characteristics of vaccinated and unvaccinated participants; (iv) reported data on at least one clinical outcome; (v) reported crude and confounder-adjusted VE estimates from at least one influenza season; (vi) reported confounder-adjusted VE estimates from at least one “control” period outside the influenza season (off-season estimate).

Literature search

Two reviewers (CR and TH) searched MEDLINE, EMBASE and Cochrane Central Register of Controlled Trials (date of last search: 25.05.2014) and independently screened each citation and subsequent full text articles. The complete search strategy is shown in Additional file 2. Electronic searches were complemented by manually searching the reference lists of all identified studies and reviews for additional studies. No restrictions were made regarding publication language and publication status (published/unpublished).

Data extraction

From each included study, two investigators (CR and TH) independently extracted the following information: country, study design, age, sex, characteristics of study population (e.g., patients with underlying comorbidities), source of patient data, identification of clinical outcomes and vaccination status, definition of influenza season and off-season, and population size. In addition, we extracted data on crude and adjusted VE point estimates for all reported outcomes during influenza seasons, adjusted off-season point estimates, and which confounder were considered. Extraction forms were pilot tested with the first two identified studies.

Assessment of risk of bias

Two investigators (CR and TH) independently assessed risk of bias. In case of disagreements, a final decision was made by consensus or resolved by a third reviewer (OW). We used the predefined criteria derived from the above mentioned methodological framework (see Table 1) to assess the risk of healthy vaccinee bias and confounding by indication in the included studies: A study was judged to be at high risk of healthy vaccinee bias if vaccinated participants had significantly fewer comorbidities (or respective indicators such as medical visits) than unvaccinated participants, as indicated by baseline characteristics. A study was judged to be at high risk of confounding by indication if vaccinated participants had significantly more comorbidities (or respective indicators) than unvaccinated participants, as indicated by baseline characteristics. For case–control studies, vaccinated and unvaccinated participants of the control groups were compared. The results of these assessments were expressed as a considered judgment, using the categories “high risk of bias”, “low risk of bias” and “unclear risk of bias”.

Two approaches, a descriptive and a meta-analytical, were used to assess whether the included studies successfully corrected for bias/confounding. First, we compared crude VE estimates to confounder-adjusted in-season estimates from the same studies. If the studies reported more than one confounder-adjusted estimate, we used the fully adjusted model. If adjustment increased the estimated VE, we concluded that data were at least in part corrected for confounding by indication. If adjustment decreased the estimated VE, we concluded that data were at least in part corrected for healthy vaccinee bias (see Table 1).

Second, we used the approach suggested by Hrobjartsson et al. [13, 14] to quantify the extent by which adjustment for confounders increased the in-season estimate compared to the crude estimate: For each outcome for which more than one study reported data, we calculated the ratio of odds ratios (crude/adjusted VE during influenza season) per study. A ratio of >1 indicates that adjustment led to a stronger effect of vaccination, i.e., an increased VE. For calculation of 95 % CI, we used the formula provided by Hrobjartsson et al. [13]. To quantify the impact of adjustment for confounders, we then meta-analysed the individual study ratios of odds ratios for each outcome separately, using random-effects models with inverse-variance methods. For this analysis, we excluded two studies [15, 16] which did not report 95 % CI for the respective point estimates.

To evaluate the presence and magnitude of off-season VE estimates, being proposed as indicators of healthy vaccinee bias, we contrasted confounder-adjusted in-season estimates to “pseudo-effectiveness” estimates measured during off-seasons.

All statistical analyses were performed using STATA 12 (StataCorp LP, Texas, USA).

Results

By systematic literature search we identified 3385 publications, of which 23 were finally included in our analysis (Fig. 1) [3, 5, 7, 15–33]. Details on the excluded studies are reported in Additional file 3. Baseline characteristics of the 23 included studies are shown in Table 2. Of these, 20 were cohort studies, while the remaining three had a case–control design. The studies were conducted in North America (n = 14), Europe (n = 6), Taiwan (n = 2) or in multiple continents (n = 1) and mainly used disease classification codes (e.g., ICD-9) or civil register data for the identification of outcomes. In three studies interviews were conducted or self-administered questionnaires were applied to collect primary data on relevant outcomes or vaccination status [24, 25, 27]. Except of four studies, which were either performed in students (n = 1), in adults aged 40+ years (n = 1), or in women who recently experienced live birth (n = 2), all studies were conducted in elderly persons. Seven studies covered populations with underlying comorbidities, namely with (chronic) heart disease (CHD), [21, 22] end-stage renal disease (ESRD), [17, 23] chronic obstructive pulmonary disease (COPD), [28, 29] or patients with diabetes or vascular disease [33].

Fig. 1
figure 1

Flow chart for the systematic review

Full size image
Table 2 Baseline characteristics of included studies
Full size table

Reported outcomes

The included studies reported VE estimates (crude and adjusted in-season plus adjusted off-season estimates) related to 11 different clinical outcomes: all-cause mortality (n = 12 studies), death due to respiratory event (n = 2), major adverse vascular event (n = 1), hospitalization due to influenza and/or pneumonia (n = 7), hospitalization for acute coronary syndrome (n = 1), influenza-like illness (n = 3), cardiac death (n = 1), hospitalization due to cardiovascular disease (n = 1), prematurity (n = 1), small for gestational age (n = 1), and medically attended respiratory infections in infants (n = 1). None of the clinical outcomes was required to be confirmed by laboratory testing for influenza viruses.

Risk of healthy vaccinee bias and confounding by indication

Of the included 23 studies, 19 (83 %) showed a high risk of bias (either healthy vaccine bias, confounding by indication, or both). Two studies we judged to be at high risk of healthy vaccinee bias but not confounding by indication (Table 3). One of these studies was performed in patients with end-stage renal disease, with vaccinated participants having more favorable prognostic markers than unvaccinated participants; [23] the other study covered patients suffering from COPD and indicated that vaccinated patients had less (severe) comorbidities as indicated e.g., by the Charlson comorbidity index, when compared to unvaccinated patients.

Table 3 Risk of healthy vaccinee bias and confounding by indication in the included studies, as judged from the baseline characteristics of vaccinated and unvaccinated participants
Full size table

Fourteen studies showed a high risk of confounding by indication, but not of healthy vaccinee bias. In 13 of them, [3, 15, 16, 18, 19, 21, 22, 24, 28, 30–32, 34] this was indicated by a significantly higher proportion of vaccinated patients with comorbidities (compared to unvaccinated participants), whereas in one study [20] medical visits served as indicator. In three studies, we found indication for both types of bias/confounding occurring simultaneously [7, 17, 33]. In these studies, the group of vaccinated participants had a higher proportion of comorbidities, while at the same time unvaccinated participants showed a higher proportion of functional impairments or other relevant comorbidities. In a further three studies, [5, 25, 27] no major differences in baseline characteristics between vaccinated and unvaccinated study participants were found. In the remaining one study, risk of bias was unclear due to unclear data and reporting (Table 3) [26].

Adjustment for confounders and impact on point estimates

In ten of 12 studies reporting on all-cause mortality, adjustment for confounders increased the estimate of VE. The same effect of adjustment was observed in all studies reporting on hospitalization, major adverse vascular events, influenza-like illness and cardiac death. For the remaining outcomes, the effect was either very small or adjustment decreased the VE estimate. All studies adjusted at least for age and comorbidities, although definitions of the latter differed between individual studies (Table 4).

Table 4 Crude and confounder-adjusted estimates of vaccine effectiveness during the influenza season in the included studies
Full size table

We pooled the data for the outcomes all-cause mortality, hospitalization due to influenza or pneumonia, and ILI since more than one study reported on these outcomes. For all-cause mortality, this ratio of odds-ratio analysis indicated that adjustment for confounders increased the effect of vaccination by 12 % (95 % CI: 7–17 %) (Fig. 2a). For hospitalization due to influenza or pneumonia, effect size increased by 9 % (95 % CI: 4–14 %) after adjustment for confounders (Fig. 2b). For the outcome ILI, adjustment for confounders increased VE estimate by 7 % (95 % CI: 4–10 %).

Fig 2
figure 2

Impact of adjustment for confounders, expressed as ratio of odds ratios (crude/adjusted): (a) All-cause mortality, (b) Hospitalization due to influenza or pneumonia

Full size image

Off-season estimates

The included 23 studies reported a total 31 off-season estimates. Three of the studies reported pre-season as well as post-season estimates [7, 20, 35]. Two studies reported only pre-season estimates, [5, 23] while five studies provided data on post-seasons “effectiveness” only [15, 16, 18, 19, 32]. The remaining studies reported off-season estimates either for the whole period outside the influenza season or for single months before and after the seasons. Most studies defined beginning and end of influenza periods according to national influenza surveillance data. If more than one off-season estimate was provided, we decided to use the post-influenza season estimate for analysis (for a detailed description of the definition of “off-season” in the studies, see Additional file 4).

Analyzing the 31 adjusted off-season estimates that were reported by the 23 included studies, we found statistically significant effects of influenza vaccination outside the influenza season in 13 studies (Figs. 3 and 4). Nine (39 %) of the 23 included studies reported at least one statistically significant VE estimate outside the influenza season (Figs. 3 and 4). These off-season effects were not restricted to the outcome all-cause mortality, but were also reported for four other outcomes (major adverse vascular events, hospitalization due to influenza/pneumonia, acute coronary syndrome, ILI). However, significant off-season estimates were more likely to occur when all-cause mortality was used as an outcome (8/13; 67 %) compared to other outcomes (5/19; 26 %; p = 0.03 by chi2 test). We then evaluated whether the occurrence of significant off-season estimates was related to the risk of healthy vaccinee bias, as judged from the baseline data of the respective study populations. We found that 46 % (6/13) of the significant off-season estimates were associated with high risk of healthy vaccinee bias at baseline. In contrast, only 6 % (1/18) of non-significant off-season estimates were associated with high risk of healthy vaccinee bias (p = 0.01 by chi2 test). Studies covering non-elderly populations did not report statistically significant off-season estimates for neither outcome.

Fig. 3
figure 3

Odds ratios (95 % CIs) of influenza vaccine effectiveness during influenza seasons (black square), during pre-influenza seasons (striped circle) and post-influenza seasons (white circle) against all-cause mortality (a), death due to respiratory event (b), death due to cardiac event (c), and major adverse vascular event (d)

Full size image
Fig. 4
figure 4

Odds ratios (95 % CIs) of influenza vaccine effectiveness during influenza seasons (black square), during pre-influenza seasons (striped circle) and post-influenza seasons (white circle) against hospitalization due to influenza or pneumonia (a), hospitalization for acute coronary syndrome (b), hospitalization due to cardiovascular diseases (c), influenza-like illness (d), prematurity (e), small for gestational age (f), and medically attended respiratory illness in infants (g)

Full size image

Discussion

In this review, we systematically assessed the frequency and impact of two major forms of bias/confounding commonly found in observational studies assessing influenza vaccine effectiveness. Our analysis revealed that the majority of included studies showed evidence for confounding by indication, as judged from the baseline characteristics of vaccinated and unvaccinated study participants. Analysis of crude and adjusted estimates showed that statistical adjustment for confounders corrected for this form of bias, at least partially. However, despite adjustment, nearly half of the studies still showed significant estimates of vaccine effectiveness outside the influenza season, which indicates the presence of unmeasured confounding due to healthy vaccinee bias. Remarkably, significant off-season estimates were not only observed in studies on all-cause mortality, but also regarding other outcomes. However, all outcomes that were used in the included studies were only based on clinical criteria, none of the studies used outcomes with laboratory confirmation of the virus.

At population level, implausibly high mortality benefits of influenza vaccination have been observed particularly in elderly persons. Observational studies found a reduction of mortality of about 50 %, while it was estimated that influenza-related mortality attributed to less than 10 % in this age-group [2]. These and other observations led to the hypothesis of healthy vaccinee bias [36]. In healthy vaccinee bias, healthy persons are preferentially vaccinated against influenza, while persons with comorbidities have a lower likelihood to get vaccinated. A small subset of unvaccinated frail and terminally ill patients are suggested to explain the large/implausible results regarding mortality mentioned above. Adjustment for conventional comorbidities as confounders has been suggested to insufficiently capture the functional status of this subgroup [35]. In fact, in the majority of included studies comorbidities were identified through ICD-codes in administrative databases, which have been shown to fail to control adequately for confounding [37].

On the contrary, other authors have suggested the opposite form of bias/confounding to be present in observational studies on influenza vaccination. They concluded that patients with comorbidities are preferentially vaccinated against influenza, which reflects current recommendations by the World Health Organization (WHO) and several National Immunization Technical Advisory Groups (NITAGs), but might result in confounding by indication [6, 38]. Looking at the baseline characteristics of the included studies, we found that the majority of studies showed indication for this type of confounding rather than for healthy vaccinee bias. Remarkably, our meta-analytic approach showed that adjustment for comorbidities had only a small impact on the point estimate of VE. Although this procedure increased the VE estimates in the majority of studies, which is consistent with removal of confounding by indication, the effect size changed on average by only 7 to 12 %. However, since nearly all studies adjusted for comorbidities and other confounders such as sex and age simultaneously in one single step, it is unclear whether and to what extent this effect can be attributed to removal of confounding by comorbidities.

Interestingly, the analyses performed in the study by Campitelli et al. [7] showed that it is possible to adjust, at least in part, for both forms of bias/confounding, given that enough information have been collected regarding comorbidities and functional status of study participants. Those authors demonstrated that the addition of comorbidities as confounders to the regression model shifted the effect estimate away from 1.0, which indicates correction for confounding by indication. They then added indicators of functional status to the model and observed a shift of the estimate towards 1.0, indicating correction for healthy vaccinee bias. However, additional analyses performed in this study demonstrated that residual confounding was likely to be still present in those data since adjustment for comorbidities and frailty indicators could not eliminate significant off-season estimates.

Our systematic review shows that these findings can be generalized to the body of literature on this issue. In nearly half of the studies identified here, significant off-season estimates were observed despite adjustment for confounders. Although significant off-season estimates were more likely to occur in studies which showed high risk of healthy vaccinee bias at baseline, they were also observed in studies that did not find indication of healthy vaccinee bias by comparing the characteristics of vaccinated and unvaccinated study participants. Interestingly, in studies covering non-elderly participants’ significant off-season estimates were not identified. However, there were only five of these studies and it is unclear whether this could be attributed to a lower prevalence of comorbidities or frailty indicators in these age groups or whether this is a chance finding.

The significance of healthy vaccinee bias as well as the suitability of off-season estimates as indicators for its presence has been debated in several publications. Nichol et al. discussed that influenza vaccination is common in patients with functional impairments and frailty, [6] speaking against the assumption that a terminally ill and frail subgroup of patients is responsible for the observation of off-season estimates. Hak et al. suggested that circulation of influenza in the few months before and after the influenza season might account for “off-season” estimates, as well as a prolonged impact of influenza on mortality which extends several months beyond illness [4]. On the contrary, the publication by Wong et al. [32] provided additional evidence that off-season estimates result from healthy vaccinee bias for which the conventional analysis failed to adjust for. Those authors used the same data base that was primarily analyzed as a cohort study to apply instrumental variable technique. Using this study design, they were able to show that quasi-randomization eliminates off-season effects of influenza vaccination, supporting the interpretation that study design and data analysis are crucial here.

The recent debate on bias in influenza VE studies mainly focusses on the outcome all-cause mortality [2, 6, 10, 35]. Our systematic review demonstrates that significant off-season estimates were also observed in the context of three other clinical outcomes, although significantly less frequent than in mortality studies. All of these outcomes have in common that they are based on unspecific case definitions without laboratory confirmation of influenza infection, which is likely to lead to outcome misclassification. It should be evaluated in future studies whether significant off-season VE estimates are still present when influenza-specific outcomes with laboratory confirmation are assessed.

Our study has several strengths. It is the first systematic review which focused on this issue and examined all published studies with relevant data for the assessment of these two types of bias. In addition, we quantified the extent to which adjustment could correct for confounding by indication regarding different clinical outcomes. However, some limitations of our study have to be addressed although they are mainly caused by limitations of the included studies: First, a number of studies could not be included since they did not provide enough information to assess risk of bias. Some of them included also more specific endpoints with laboratory confirmation. For this reason, the proportion of studies with such biases might be an overestimation. Second, as it is often the case in administrative database-related studies, multiple groups of authors used the same data base and potential overlap between study populations cannot be completely excluded. We detected for example potential overlap between the studies by Campitelli et al. [7] and Wong et al. [32]. Third, since studies used different covariates for confounder-adjusted VE estimates and different definition of influenza and off-season periods, direct comparison of the results have to be taken with caution. Furthermore, in nearly all studies statistical adjustment was made in multivariate analysis for a variety of confounder simultaneously. Those sets of confounders did not only include comorbidities, but also age, sex and demographic characteristics. Therefore, the adjusted odds ratios used for our analysis do not accurately reflect confounding by indication. Finally, other types of bias, such as errors in diagnosis or vaccination status, could also have influenced study findings but were not in the focus of our analysis.

Conclusions

To conclude, this systematic review supports the hypothesis that confounding by indication and healthy vaccine bias operate simultaneously in observational studies on influenza vaccination using unspecific outcomes. Consequently, it seems impossible to infer whether the adjusted vaccine effectiveness estimates under- or overestimate the true effect of the vaccine. Cohort study designs using administrative data bases with unspecific outcomes such as all-cause mortality should no longer be used to measure the effects of influenza vaccination. Instead, other study designs, including test-negative design studies [39] and quasi-randomized studies using influenza-specific laboratory-confirmed outcomes, are needed to obtain more reliable estimates of influenza vaccine effectiveness. However, one should be aware that in these study types other forms of bias might operate. This should be assessed in further methodological studies.