Introduction

Several phase III trials on vaccines against SARS-CoV-2 are ongoing and initial results highly promising. A major issue of these phase III trials is to what extent the included study population is representative of the intended (or target) population, i.e. external validity.

For these trials, the intended target population is initially comprised of high-risk individuals usually considered to be elderly persons as well as those with comorbidities, and ultimately the entire world population. It remains unclear whether these target populations are representatively recruited into ongoing trials. This information is pivotal, since clinical recommendations for any approved vaccine should incorporate the proper target populations for which these vaccines have shown efficacy, and also determine those populations not sufficiently represented in the trials.

We sought to quantify the external validity of the various ongoing trials to a middle-aged and elderly West-European population from the Rotterdam Study. Specifically, we were interested to quantify what proportion of this study population would be eligible to participate in these trials and how many of those eligible are from high-risk categories.

Methods

Extensive methods are available in the Online Resource.

Briefly, for this study we screened www.clinicaltrials.gov for ongoing phase III trials focused on vaccine development against SARS-CoV-2 and COVID-19.

The group at high risk of severe COVID-19 was defined according to the criteria of the Dutch National Institute for Public Health and the Environment (RIVM) [1], and included participants aged 70 or higher, and participants with asthma and COPD, diabetes, cancer, participants with current use of antineoplastic and immunosuppressive agents, obesity, end-stage kidney disease, liver steatosis and cirrhosis and cardiac diseases.

We carried out our analyses in the population-based Rotterdam Study [2] and used data collected from 2009 to 2014, which yielded 7162 persons (mean age 70 years (SD 9.8), 58% women) for analysis (Online Resource Table 1). These calendar-years were chosen such to maximize the number of living participants as well as their available data. Data on the comorbidities was ascertained during the in-person examinations complemented by automated linkage of medical and pharmacy records to our study database.

We applied eligibility criteria from each separate trial to our study population and calculated the following proportions: the proportion of participants eligible for any trial, and for each trial separately, the number of high-risk individuals in our study eligible for any trial, and for each trial separately.

We performed two complementary analyses and calculated the abovementioned proportions in each analysis separately. These two analyses differed with respect to the interpretation of an eligibility criterion that was not always explicitly specified in the various trial protocols. This criterion was often stated as follows: ‘preexisting (un)stable disease’, ‘an acute course of disease’, or ‘other medical or psychiatric condition or laboratory abnormality that may increase the risk of study participation or, in the investigator’s judgment, make the participant inappropriate for the study’. In our dataset, we operationalized this criterion as follows: diagnosis of dementia, diagnosis of moderate to severe COPD, current clinically significant depressive symptoms, abnormal kidney function, current liver disease (defined as liver steatosis, and liver cirrhosis) or a new diagnosis within the preceding three months for the following conditions: stroke, cancer, (including antineoplastic agents), diabetes mellitus, COPD, cardiac disease (heart failure, myocardial infarct, atrial fibrillation, and revascularisation). In the first analysis, we included everyone as eligible, who met this operationalization and in the second analysis, we excluded anyone who met this operationalization. The general characteristics of individuals with or without acute or unstable disease are presented in Online Resource Tables 2 and 3).

Finally, in sensitivity analyses we incrementally restricted the study population to persons aged over 60, 70, and 80 years (Online Resource Fig. 1).

Fig. 1
figure 1

Venn diagram for the proportion of eligible Rotterdam Study participants for any trial. The colours are coded as follows: blue circle: total study population; yellow circle: persons at high risk of severe COVID-19; red circle: persons eligible for any trial in the first analysis; red dotted circle: persons eligible for any trial in the second analysis, in which individuals with acute course of disease were excluded

Full size image

Results

Table 1 presents the exclusion criteria from the 9 included trials, and we note that seven of these mentioned the eligibility criterion of ‘acute’ or ‘unstable preexisting’ disease without further specification. Therefore, we performed two analyses to calculate the proportion of the eligible participants for each trial. In the first analysis, we included all eligible participants, including those who met our operationalization of the eligibility criterion of ‘acute’ or ‘unstable preexisting’ in the seven trials that mentioned this criterion without further specification. In the second analysis, we repeated the first analysis but now excluding those who met our operationalization of ‘acute’ or ‘unstable preexisting’ criterion.

Table 1 Overview of the included trials with the selection of exclusion criteria per trial, for which data from the Rotterdam Study was available
Full size table

97% (N = 6945) of the total Rotterdam Study participants would be eligible for any trial in the first analysis, while this percentage dropped to 43% (N = 3107) in the second analysis. Among the 5781 participants at high-risk of severe COVID-19, 96% (N = 5564) would be eligible for any trial in the first analysis and 36% (N = 2102) in the second analysis (Fig. 1).

Figure 2a shows the percentages for the two analyses for each trial separately. Whereas in the first analysis the proportion included for the most inclusive trial was 97%, this number dropped considerably in the second analysis to 43%. Figure 2b shows the corresponding numbers from participants at high-risk of severe COVID-19. Finally, sensitivity analyses revealed similar patterns at various age cut-offs (Online Resource Fig. 1).

Fig. 2
figure 2

The number and proportion of eligibility from the Rotterdam Study population for ongoing clinical phase III trials on a vaccine against SARS-CoV-2. White bars indicate data from the first analysis; black bars indicate data from the second analysis. The difference between these two analyses is the operationalization, and thus inclusion or exclusion, of the eligibility criteria ‘(un)stable disease’, ‘acute course of disease’, or ‘other condition increasing risk of participation’. a total population of the Rotterdam Study and b individuals at high risk of severe COVID-19

Full size image

Discussion

In a middle-aged and elderly population in the Netherlands from predominantly West-European descent, we found that 97% of this population would be eligible to participate in any of the nine currently ongoing vaccine trials against SARS-CoV-2. For persons at high-risk of severe COVID-19, the eligibility for any trial was 96%. Importantly, applying stricter exclusion based on the criterion ‘acute’ or ‘unstable preexisting’ disease drastically reduced the eligibility for any trial to 43% of the entire study population and 36% of the high-risk individuals.

Clinical trials are often considered the golden standard in efficacy research, due to several strengths by design with respect to internal validity. In contrast, whether findings from clinical trials are adequately translated to clinical practice also depends on their external validity. External validity is the extent to which findings from one study are applicable to target populations not represented in the actual study population. To properly gauge the actual study population, it is crucial that clinical trials explicitly report the setting of the trial, the exact intervention, the inclusion and exclusion criteria, and the characteristics of the actually recruited population [3, 4] in [5]. For the ongoing clinical trials, any effective vaccine can be considered effective in similar settings as the original trial. The judgement whether that vaccine will be also effective in other settings, i.e. external validity, is based on prior knowledge, biological plausibility, statistical considerations and eligibility criteria of the original trial [3]. Many countries worldwide have prioritized the elderly and those most vulnerable in their vaccination campaigns, indicating that policy-makers consider the external validity of the ongoing trials to these populations sufficient.

The population of the Rotterdam Study is a lower-middle class population of primarily European descent. Previously, this population has shown good generalizability to the population of the Netherlands [6,7,8]. Another important metric in this regard is the response rate, which has continuously exceeded 70% for the Rotterdam Study [2]. This is a major strength of our study which makes it not only population-based but also population-representative, and far exceeds response rates for other larger efforts ongoing worldwide [9]. Notwithstanding these considerations, the drastic drop in eligibility for the stricter criterion in the second analysis would likely have been of the same magnitude in any other population, irrespective of geographical or ethnic setting or response rate.

In conclusion, we found that eligibility for ongoing vaccine trials against SARS-CoV-2 can reduce by half depending on interpretation and application of a single unspecified exclusion criterion. This exclusion criterion in our study would especially affect the elderly and those with pre-existing morbidities. These findings thus indicate the difficulty as well as importance of developing clinical recommendations for vaccination and applying these to the appropriate target populations. This becomes especially paramount considering the fact that many countries worldwide have initiated their vaccination programs by first targeting the elderly and most vulnerable persons.