Background

Chalmers and Glasziou estimated that 85% of clinical research is wasteful, with 50% of studies having design or major methodological weaknesses [1]. In these companion reports, we assessed research waste in a single field - calcium and vitamin D research. In the first report [2], we focused on redundant research characterized by the undertaking and publication of uninformative observational studies and randomised controlled trials (RCTs) with surrogate endpoints long after the need for large RCTs with ‘hard’ clinical endpoints was established. In this second report, we address waste characterised by conducting RCTs in poorly targeted population groups.

Clinical guidelines disagree on the serum 25-hydroxyvitamin D (25OHD) concentrations that constitute vitamin D sufficiency. The Institute of Medicine recommends ≥50 nmol/L to ensure adequate 25OHD for 97.5% of the population, with a median target value of 40 nmol/L [3], whereas the Endocrine Society recommends ≥75 nmol/L [4]. However despite this disagreement, there is general agreement that 25OHD < 25 nmol/L indicates deficiency, and recent UK guidance on vitamin D supplementation is based on maintaining 25OHD above this threshold [5]. Mildly low 25OHD is often termed vitamin D insufficiency, and moderately low 25OHD vitamin D deficiency. Throughout the text, we have used vitamin D deficiency to refer to 25OHD < 25 nmol/L, and insufficiency to 25OHD < 50 nmol/L [6]. Intuitively, supplementing populations that are vitamin D deficient is more likely to produce benefits than supplementing populations with higher 25OHD [7]. Potential benefits of vitamin D could be obscured if a high proportion of participants in RCTs are vitamin D sufficient. Thus, RCTs in cohorts that are vitamin D sufficient are unlikely to show benefits of vitamin D supplementation and could be considered research waste. Recent systematic reviews of RCTs of vitamin D supplementation have not shown benefits on skeletal or non-skeletal endpoints [8,9,10,11]. We set out to determine what proportion of RCTs of vitamin D supplementation with clinical endpoints has been conducted in vitamin D deficient populations, and whether baseline 25OHD in such RCTs have changed over time. We then focused on large RCTs either already completed or in progress, identified data on target population vitamin D status available prior to the trial, and determined whether the investigators reported relevant data on vitamin D status. We also determined whether investigators reported relevant systematic reviews in their trial justification, as recommended [1, 12].

Methods

Literature searches

In December 2015, we searched Pubmed for RCTs of vitamin D in adults (>18y) (Additional file 1: Table S1) and for recent systematic reviews on clinical conditions and major surrogate endpoints that were the primary endpoints in identified RCTs (Additional file 1: Tables S2 and S3). We included trials with an untreated or placebo group, trials comparing different vitamin D doses, trials with or without calcium supplements, and trials with multiple interventions provided that 2 study arms differed only by the use of vitamin D. We included quasi-randomized trials but excluded trials where the method of allocation was sequential or unreported, trials where vitamin D was administered routinely post-thyroidectomy, and trials of hydroxylated vitamin D analogues. The flow of articles is shown in Additional file 1: Figure S1.

In December 2015, we also searched ClinicalTrials.gov (https://clinicaltrials.gov/), the International Standard Randomised Controlled Trial Number (ISRCTN) registry (http://www.isrctn.com/) and the Australian New Zealand Clinical Trials Registry (ANZCTR) (http://www.anzctr.org.au/) for completed and ongoing trials, using vitamin D as the search term.

Finally, we obtained vitamin D status survey data from published systematic reviews [13,14,15,16,17]. supplemented by Medline, Embase, and Google searches using our vitamin D search strategy and text words for the countries of interest, and checked citations in reference lists.

Trial classification

We categorised each RCT according to whether clinical or surrogate endpoints were reported in the Abstract (or full-text where there was no Abstract), using the Institute of Medicine definition of surrogate outcomes as “biomarker[s] intended to substitute for a clinical endpoint [and] expected to predict clinical benefit (or harm. ..) based on epidemiologic, therapeutic, pathophysiologic, or other scientific evidence” [18]. Where multiple endpoints were reported, we recorded the most relevant clinical endpoint, and if there were no clinical endpoints, the most clinically relevant surrogate endpoint. Where there were multiple publications from the same RCT, we included the study with the most relevant clinical endpoint or the most clinically relevant surrogate endpoint.

Vitamin D status survey data

For large (N ≥ 1000) completed and ongoing RCTs, we identified surveys of vitamin D status undertaken in the same country and most similar population group, based on age and sex, prior to the trial being undertaken. We preferentially sought data from the five years before trial inception or 10 years before trial completion/publication, but used older data if we could not locate such data.

Analyses

A single author (MB or AA) extracted relevant data. One author (MB) classified trials as having clinical or surrogate endpoints, and a second author (AG) checked the classifications. We report the proportions of trials with mean/median baseline 25OHD < 25, 25–49, 50–74, ≥75 nmol/L over time. In trials with mean/median baseline 25OHD < 25 nmol/L and trials that reported a subgroup analysis based on baseline 25OHD, two authors (MB, AG) independently assessed whether the results of the trial or subgroup analysis were beneficial, neutral, or harmful and disagreements were resolved by consensus.

We examined primary trial publications, and trial protocols (where available), for large RCTs (N ≥ 1000) and assessed whether trial investigators discussed prior relevant evidence on vitamin D status of the intended trial population in their justification for the trial. We also examined whether trial investigators discussed systematic reviews of randomised trials relevant to the primary endpoint that were available before trial recruitment commenced in the Introduction section of the primary publication.

Early 25OHD competitive binding protein (CBP) assays overestimated 25OHD concentrations [19]. As an approximation, we used an adjustment factor of 0.54 for CBP assays in papers published before 2000 [19]. and 0.76 to adjust for overestimation from the Nicholls’ immunoassay [20]. We have presented the RCT and survey data in Tables 3 and 4 corrected for these overestimations.

Results

Baseline 25OHD in randomised controlled trials

From 4682 unique Pubmed records and 38 systematic reviews, we identified 779 publications from 547 RCTs of vitamin D, of which 137 (111,976 participants) reported a clinical endpoint in the Abstract (Additional file 1: Tables S1, S2, S3 and Figure S1). Figure 1a shows that the rate of publication of RCTs has increased markedly, with 11 RCTs in 2001–5, 28 in 2006–10, and 88 in 2011–15. Mean/median baseline 25OHD was reported in 118/137 (86%) RCTs (Fig. 1b), with 62%, 82%, and 93% of RCTs reporting baseline 25OHD before 2006, in 2006–10, and in 2011–15 respectively. Overall, mean/median baseline 25OHD was < 25, 25–49, 50–74, and ≥ 75 nmol/L in 12 (10%), 62 (53%), 36 (31%), and 8 (7%) RCTs, respectively. In 70% of RCTs, baseline 25OHD was > 40 nmol/L. Of 12 RCTs with baseline 25OHD < 25 nmol/L, 8 had neutral findings (Table 1).

Fig. 1
figure 1

Panel a shows the number of randomized controlled trials (RCTs) of vitamin D with clinical endpoints in the Abstract published over time by year (bars) and cumulatively (line). Panel b shows the distribution of mean/median baseline 25-hydroxyvitamin D (25OHD) concentrations in these RCTs. Panel c shows the 25OHD concentrations plotted against year of publication with a line of best fit. Panel d shows the proportion of trials with mean/median baseline 25OHD < 25, 25–49, 50–74 and ≥ 75 nmol/L by year of publication. Above each bar is the number of trials

Full size image
Table 1 Characteristics of 12 randomised controlled trials of vitamin D supplements in populations with mean/median 25OHD < 25 nmol/L and clinical endpoints reported in abstract
Full size table

Figure 1c and d show that mean/median baseline 25OHD has increased over time. Before 2006, 38% of RCTs had 25OHD < 25 nmol/L, 62% between 25 and 49 nmol/L, and none ≥50 nmol/L. In 2006–10 and 2011–15, 0% and 9% respectively of RCTs had 25OHD < 25 nmol/L, while 61% and 49% respectively had 25OHD 25–49 nmol/L, 26% and 37% respectively had 25OHD 50–74 nmol/L, and 13% and 6% respectively had 25OHD ≥75 nmol/L.

Of 118 RCTs that reported mean/median baseline 25OHD, 19 (16%) reported a subgroup analysis for baseline 25OHD (Table 2). The 25OHD thresholds used in these analyses ranged from 20 to 80 nmol/L, with 5 analyses based on thresholds ≤25 nmol/L and 16 on thresholds ≤50 nmol/L. Table 2 shows that 17 RCTs reported similar results in the subgroup analysis and the main analysis for the primary endpoint (16 both analyses neutral, and 1 both analyses showed benefit for vitamin D). Three of these 17 RCTs reported a benefit for vitamin D for a secondary endpoint in a subgroup analysis. Another RCT did not report the result of the subgroup analysis for the primary endpoint, but reported a benefit for vitamin D for a secondary endpoint. Lastly, one RCT had co-primary endpoints and neutral results in the main analyses, but in the subgroup analyses there was a benefit for vitamin D for one endpoint and neutral results for the other. Four of the five RCTs that reported subgroup analyses with a 25OHD threshold of ≤25 nmol/L had neutral results for the primary endpoint in the main analysis, and none of these 4 RCTs reported beneficial effects for the primary endpoint in the subgroup analysis.

Table 2 Results of 18 randomised controlled trials of vitamin D supplements reporting subgroup analyses for baseline 25-hydroxyvitamin D
Full size table

Large randomised controlled trials and prior 25OHD surveys

Tables 3 and 4 show 18 published RCTs of vitamin D with ≥1000 participants (101,383 participants), and 7 planned and ongoing trials (79,939 intended participants). We included the pilot stage for the UK VIDAL trial with 1600 participants, which aimed to continue and recruit 20,000 participants, but has not yet received funding for the full roll out. All trials were/are conducted exclusively in North America, Europe, Australia or New Zealand, except for two multinational trials with countries from South America, Asia and Africa. 22/25 trials were in single countries: we did not examine prior 25OHD surveys for the 3 multinational trials.

Table 3 Large randomised controlled trials of vitamin D supplements with relevant prior 25-hydroxyvitamin D surveys
Full size table
Table 4 Planned and ongoing large randomised controlled trials of vitamin D supplements with relevant prior 25-hydroxyvitamin D surveys
Full size table

Table 3 shows that only 8 [21,22,23,24,25,26,27,28] of the 18 completed trials (44%) cited the vitamin D status of a population similar to the recruited cohort in the primary publication. One further trial [29] discussed survey data in the trial paper’s introduction, but this was not used in the grant application. Investigators from two of these trials [21, 22] had undertaken prior relevant 25OHD surveys. Four of the eight trials cited old survey data, from at least 16 years [24, 27] and 6–9 years [23, 26] before trial recruitment. A trial from Finland that studied older adults (mean age 62y) cited survey data that lacked relevance, being from the USA and from young Finnish adults (mean age 38y) [30].

Table 4 shows that all four ongoing trials with accessible documents discuss the vitamin D status of their intended trial population. One trial in Australia conducted a pilot study that included assessment of vitamin D status. The US VITAL trial which started recruitment in 2010, used NHANES III (1988–94) data in its rationale and design paper justification [31, 32].

Table 3 shows that baseline 25OHD in large completed RCTs and relevant survey 25OHD data were comparable, apart from one Norwegian trial, where one survey indicated considerably worse vitamin D status than was observed in trial participants [23]. Only one [21] of the completed trials was conducted in a population that was clearly vitamin D deficient, based on trial (mean baseline 25OHD 20 nmol/L) and survey data (mean 11–23 nmol/L). Three trials [22, 24, 29] were undertaken in populations comprised largely of participants with vitamin D insufficiency. Of the remaining 13 single country trials with baseline 25OHD or relevant survey data, five trials [23, 27, 33,34,35] had mean baseline 25OHD ≥40 nmol/L and four trials 25OHD ≥50 nmol/L [26, 28, 30, 36]. Four trials [25, 37,38,39] did not report baseline 25OHD, but surveys and data from similar RCTs suggest that baseline 25OHD in the RCT would have been ≥40 nmol/L in three of these trials [25, 37, 38]. In these 13 trials, a substantial proportion of participants would have had 25OHD ≥50 nmol/L, consistent with the IOM definition of vitamin D sufficiency [3].

Table 4 shows that, based on survey data from the relevant population, all the ongoing single country trials are likely to recruit participants in whom the mean/median baseline 25OHD will be > 40–50 nmol/L, and none describe specific strategies for recruiting participants with 25OHD < 25 nmol/L.

Large randomised controlled trials and citation of prior systematic reviews of randomised controlled trials

We identified a relevant systematic review on vitamin D and fracture [40] published prior to trial recruitment starting for 8 completed large RCTs, and on mortality [41] for 2 RCTs, but no prior systematic reviews on colorectal adenoma or neonatal bone mineral content for two RCTs (Table 3). Thus, systematic reviews capable of informing the trial justification and design were available before trial recruitment in 10/18 (56%) of completed large RCTs. Only three [28, 29, 42] of the 10 RCTs (30%) cited such a systematic review in their primary publication. Nine trials (50%) cited systematic reviews that would have occurred after the decision had been made to undertake the trial. Four of the seven planned or ongoing trials with accessible relevant documents discuss systematic reviews in their protocols or publications: for three of the trials, the systematic reviews predate trial recruitment.

Discussion

Our results suggest a high proportion of research waste in RCTs of vitamin D supplementation. The recent proliferation of vitamin D RCTs was accompanied by increasing baseline 25OHD concentrations and therefore a declining proportion of RCTs conducted in vitamin D deficient cohorts. Only 10% of trials were carried out in populations that would be widely accepted as vitamin D deficient, in which benefits of vitamin D supplementation still have not been unequivocally established (Table 1). Because many participants in recent trials were vitamin D sufficient, they would be unlikely to benefit from vitamin D. Further, their inclusion could have obscured potential benefits from vitamin D for those participants who were vitamin D deficient. This issue applies to RCTs with mean/median baseline 25OHD in ranges variously defined as sufficient (7% with 25OHD ≥75 nmol/L, 37% with 25OHD ≥50 nmol/L). It likely also applies to the 33% of trials with baseline 25OHD 40–49 nmol/L in which a substantial proportion of participants will have had 25OHD ≥50 nmol/L. Thus, 7–37% of trials can be considered research waste, because they were conducted in the wrong population, but this proportion is as high as 70% if a 25OHD threshold for sufficiency of 40 nmol/L was applied, based on the Institute of Medicine’s target median value.

Very importantly, research waste was prevalent in large RCTs that were designed to inform clinical practice. Only 1 such trial was carried out in a vitamin D deficient population and another 3 in populations with vitamin D insufficiency. Twelve (71%) of the remaining completed and 5 (100%) ongoing single country trials had, or are likely to have, mean baseline 25OHD > 40 nmol/L, and based on survey and other trial data, we estimate that about 50% would have 25OHD ≥50 nmol/L. Failure to incorporate key available data during protocol development may have contributed to the high prevalence of waste. Few (44%) of the large completed RCTs cited or undertook prior relevant surveys of vitamin D status in their intended trial population. Only 56% of large completed RCTs had a relevant systematic review of randomised trials published prior to trial recruitment starting and, of these, only 30% cited such a review. When systematic reviews of randomised trials were discussed, they tended to have been published after the trial had commenced or been completed. Collectively, this suggests that these large, costly RCTs were not optimally designed to address the question of benefits of vitamin D supplements.

An important strength of this study assessing research waste is that we analysed the complete set of RCTs of vitamin D published over 30 years. The results from this single research area might not apply to other research fields, and waste may be more prevalent in mature as opposed to emerging areas of research. In assessing whether trials cited 25OHD surveys or relevant systematic reviews, we examined primary publications and protocols where available. Our results may have changed if we were able to examine grant applications and trial protocols, but protocols were often not available, and we had access to only one grant application [29]. Early 25OHD assays tended to overestimate 25OHD- we used 25OHD concentrations corrected for these overestimates. The corrected values are approximations, but nevertheless lower than the original values in the relevant trials and surveys, and therefore the proportions of participants with vitamin D deficiency in our analyses are higher than in the original publications. Very few RCTs reported the season when 25OHD measurements were obtained. Although seasonal changes in 25OHD will occur in all treatment arms, it is possible that seasonal effects of 25OHD might confound some trial results. A limitation of this study is that the literature search was conducted in December 2015.

The implications of this research are that the current body of RCTs of vitamin D with clinical endpoints, including large RCTs with ≥1000 participants, is largely conducted in populations that are not vitamin D deficient. Recent, large systematic reviews of these RCTs report no benefits of vitamin D [8,9,10,11]. In trials included in these meta-analyses reporting 25OHD, 72–75% had baseline 25OHD < 50 nmol/L [10, 43], consistent with Fig. 1d showing that the majority of trials prior to 2011 had baseline 25OHD < 50 nmol/L. Thus, it is reasonable to conclude that current evidence is sufficient to exclude benefits from vitamin D supplementation for unselected community-dwelling individuals with 25OHD > 30–40 nmol/L. Relatively few trials, including only 5003 participants (Table 1), have been carried out in populations with lower baseline 25OHD and their results are inconsistent, with only 33% of such trials reporting beneficial results from vitamin D. Subgroup analyses of participants with lower 25OHD at baseline were frequently undertaken but their results were invariably similar to the results of the main analyses for the primary endpoint, even when the subgroup was restricted to people with 25OHD ≤25 nmol/L. Therefore, it is uncertain whether vitamin D supplementation benefits people with clearly low 25OHD. Based on data from relevant 25OHD surveys, the large RCTs currently underway will not test the effects of vitamin D supplementation in deficient populations and therefore are unlikely to address this knowledge gap. Instead of continuing to spend resources on trials in vitamin D sufficient populations, investigators should focus on vitamin D deficient populations. Food fortification policies [15, 16], together with independent action by food manufacturers and new advice on supplementation [5], make it even less likely that future trials in deficient populations will be possible.

Our analyses suggest that up to 70% of RCTs with clinical endpoints, 71% of large (N ≥ 1000) completed RCTs, and 100% of ongoing large RCTs could be considered research waste because they studied cohorts with a high proportion of vitamin D sufficiency. In our companion paper [2], we reported that 69% of RCTs of vitamin D conducted since 2005 with skeletal endpoints of bone mineral density or fracture were research waste because they lacked novelty or did not add to existing clinical knowledge. Taken together, these findings support the very high proportions (> 85%) for research waste estimated by Chalmers and Glasziou [1].

Conclusions

We identified a very high proportion of research waste in RCTs of vitamin D with clinical endpoints. Few RCTs were carried out in vitamin D deficient populations most likely to benefit from vitamin D supplementation, and conversely most RCTs were carried out in populations unlikely to benefit from supplementation. Few large RCTs appeared to consider systematic reviews in their design. Ongoing large RCTs share the same weaknesses of previous trials. Strategies to improve the design of RCTs should be introduced and studied to determine whether they can reduce research waste.