Background

Dietary factors are the main risk factor for chronic diseases, especially for type 2 diabetes and cardiovascular diseases, according to the Global Burden of Disease Study [1]. Decision-making in the context of diet-related diseases should be informed by the most reliable available research evidence. Thus, dietary guidance is frequently derived from systematic reviews (SRs) of randomised controlled trials (RCTs) [2]. RCTs are considered to provide the most trustworthy effect estimates of dietary interventions and are the gold standard for establishing causal relations between dietary exposures (e.g. nutrients, foods, or dietary patterns) and health outcomes (e.g. event rates or intermediate disease markers) [3].

Limitations in a RCT, such as in the design, conduct, and analysis of the study, can bias the effect estimates resulting in an overestimation or underestimation of the true intervention effect [4]. This can potentially lead to suboptimal decision-making in nutrition research, e.g. implementing or not a specific dietary intervention for a certain disease. Thus, it is important for the authors of SRs to assess the internal validity of the included primary studies by evaluating the risk of bias (RoB) and incorporating the findings into their statistical analysis, GRADE (Grading of Recommendations, Assessment, Development and Evaluation) assessment and conclusion [5].

The Cochrane tool [6] is the most widely used tool to assess RoB in evidence synthesis in medical and nutrition research [7]. It guides review authors through six pre-defined domains (and methodological characteristics) that need to be considered for the RoB judgement: selection bias (random sequence generation and allocation concealment), performance bias (blinding of participants and personnel), detection bias (blinding of outcome assessment), attrition bias (incomplete outcome data), reporting bias (selective reporting), and other bias [6].

Instruments to critically appraise RCTs are based on evidence from meta-epidemiological studies. To date, several meta-epidemiological studies have investigated the influence of reported study design characteristics on intervention effect estimates in the medical field. They found that trials with inadequate sequence generation and allocation concealment and lack of blinding of participants and outcome assessors yielded more beneficial estimates of intervention effects [8,9,10]. However, to the best of our knowledge, no such systematic evaluation has been conducted for RCTs in the nutritional field.

To close this important research gap, this meta-epidemiological study aims to synthesise evidence on the average bias associated with seven commonly reported methodological trial characteristics in nutrition research. Moreover, we aim to determine whether average bias estimates are relatively similar or not across meta-analyses addressing different interventions, and outcomes. The findings from this study will allow us to better understand and explore the impact of trial characteristics on effects estimates from nutrition RCTs.

Methods

This meta-epidemiological study adheres to the guidelines for reporting meta-epidemiological methodology research [11].

Search strategy and data selection

The sample consists of 33 SRs of RCTs which were identified through a previous meta-epidemiological study [12]. For this project, we searched for SRs of RCTs in the Cochrane Database of Systematic Reviews published between 01 January 2010 and 31 December 2019, which investigated the effects of nutritional interventions (e.g. dietary pattern, micronutrients) on patient-relevant outcomes (e.g. mortality, cardiovascular disease [CVD]). Details of the inclusion and exclusion criteria, as well as the search strategy, are displayed in Additional file 1: Appendix S1-S2.

For each identified Cochrane review, we chose a maximum of six outcomes (maximum three binary and three continuous outcomes) for each given intervention, based on the ranking in the Summary of Findings tables (from top to bottom). Furthermore, we selected only the meta-analyses that contain at least one trial judged as ‘low risk of bias’ and another for ‘unclear or high risk of bias’ for a specific methodological characteristic according to the RoB assessment provided by the original SR authors.

Data extraction

We extracted the following data for each included Cochrane review: name of the first author, year of publication, type of intervention, description of outcome, effect estimates (risk ratio [RR] or mean difference [MD] with the corresponding 95% confidence interval [CI]), and the number of included RCTs. For continuous outcomes, we calculated the standard deviations [SD], and the effect measures were converted to standardised mean differences [SMD] with the transformation formula SMD = MD/SD.

For each RCT included in the eligible meta-analyses of each considered Cochrane review, we additionally extracted all effect estimates with their corresponding 95% CI and the judgements (low, unclear, high) for each methodological trial characteristic of the Cochrane tool [6] (random sequence, generation and allocation concealment, blinding, incomplete outcome data, and selective reporting) as provided by the original SR authors. Although dietary compliance was not suggested in the RoB domains of the Cochrane tool, we extracted judgements if available, due to its importance for nutrition RCTs [13]. Judgements for each methodological trial characteristic were extracted from the risk of bias summary figure and/or the ‘characteristics of studies’ tables in the appendix.

Two authors extracted data using a piloted standardised data collection form. Data extraction was performed by one reviewer (IR) and checked by a second reviewer (JS). Divergences were resolved by consensus, or by consulting a third reviewer (LS).

Statistical analysis

We standardised the direction of effect of the outcomes so that binary effect estimates < 1 (or continuous outcomes < 0) are always expressing a beneficial effect.

Relative effect estimates (quantified as the ratio of risk ratios [RRR] or difference in standardised mean differences [DSMD]) between RCTs with high/unclear and those with low RoB were measured for each methodological trial characteristic. To do so, we re-calculated the effect estimate [RR or MD] for each group of trials included in the same meta-analysis, by pooling first only those RCTs which were judged to be at low RoB according to the methodological trial characteristic under investigation and second only those RCTs with an unclear or high RoB judgement (see Additional file 2: Fig. S1). By using RCTs with low RoB judgements as the reference group, we examined the pooled estimate to see whether there was a relatively larger or smaller estimate coming from RCTs with high/unclear RoB judgements. For example, a RR from high/unclear RoB RCTs of 0.95 and a RR from low RoB RCTs of 0.90 result in a RRR of 1.06, whereas a RR of 1.00 in low RoB RCTs compared to a RR of 1.06 in high/unclear RoB RCTs is also a RRR of 1.06. Therefore, the RRRs should not be interpreted as larger or smaller treatment effects in one RoB category, but only as differences between the two RoB categories, and the direction of difference depends on the direction of effect of the underlying RCTs.

RCTs with high and unclear RoB judgements were grouped together, since the number of available judgements was small (Table 1). This approach was in line with landmark meta-epidemiological studies [8, 10, 14]. We performed meta-analyses to synthesise the relative effect estimates (average bias). Statistical analyses were conducted across all outcomes and for each methodological characteristic independently. When methodological trial characteristics were defined differently across the Cochrane reviews (e.g. ‘blinding’ in general), we excluded these from the analyses. We performed subgroup analyses with respect to the different types of interventions (e.g. dietary approach, fatty acids, micronutrients), and types of outcomes (e.g. mortality, cardiovascular disease, cancer). Moreover, we performed subgroup analyses comparing the objective (e.g. all-cause mortality) versus pregnancy outcomes versus mostly subjective outcomes according to Savovic and colleagues [9, 10].

Table 1 Number of randomised controlled trials included in the data set, by risk of bias judgement
Full size table

We also performed a sensitivity analysis excluding the highly correlated outcomes (Additional file 3: Tables S1-S7). The determination of highly correlated outcomes was based on experts’ opinions and is in line with our previous study [12]. For each intervention, we only chose the outcome that was mentioned first from the Summary of Findings tables of the identified Cochrane reviews. For example, for the intervention of α-linolenic acid, the outcomes CVD, CVD mortality, and coronary heart disease are probably highly correlated. CVD mortality outcome was chosen as it was mentioned first in the Summary of Findings table, while the other two outcomes were excluded. In a post hoc analysis, we conducted a sensitivity analysis including only one outcome per comparison from each included SR. We chose the outcome with the largest number of RCTs included. In case there were two outcomes with the same number of RCTs, we chose the one that was ranked higher in the Summary of Findings table.

We used the random-effects model in meta-analysis to take into account the between-study variability (heterogeneity). We assessed heterogeneity with I2 and τ2 measures [15, 16]. The τ2 statistic was estimated by the restricted maximum-likelihood (REML) method [17], which performs well for both binary and continuous outcomes [18,19,20]. We also calculated 95% prediction intervals (PI) to provide the range of possible values for the difference between the characteristics of RCTs that might be observed in future comparisons.

We performed post hoc multivariable meta-regressions to account for confounding variables such as sample size and conflicts of interest. For each methodological trial characteristic, we investigated whether ‘sample size’ or reported ‘conflicts of interest’ are associated with the differences in the effect estimates across the groups of low and high/unclear RoB RCTs.

All statistical analyses were conducted using the R package meta (version 5.2–0) [21].

Results

Of the 33 SRs of RCTs identified in our previous study [22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54] including 97 meta-analyses, five SRs [25, 32, 35, 41, 49], including 11 meta-analyses, were excluded since they provide only one RCT for any eligible outcome. Moreover, we excluded nine meta-analyses in five eligible SRs [29, 44, 48, 51, 53] since comparisons were not possible for any methodological trial characteristic. Overall, 27 SRs of RCTs (Cochrane reviews) [22,23,24, 26,27,28,29,30,31, 33, 34, 36,37,38,39,40, 42,43,44,45,46,47,48, 50, 52,53,54] with 77 meta-analyses (n = 927 RCTs) were included. The details of the selection process are displayed in Fig. 1 and Additional file 3: Tables S8-S9 [55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86].

Fig. 1
figure 1

Flow diagram showing study selection process for eligible Cochrane reviews. MA, meta-analyses; RCT, randomised controlled trials; SR, systematic reviews. Reasons for exclusion are displayed in Additional file 3: Tables S8-S9

Full size image

Characteristics of meta-analyses of randomised controlled trials

The meta-analyses included a median of ten RCTs (range: 2 to 64). The interventions evaluated in the identified meta-analyses can be categorised into micronutrients (n = 36), fatty acids (n = 16), dietary approach (n = 14), food groups (n = 5), fibre (n = 4), and food (n = 2).

The intervention was administered in the form of dietary supplements (n = 31), dietary intake (n = 30), or both (n = 16). The outcome clusters included intermediate disease markers (n = 21), CVD (n = 18), pregnancy outcomes (n = 17), all-cause mortality (n = 12), cancer (n = 6), bone health (n = 2), and eye disease (n = 1). Additional file 3: Tables S10-S11 presents the characteristics of all meta-analyses in our sample.

All included SRs assessed the RoB of the primary studies using the criteria of the original version of the Cochrane tool [6]. Among the 77 meta-analyses, 60 were informative for ‘random sequence generation’, 69 for ‘allocation concealment’, 39 for ‘blinding of participants and personnel’, 49 for ‘blinding of outcome assessment’, 60 for ‘incomplete outcome data’, 36 for ‘selective reporting’, and 16 for ‘dietary compliance’. Overall, we included 52 meta-analyses on binary outcomes and 25 on continuous outcomes (Additional file 3: Table S12).

For ‘random sequence generation’, 618 trials (74.4%) included in the meta-analyses were judged as having low RoB by the original review authors (see Table 1). For ‘allocation concealment’, there were 541 trials (58.5%) with a low RoB judgement, for ‘blinding of participants and personnel’ 264 trials (61.4%), ‘blinding of outcomes’ 348 trials (69.6%), ‘incomplete outcome data’ 583 trials (66.9%), and ‘selective reporting’ 369 trials (65.1%). The proportion of trials judged as being at low RoB was lowest for the ‘dietary compliance’ domain (50.9%, 119 trials). In the comparison group enclosing unclear and high RoB trials, most trials (51.6–96.4%) were judged as having an unclear RoB by the original review authors. Most of the trials with a high RoB were found for ‘blinding of participants and personnel’ (78 trials, 18.1%) and ‘incomplete outcome data’ (140 trials, 16.0%).

Average bias associated with the methodological trial characteristics

Tables 2 and 3 present the results of the association of the reported characteristics on intervention effect estimates, from the main analyses, subgroup analyses, and sensitivity analyses. All meta-analyses conducted are illustrated with forest plots in Additional file 2: Figs. S2-S47.

Table 2 Overview of the main results for binary outcomes
Full size table
Table 3 Overview of the main results for continuous outcomes
Full size table

Random sequence

Based on 39 meta-analyses with binary outcomes [22,23,24, 26,27,28,29, 31, 36, 37, 40, 42, 46, 53, 54], high/unclear RoB judgement (versus low) for sequence generation may not exaggerate intervention effect estimates on average (RRR 0.97, 95% CI 0.93 to 1.02; I2 = 28%; τ2 = 0.002; 95% PI 0.88 to 1.08; see Additional file 2: Fig. S2). The sensitivity analyses excluding highly correlated outcomes (Additional file 2: Fig. S3) or including only one outcome per comparison from each included SR (Additional file 2: Fig. S4), and subgroup analyses for the types of intervention and outcomes (Additional file 2: Figs. S5-S7) also yielded no exaggeration of intervention effect estimates on average, comparing RCTs with high/unclear RoB judgement (versus low) for sequence generation.

Based on 21 meta-analyses with continuous outcomes [22, 24, 30, 33, 34, 39, 43, 45, 47, 48, 52], no exaggeration of intervention effect estimates on average was detected (Additional file 2: Fig. S8), also when excluding highly correlated outcomes (Additional file 2: Fig. S9) or including only one outcome per comparison from each SR (Additional file 2: Fig. S10).

Allocation concealment

Based on 46 meta-analyses with binary outcomes [22,23,24, 26,27,28,29, 31, 36,37,38, 40, 42, 45, 46, 50, 53, 54], high/unclear RoB (versus low) judgement for allocation concealment may not exaggerate the intervention effect estimates on average (RRR 1.00, 95% CI 0.96 to 1.04; I2 = 27%; τ2 = 0.001; 95% PI 0.92 to 1.08; see Additional file 2: Fig. S11). This finding was confirmed in a sensitivity analysis by excluding highly correlated outcomes (Additional file 2: Fig. S12) or including only one outcome per comparison from each SR (Additional file 2: Fig. S13), and the additional subgroup analyses for the different types of interventions and outcomes (Additional file 2: Figs. S14-S16).

For continuous outcomes [22, 24, 30, 34, 37, 39, 43, 45, 47, 48, 50, 52], the average effect of high/unclear RoB (versus low) judgement for allocation concealment was close to the null (DSMD 0.03, 95% CI − 0.07 to 0.12; I2 = 51%; τ2 = 0.03; 95% PI − 0.32 to 0.37; see Additional file 2: Fig. S17), also in the sensitivity analysis with excluding highly correlated outcomes (Additional file 2: Fig. S18) or including only one outcome per comparison from each SR (Additional file 2: Fig. S19).

Blinding of participants and personnel

A total of 29 meta-analyses with binary outcomes [22,23,24, 28, 36, 40, 42, 45, 46, 54] contributed to the analysis of RCTs with high/unclear RoB (versus low) judgement for blinding of participants and personnel. No exaggeration of intervention effect estimates on the average was observed in the main analysis (RRR 0.95, 95% CI 0.91 to 1.00; I2 = 23%; τ2 = 0; 95% PI 0.90 to 1.01; see Additional file 2: Fig. S20) and by excluding highly correlated outcomes (Additional file 2: Fig. S21) or including only one outcome per comparison from each SR (Additional file 2: Fig. S22). When analyses were stratified according to the type of intervention or subjective and objective outcomes, the average effect of bias associated with high/unclear RoB (versus low) judgement for blinding of participants and personnel was not exaggerated (Additional file 2: Figs. S23-S24). However, in subgroup analysis stratified by the cluster of outcomes, the outcome ‘cancer’ showed an average of 32% exaggeration (RRR 0.68, 95% CI 0.51 to 0.91; I2 = 0%; τ2 = 0; 95% PI 0.36 to 1.30; see Additional file 2: Fig. S25). It is worth notable that this finding is limited by the low number of included meta-analyses (n = 4).

Based on ten meta-analyses with continuous outcomes [22, 24, 30, 34, 45], no exaggeration of intervention effect estimates on average was detected, in both the main and the sensitivity analyses (Additional file 2: Figs. S26-S28).

Blinding of outcome assessment

In the analysis of 28 meta-analyses with binary outcomes, the influence of high/unclear RoB (versus low) judgement for blinding of outcome assessors on the effect estimates was substantial (RRR 0.81, 95% CI 0.70 to 0.94; I2 = 26%; τ2 = 0.03; 95% PI 0.54 to 1.23; see Additional file 2: Fig. S29). This finding was confirmed in the sensitivity analyses excluding highly correlated outcomes (Additional file 2: Fig. S30), but not when including only one outcome per comparison from each SR (RRR 0.88, 95% CI 0.73 to 1.07; I2 = 17%; τ2 = 0.03; 95% PI 0.58 to 1.35; see Additional file 2: Fig. S31).

Subgroup analysis stratified by the type of interventions (Additional file 2: Fig. S32) indicated that the exaggeration of intervention effect estimates on average was driven by ‘fatty acids’ interventions (RRR 0.62, 95% CI 0.47 to 0.82; I2 = 0%; τ2 = 0; 95% PI 0.45 to 0.86) and differed substantially from ‘micronutrients’ interventions (RRR 0.90, 95% CI 0.77 to 1.05; I2 = 37%; τ2 = 0.02; 95% PI 0.62 to 1.29). Moreover, when analyses were stratified according to the type of outcomes (Additional file 2: Figs. S33-S34), exaggeration of intervention effect estimates on average was driven by ‘mostly subjectively assessed’ outcomes (RRR 0.74, 95% CI 0.59 to 0.93; I2 = 0%; τ2 = 0.02; 95% PI 0.48 to 1.13). It was not statistically significant for all-cause mortality (RRR 0.83, 95% CI 0.64 to 1.08; I2 = 0%; τ2 = 0.02; 95% PI 0.46 to 1.51) or pregnancy outcomes (RRR 0.86, 95% CI 0.62 to 1.18; I2 = 49%; τ2 = 0.11; 95% PI 0.37 to 1.98).

For continuous outcomes [22, 24, 30, 33, 34, 43, 45, 47, 48, 50], the average effect of high/unclear RoB judgements for blinding of outcome assessment was close to the null in the main analysis (DSMD 0.07, 95% CI − 0.02 to 0.16; I2 = 33%; τ2 = 0.01; 95% PI − 0.19 to 0.33; see Additional file 2: Fig. S35) and when excluding highly correlated outcomes (Additional file 2: Fig. S36) or including only one outcome per comparison from each SR (Additional file 2: Fig. S37).

Incomplete outcome data

Based on our analysis of 37 meta-analyses [22,23,24, 27,28,29, 36,37,38, 40, 42, 44, 45, 50], high/unclear RoB (versus low) judgement for incomplete outcome data may exaggerate the intervention effect estimates, on average by 8% (RRR 0.92, 95% CI 0.88 to 0.97; I2 = 22%; τ2 = 0.001; 95% PI 0.85 to 1.00; see Additional file 2: Fig. S38). However, this was also the case when excluding highly correlated outcomes in the sensitivity analysis (RRR 0.93, 95% CI 0.88 to 0.99; I2 = 19%; τ2 = 0.001; 95% PI 0.84 to 1.03; see Additional file 2: Fig. S39) or including only one outcome per comparison from each SR (RRR 0.92, 95% CI 0.86 to 0.98; I2 = 38%; τ2 = 0.002; 95% PI 0.82 to 1.03; see Additional file 2: Fig. S40).

When analyses were stratified according to type of interventions or type of outcomes, the average effect of bias associated with high/unclear RoB judgement for incomplete outcome data was largest for ‘fatty acids’ (RRR 0.82, 95% CI 0.72 to 0.94; I2 = 0%; τ2 = 0; 95% PI 0.71 to 0.96; see Additional file 2: Fig. S41) and ‘all-cause mortality’ (RRR 0.91, 95% CI 0.85 to 0.98; I2 = 0%; τ2 = 0; 95% PI 0.84 to 0.99; see Additional file 2: Fig. S42-S43), respectively. The subgroup analysis focusing on cancer as an outcome showed an average of 43% exaggeration (RRR 0.57, 95% CI 0.34 to 0.95; I2 = 0%; τ2 = 0; 95% PI 0.02 to 16.15; see Additional file 2: Fig. S42), but this finding is limited by the low number of included meta-analyses (n = 3).

Based on 23 meta-analyses with continuous outcomes [22, 24, 30, 34, 37, 39, 43, 45, 47, 48, 50, 52], no exaggeration of intervention effect estimates on average was observed (Additional file 2: Fig. S44), also in the sensitivity analyses when excluding highly correlated outcomes or including only one outcome per comparison from each SR (Additional file 2: Figs. S45-S46).

Selective reporting

Based on the meta-analysis of 23 meta-analyses [22,23,24, 27,28,29, 36, 40, 45, 50], there was no evidence that trials rated at high/unclear (versus low) RoB have different effect estimates due to selective reporting (RRR 0.97, 95% CI 0.92 to 1.02; I2 = 24%; τ2 = 0; 95% PI 0.92 to 1.02; see Additional file 2: Fig. S47). This finding was confirmed in sensitivity analyses and also across all subgroup analyses (Additional file 2: Figs. S48-S52).

For the continuous outcomes [22, 24, 39, 43, 45, 48, 50, 52], an exaggeration of intervention effect estimates on average was observed (DSMD − 0.10, 95% CI − 0.18 to − 0.03; I2 = 0%; τ2 = 0; 95% PI − 0.19 to − 0.02; see Additional file 2: Fig. S53). This was also the case when only outcome per intervention was included in the sensitivity analysis (Additional file 2: Fig. S54). However, the sensitivity analysis excluding highly correlated outcomes does not confirm this finding (DSMD − 0.08, 95% CI − 0.18 to 0.03; I2 = 16%; τ2 = 0.002; 95% PI − 0.25 to 0.10; see Additional file 2: Fig. S55).

Dietary compliance

Based on 15 meta-analyses with binary outcomes [22,23,24, 38, 40], there was no evidence that trials rated at high/unclear (versus low) RoB for dietary compliance have different effect estimates (RRR 0.95, 95% CI 0.89 to 1.02; I2 = 0%; τ2 = 0.00; 95% PI 0.88 to 1.03; see Additional file 2: Fig. S56). This finding was confirmed in the sensitivity and subgroup analyses (Additional file 2: Figs. S57-S61).

Due to the low number of meta-analyses available (n = 1) [22], we did not perform any analysis for continuous outcomes for dietary compliance.

Meta-regression analyses did not show any statistically significant effects of the confounding variables ‘sample size’ and ‘conflicts of interest’ on the pooled estimates, in any of the methodological trial characteristics (Additional file 3: Tables S13-S25).

Discussion

Summary of findings

This meta-epidemiological study of 77 meta-analyses synthesised evidence on the average bias associated with reported methodological trial characteristics such as random sequence generation, allocation concealment, blinding, incomplete outcome data, selective reporting, and dietary compliance of RCTs in the nutritional field. The main findings suggest that most characteristics of nutrition RCTs considered in our sample may not exaggerate the average intervention effect estimates. However, we observed that intervention effect estimates may exaggerate on average by 18% in trials rated at high/unclear (versus low) RoB for blinding of outcome assessment. For this methodological trial characteristic, the average bias appears to be larger for subjective outcomes, whereas for all-cause mortality and pregnancy outcomes, no association was observed. Moreover, high/unclear (versus low) RoB judgements for incomplete outcome data may exaggerate intervention effect estimates on average by 9%, which was confirmed in various subgroup and sensitivity analyses. Overall, the statistical heterogeneity was low, which was confirmed by a narrow 95% PI. For a pooled estimate of continuous outcomes, no differences were observed between high/unclear (versus low) RoB methodological trial characteristics, except for selective reporting in the main analysis.

Comparison with other studies

We did not identify any similar empirical study investigating the impact of study design biases on intervention effects across meta-analyses of nutrition RCTs. However, several meta-epidemiological studies have been conducted in the medical research field so far. Savovic and colleagues [14], for instance, showed that intervention effect estimates seemed to be exaggerated in trials with inadequate/unclear (versus adequate) sequence generation (by 11%), allocation concealment (by 7%), or double-blinding (by 13%). This exaggeration was, however, not statistically significant for mortality and other objective outcomes, which is in line with our findings. Similarly, Page and colleagues [8] synthesised evidence from 24 meta-epidemiological studies and observed that certain characteristics of RCTs such as sequence generation, allocation concealment, and double blinding may exaggerate the intervention effect estimates for subjective compared with objective outcomes. Moreover, inadequate/unclear blinding of the outcome assessor (versus adequate blinding) may exaggerate the intervention for subjective outcomes, which is in line with our findings. Regarding the detected exaggeration of intervention effects observed in RCTs with high or unclear RoB judgements for incomplete outcome data, previous studies reported inconsistent findings: Savovic and colleagues [10] found little evidence that intervention effects were exaggerated in trials which were judged to be at high or unclear RoB for incomplete outcome data, whereas Abraha and colleagues [87] observed an exaggeration.

In congruence with our findings, two small meta-epidemiological studies concluded that there was no convincing evidence that trials rated at high/unclear (versus low) risk of bias due to selective reporting have different effect estimates [88, 89].

A recent meta-epidemiological study investigated the impact of blinding and found no evidence for an average difference in estimated treatment effect between trials with and without blinded patients, healthcare providers, or outcome assessors [90]. This study result is in agreement with our findings on double blinding of participants and personnel but in disagreement with our findings on blinding of outcome assessment for subjective outcomes.

Implications for the research nutrition field

Systematic evidence syntheses based on high-quality evidence are highly needed to generate trustworthy dietary guidance. Ideally, these should base on direct evidence from RCTs with a low risk of bias. In the field of nutrition research, however, trialists face a number of methodological challenges that are difficult to resolve [13]. Whereas certain trial characteristics such as random sequence generation, allocation concealment, and blinding of outcome assessment can be performed for most RCTs, irrespective of the research field, others are context-specific and are more difficult to incorporate such as blinding of participants and personnel, incomplete outcome data, and dietary compliance. For instance, people are in general aware of what they consume. Therefore, it is often impossible to blind study participants, except for RCTs with dietary supplements. Moreover, longer-term nutrition research trials often face attrition of participants (40–50% dropout rate is fairly common in free-living populations [91]) and low compliance to a specific dietary regimen is often observed [92].

Nonetheless, in our meta-epidemiological study, we observed that on average, most characteristics of nutrition RCTs may often not exaggerate intervention effect estimates. Of note, our sample is mainly based on interventions with dietary supplements where blinding is possible and more than 60% of the included RCTs were judged as low RoB for blinding of participants and personnel. Moreover, two-third of the RCTs in our sample were judged with low RoB for incomplete outcome data. Attrition seems to be lower in interventions with supplements, as the lifestyle behaviour change introduced by the intervention is small [93].

Regarding dietary compliance, about half of the RCTs (n = 119) included in our sample were judged with a low RoB, and all interventions based on the supplementation (and partly on dietary intake) of fatty acids or micronutrients. Surprisingly, RCTs with high/unclear RoB for dietary compliance showed no differences compared to low RoB trials. This could possibly be explained by the fact that most of the included RCTs were rated as ‘unclear’ (n = 100) and only few (n = 15) at ‘high’ RoB. Furthermore, the number of included studies which were informative about dietary compliance was small. Dietary compliance is not a specific domain in the original version of the Cochrane tool [6]. The authors of the Cochrane reviews included in this study assessed dietary compliance by introducing a new RoB domain [23, 40], within ‘other sources of bias’ [22, 38] or did not provide any details on the assessment in the methods Sect. [24]. Moreover, definitions of ‘dietary compliance’ differ between the included reviews: some authors, for instance, based their judgement on thresholds, whereas others on reported differences in intake or biomarkers between the control and treatment groups (see Additional file 3: Table S26). In the most recent version of the Cochrane tool (RoB 2.0), however, compliance to a specific dietary intervention will be assessed within the domain ‘assessing deviations from intended interventions’ [94].

In order to highlight the methodological weaknesses of included RCTs and address the study design bias, it is crucial for the authors of SRs to take RoB assessments into account in the statistical analysis, in the GRADE assessment, and conclusions of the SRs.

Strengths and limitations

This study has several strengths. First, to the best of our knowledge, no similar study has been conducted in the nutritional field so far. Second, we analysed a large sample of diet-disease effects (n = 77 meta-analyses). Third, we selected meta-analyses of RCTs published as Cochrane reviews, which are internationally recognised as the highest standard in evidence-based healthcare. The high methodological quality of Cochrane nutrition reviews was confirmed recently [95]. Fourth, our study was based on meta-analyses of both binary and continuous outcomes.

However, our study has also several limitations: First, although we pooled a large sample of diet-disease associations, our sample may not be representative of all meta-analyses, and the totality of the evidence of available diet-disease associations might provide different results. Second, some trials may be included multiple times in the main analysis due to the different outcomes. However, in our post hoc sensitivity analyses, we also took a conservative approach by including only one outcome per comparison from each included SR. These sensitivity analyses confirmed all the findings of the main analysis. Third, we did not use the most recent version of the RoB tool [94] since we extracted the RoB judgements as provided by the original review authors. The usage of this tool, which was published in 2019 [94], would have made it impossible to generate an adequate sample. Additional considerations added to the revised tool were thus not considered for our analysis. For example, differences in baseline characteristics or the effect of adhering to intervention are not addressed by the original tool [6]. Fourth, we did not verify the RoB assessment of the included Cochrane reviews including the judgements made for ‘dietary compliance’. Finally, the RoB assessment is based on the reported methodological characteristics of the included primary studies. It remains however unclear, whether trials that were classified as ‘unclear’ might have been in fact adequately conducted. Since we merged both, RCTs with high and unclear risk of bias, we might have misclassified some of the trials which might be at low risk of bias. Due to these limitations, our findings need to be interpreted with caution.

Conclusions

On average, most characteristics of nutrition RCTs may not exaggerate intervention effect estimates, but the average bias appears to be greatest in trials of subjective outcomes. These results could reflect that certain trial characteristics are less important than often believed for RCTs in the nutritional field. We suggest the replication of this meta-epidemiological study in the nearly future to keep this evidence up to date, and to perform it with a larger sample to gain more insight into the impact of methodological quality on the exaggeration of effect estimates. In particular, the impact of dietary compliance with regard to subjectively assessed outcomes needs to be investigated. Moreover, future meta-epidemiological analyses should focus on the methodological trial characteristics as defined by the most recent version of the RoB tool [94]. We conclude that the RoB assessment should remain a methodological safeguard in nutrition trials.