Skip to main content
Download PDF

Improving the trustworthiness of findings from nutrition evidence syntheses: assessing risk of bias and rating the certainty of evidence

European Journal of Nutrition volume 60, pages 2893–2903 (2021)Cite this article

Abstract

Suboptimal diet is recognized as a leading modifiable risk factor for non-communicable diseases. Non-randomized studies (NRSs) with patient relevant outcomes provide many insights into diet–disease relationships. Dietary guidelines are based predominantly on findings from systematic reviews of NRSs—mostly prospective observational studies, despite that these have been repeatedly criticized for yielding potentially less trustworthy results than randomized controlled trials (RCTs). It is assumed that these are a result of bias due to prevalent-user designs, inappropriate comparators, residual confounding, and measurement error. In this article, we aim to highlight the importance of applying risk of bias (RoB) assessments in nutritional studies to improve the credibility of evidence of systematic reviews. First, we discuss the importance and challenges of dietary RCTs and NRSs, and provide reasons for potentially less trustworthy results of dietary studies. We describe currently used tools for RoB assessment (Cochrane RoB, and ROBINS-I), describe the importance of rigorous RoB assessment in dietary studies and provide examples that further the understanding of the key issues to overcome in nutrition research. We then illustrate, by comparing the Grading of Recommendations Assessment, Development, and Evaluation (GRADE) approach with current approaches used by United States Department of Agriculture Dietary Guidelines for Americans, and the World Cancer Research Fund, how to establish trust in dietary recommendations. Our overview shows that the GRADE approach provides more transparency about the single domains for grading the certainty of the evidence and the strength of recommendations. Despite not increasing the certainty of evidence itself, we expect that the rigorous application of the Cochrane RoB and the ROBINS-I tools within systematic reviews of both RCTs and NRSs and their integration within the GRADE approach will strengthen the credibility of dietary recommendations.

Introduction

Non-communicable diseases (NCDs) account for over 70% of total deaths worldwide [1]. According to the Global Burden of Disease studies, suboptimal diet is the leading risk factor for ~ 50% of disabilities from cardiovascular diseases [2]. As in many other areas, systematic reviews (SRs) have been established as the method of choice to synthesize data from primary research studies in the field of nutrition. Hence, the Global Burden of Disease studies, as well as dietary guidelines are based on findings from SRs. Many of these SRs do not exclusively include randomized controlled trials (RCTs), but rely primarily on non-randomized studies (NRSs: in this article the term “NRS” is used exclusively as a synonym for cohort studies), for example prospective cohort studies, because RCTs are not available or considered not applicable [3, 4]. The acceptance of SRs as the main basis for dietary guidelines, Global Burden of Disease studies, and public health nutrition policies constitutes a great opportunity for strengthening the repeatedly criticized trustworthiness (in this article the term “trustworthiness” is used exclusively as a synonym for certainty of evidence [5]) of dietary recommendations [6]. We begin by highlighting the importance and challenges of dietary RCTs and NRSs, and provide potentially less trustworthy results of NRSs, by reporting examples of discordance between findings of RCTs and NRSs. We describe the importance of rigorous risk of bias (RoB) assessment in such studies and provide examples that help understanding the key issues to overcome. We then illustrate, by comparing the Grading of Recommendations Assessment, Development, and Evaluation (GRADE) approach with current approaches (used by United States Department of Agriculture Dietary Guidelines for Americans [7], and the World Cancer Research Fund [8]), how to establish trust in dietary recommendations.

Why we need more RCTs, and why RCTs are difficult to conduct in the nutrition field?

We need good science to trust dietary advice, ideally unbiased and direct evidence from RCTs to overcome bias. RCTs, if well-designed and -conducted, give robust answers to the research questions they address and are widely encouraged as the ideal methodology for causal inference [9]. However, due to the difficulty of inducing and maintaining dietary changes, randomization to allocate people to alternative diets and to investigate effects of long-term lifestyle behaviors on patient relevant outcomes remains challenging (Table 1). RCT methodologies are accompanied by a number of specific challenges in nutritional research. First, in dietary RCTs, it is often impossible to ensure that participants are unaware of their treatment (except for placebo RCTs of dietary supplements) because people are generally aware of what they are eating. Second, in nutrition research trials, low adherence to a specific dietary regimen is often observed. Third, investigating effects of long-term lifestyle behaviors on patient relevant outcomes is difficult. Well-controlled feeding trials could overcome some of these limitations, since study participants are expected to adhere to strict diet by consuming only food provided by the research kitchen [10]. Moreover, supermarket models have been implemented successfully in RCTs in which the participants receive all groceries free of charge for a period of time, for example, for 6–12 months, in a university supermarket. Bar codes and special computer programs were used to monitor and examine whether the participants followed the right composition of the diets they were allocated to. These intervention models work best for single people, and therefore, generalizability of the results is limited for the general population. Biomarkers of intake have shown that this is a superior method to ensure high compliance and, hence, good validity of the efficacy of the diet intervention [11, 12].

Table 1 Strengths and limitations of randomized controlled trials (RCTs) and non-randomized studies (NRSs)
Full size table

Why NRSs provide sometimes potentially less trustworthy results, and how we can identify plausible results?

NRSs, predominately prospective observational studies with patient relevant outcomes (e.g., cardiovascular disease), provide many insights into diet–disease relationships and are the most important source to derive updated Global Burden of Disease studies-reports and dietary guidelines for the primary prevention of NCDs until to date [13]. However, nutrition epidemiology has been repeatedly criticized for providing potentially less trustworthy results. For example, most nutrients not only have been associated with cancer risk but for several of the nutrients there are published reports that show an increased risk in one NRS and a decreased risk in another [14]. In the past, several RCTs comparing dietary interventions with placebo or control interventions have failed to replicate the (presumably protective) associations between dietary factors (e.g., nutrients) and risk for NCDs found in large scale cohort studies [15,16,17,18,19]. For example, RCTs found no evidence for a beneficial effect of fiber intake on CRC risk [20], vitamin E and cardiovascular disease (CVD) [21]. On the contrary, some consistent findings between cohort studies and RCTs have been reported as well (e.g., total fat and coronary heart disease or breast cancer) [22]. Recently, Ioannidis suggested that RCTs should largely replace NRSs in human nutrition research [14] due to the core limitations of NRSs, such as bias due to prevalent-user designs, inappropriate comparators, residual confounding, measurement error, and the fact that small effect sizes are common in nutrition research [23, 24]. Across dietary NRSs, social desirability biases are prevalent: Participants may give perceived “healthy” responses, such as over-reporting fruit and vegetable intake or underestimating fat intake [25], whereas obese patients are more likely to underreport nutritional intake, particularly energy, which can lead to the underestimation of the intake of dietary components assessed [26]. Unfortunately, common tools used to measure dietary adherence in not only NRSs but also RCTs, such as food frequency questionnaires, dietary records, or 24-h dietary recalls, are prone to measurement error [27]. Overall, nutritional research in general poses a number of specific challenges for various empirical approaches.

We postulate that rigorous RoB assessment and the use of the GRADE approach to assess the certainty of the evidence could help to identify plausible results and to address some of the criticism levied at RCTs, NRSs, meta-analyses of such studies, and dietary guidelines, with the aim of overcoming disagreement between classic epidemiologists and interventionalists. This can be done by exploring the nature of potentially less trustworthy results, a process that is often omitted, inconsistently applied across studies, or flawed [28].

What tools should be used to address risk of bias?

At SR level, the established approach to evaluate the credibility of results from primary studies is RoB assessment. The RoB of a single RCT or RCTs included in a SR should be assessed with a well-established and validated tool, such as the RoB tool by Cochrane [29]. Within the Cochrane RoB tool for RCTs, RoB is assessed for six domains: (i) selection bias, (ii) performance bias, (iii) attrition bias, (iv) detection bias, (v) reporting bias, and (vi) other bias (e.g., carry-over effects in cross-over trials) (Table 2) [29]. In a previous analysis of 50 (18% of them Cochrane Reviews) randomly selected nutrition-specific SRs of RCTs [23], it was shown that 70% used the Cochrane RoB assessment tool [23, 29], 14% reported no RoB assessment, 10% the Jadad Scale [30], and 6% applied their own score. Recently, the RoB 2.0 tool has been published [31]. To this day, dietary adherence has not been included as a specific RoB domain in the Cochrane RoB tool. However in the Cochrane RoB tool 2.0, lack of adherence to a specific dietary intervention will be evaluated within the bias domain assessing deviations from intended interventions [31].

Table 2 Comparison of risk of bias domains in RCTs and NRSs, and example of the application of the ROBINS-I tool in a recent meta-analysis investigating the association between adherence to a Mediterranean diet and risk of stroke [51], and the corresponding quality rating by applying the Newcastle–Ottawa Scale
Full size table

Focusing on NRSs, a SR identified 86 tools to assess study quality in NRSs showing high inconsistency in selection/inclusion and weighting of domains across tools [32]. In 50 nutrition-specific SRs of NRSs, it was shown that in 40% of these, no study quality assessment was done [23], 38% used the Newcastle–Ottawa Scale, while the remaining 22% used a variety of other, less well-established tools. When using Newcastle–Ottawa Scale, the most widely applied tool, each study will be judged in relation to eight items (Table 2). However, Stang [33] criticized the NOS for its arbitrary definitions and concluded that this score appeared to be unacceptable for the assessment of study quality, of NRSs. An empirical study has recently shown a fundamental problem when applying the NOS: out of 89 observational nutritional studies, 81 studies (91%) included in 14 meta-analyses were rated as high-quality studies [34]. The threshold to define high quality is apparently so low within NOS that there is no discriminatory effect when applying NOS.

The term “study quality” is often used in this context interchangeably with RoB, but it is important to distinguish between quality and RoB. The term suggests an investigation of the extent to which study authors conducted their research to the highest possible standards. A study may be performed to the highest possible standards yet still have an important RoB. For example, often it is impractical or impossible to blind participants or study personnel to the intervention group. It is inappropriately judgmental to describe all such studies as of “low quality”, but that does not mean they are free of bias resulting from knowledge of intervention status [35]. Moreover, reporting a study (quality) in line with reporting guidelines such as the CONSORT statement (for RCTs) [36] or the STROBE statement [37] is unlikely to have direct implications for risk of bias.

To overcome the problems of the NOS, the risk of bias in non-randomized studies of interventions (ROBINS-I) tool has been developed, and published in 2016 [38] (Table 2). A modified version to assess the RoB in non-randomized studies of exposures (ROBINS-E) is under development, and adaptation of ROBINS for exposure studies is ongoing [39]. For example, Morgan and colleagues published recently a user’s guide on how to apply, interpret, and present the results of ROBINS to assess the RoB in NRSs dealing with effects of exposures (e.g., bisphenol A) on health outcomes (e.g., obesity) [40]. In their user’s guide, the authors applied the draft ROBINS-(exposure) tool successfully, to a variety of study designs including prospective cohort studies and cross-sectional studies [40]; ROBINS-E was also recently used to evaluate RoB in case–control studies [41]. Detailed methods of the application of ROBINS-I and the current development of a RoB instrument for NRSs of exposures have been described in detail by Morgan and colleagues [42]. For example, domains 3 (“bias in classification of interventions”) and 4 (“bias due to departures from intended interventions”) of ROBINS-I have been changed to “bias in classification of exposures” and “bias due to departures from exposures” [42] (Table 2). The COSMOS-E reporting guideline (Conducting Systematic Reviews and Meta-Analyses of Observational Studies of Etiology) has recently been published; COSMOS-E also recommends the use of the ROBINS tool to evaluate RoB in observational studies, and the GRADE approach to rate the certainty of evidence [43]. GRADE is not recommending a specific tool to assess RoB, however, because the tools have different advantages and disadvantages that influence the choice. As long as RoB is assessed across studies, any validated and appropriate tool can be used.

Examples of critical risk of bias in certain domains in a dietary Cochrane Review

To exemplify the usage and judgements of Cochrane RoB domains for dietary RCTs, we chose a highly cited Cochrane Review on Mediterranean diet (MedDiet) and prevention of CVD, which included 49 papers. In this Cochrane Review, the authors took into account the difficulties of blinding participants (although, double blinded designs are not always ideal for providing a reliable answer to the trial’s research question [44]) in dietary interventions and rated this as unclear rather than high RoB [45]. The procedure of the author’s shows that especially for the RoB assessment of dietary RCTs, a puristic approach to judge RoB is not always sensible. Dietary adherence is probably the most important limitation of dietary RCTs, mainly in long-term RCTs [46]. Not only has dietary adherence not been assessed as a RoB item to date, but also many SRs do not even investigate dietary adherence at all, like our exemplary chosen Cochrane Review on MedDiet and CVD [45].

Difficulty of attrition (in free living populations, a 40–50% dropout rate is fairly common [47]) is mainly observed in longer-term dietary RCTs [48]. In the Cochrane Review on the MedDiet and CVD, only two out of 30 RCTs conducted intention-to-treat analyses, and only 11 RCTs were rated with a low RoB for attrition bias [45].

Why is risk of bias so important to evaluate the credibility of study results?

Sensitivity analyses including only low RoB- or excluding high-RoB RCTs are an important means to explore the impact of bias on pooled results in a meta-analysis. A methodology study evaluating 59 SRs showed that only 50% of these SRs conducted sensitivity analyses for low RoB studies [49]. In some circumstances, when conducting sensitivity analyses excluding trials with a high RoB, significant summary estimates become statistically non-significant or vice versa. For example, in a large Cochrane Review investigating the effects of antioxidant supplements for prevention of mortality, risk increasing effects for beta-carotene and vitamin E were only observed in the sensitivity analyses for low RoB RCTs, whereas the primary analysis showed no effects [50]. Because the ROBINS-I tool has only recently been published, it lacks application in nutrition-specific meta-analyses. A recent SR of NRSs investigating the relationship between adherence to the Mediterranean Diet and risk of stroke applied the ROBINS-I tool [51]. Out of 20 included studies, no NRS was rated with a low RoB, 13 NRSs (65%) were rated as moderate RoB, and seven NRSs (35%) as serious RoB (Table 2). On the contrary, the application of the NOS in those studies resulted in a high-quality (low RoB) judgement for all 20 NRSs (Table 2). Rigorous application of the Cochrane RoB tool for RCTs and the ROBINS-I tool for NRSs, would improve evaluation of validity, transparency, interpretation and conclusions of a single dietary RCT or RCTs included in a SR.

Why can rating the certainty of evidence improve the trustworthiness of findings?

RoB assessment is a fundamental part of the GRADE approach. The GRADE working group [52] has developed a common and transparent approach for grading the certainty of evidence and strength of recommendations based on a body of evidence (e.g., SR of RCTs). GRADE is also used in the field of nutrition research [53]. The GRADE approach classifies bodies of RCTs as initially starting at high certainty and bodies of NRSs at initially starting at low certainty [54]. In 2016, the NutriGrade scoring system by Schwingshackl and colleagues was published [23]. The main proposed adaptation of GRADE was a modified initial classification of a body of evidence of RCTs and cohort studies. NutriGrade, and its proposed adaptations was discussed extensively in the scientific community [55, 56]. Afterwards, the GRADE working group has acknowledged limitations that in certain research fields (e.g., nutrition research), RCTs on patient relevant outcomes (e.g., cardiovascular disease) are sparse or not feasible, and that the application of the current GRADE approach to classify study designs might be limited. It has been pointed out that using the GRADE approach, in particular in relation to RoB assessment, is challenging and may lead to excessive downgrading. For example, GRADE users may inappropriately double count the risk of confounding and selection bias by downgrading the initial body of evidence to low, followed by further downgrading due to unknown confounders [23, 56,57,58]. Therefore, guidance on how to assess the certainty of evidence within GRADE when ROBINS-I is being used was published in 2018. RoB instruments, such as ROBINS that allow for the comparison of a body of evidence from NRSs to RCTs eliminate the GRADE requirement for starting an assessment of a body of evidence as “high” or “low” certainty based on study design (Table 3) [57]. This will lead to a better comparison of evidence from RCTs and NRSs because they are placed on a common metric for RoB. Due to its enhanced development we are now suggesting the GRADE approach to rate the certainty of the evidence.

Table 3 Main methodological differences between the GRADE approach [54], and approaches taken by the 3rd World Cancer Research Fund/ American Institute for Cancer Research Expert report [8], and the USDA Dietary Guidelines for Americans 2015–2020 [7] to rate the certainty of evidence
Full size table

The GRADE domains: RoB, inconsistency, indirectness, imprecision, publication bias, large effect, dose–response, and direction of plausible residual confounding are taken into account to arrive at the certainty of the evidence for a given outcome across studies (Table 3). Overall, GRADE specifies four levels of certainty of evidence: high, moderate, low, and very low. For example, high certainty of evidence is defined as: “High certainty that a true effect lies on one side of a specified threshold or within a chosen range” [5]. Guideline authors then consider the direction and strength of recommendation (strong vs. conditional) [59] based on overall certainty of evidence across outcomes and in light of various other criteria including values or importance of the outcomes, resource use, equity, acceptability and feasibility [60]. Although only few nutrition-specific SRs have evaluated the certainty of evidence so far, the GRADE approach is now being applied increasingly in nutrition research [23, 58].

What are current approaches for making dietary recommendations?

Table 3 highlights the main methodological differences in rating the certainty of evidence by comparing the GRADE approach with the approaches by the WCRF [8] and the USDA Dietary Guidelines for Americans 2015–2020 [7]. Overall, the GRADE approach provides more transparency about the single domains.

Separating certainty of the evidence assessment from confidence in a recommendation and decision will bring clarity to the field, lead to better research and could overcome the incommunicado between different stakeholders. Table 4 highlights the Evidence to Decision (EtDs) framework which aims to use evidence in a systematic and transparent way to inform decisions established by the GRADE working group [61]. Guidelines approaches neither by the WCRF nor the USDA have integrated patient and community values and preferences, nor applied strict safeguards against conflicts of interest. However, several components of the EtDs framework have been addressed to be implemented in future UDSA dietary guidelines. In the future, developers of recommendations should take a population perspective for general public health nutrition recommendations, and may consider also biological plausibility or sustainability, which are highly relevant for dietary recommendations [62, 63]. Biological plausibility is a domain which is considered by the WCRF to rate the certainty of evidence [8], whereas GRADE does not consider the issue of biological plausibility as domain of certainty. GRADE argues that biological plausibility is considered in three ways: (1) during question formulation; (2) in the evaluation of other, indirect evidence (e.g., similar nutrient or population); and (3) how directly the intervention affects a surrogate outcome [64].

Table 4 Main methodological differences between the GRADE Evidence to Decisions framework [60], and approaches taken by the 3rd World Cancer Research Fund/ American Institute for Cancer Research Expert report [8], and the USDA Dietary Guidelines for Americans 2015–2020 [7]
Full size table

Conclusion

Despite not increasing the certainty of evidence itself, we expect that the rigorous application of the Cochrane RoB and the ROBINS-I tools within SRs of both RCTs and NRSs and their integration within the GRADE approach will strengthen the credibility of dietary recommendations.

Financial support

This research received no specific grant from any funding agency, commercial or not-for-profit sectors.

Abbreviations

GRADE:

Grading of Recommendations Assessment, Development, and Evaluation

NRSs:

Non-randomized studies

RCTs:

Randomized controlled trials

RoB:

Risk of bias

ROBINS-I/E:

Risk of bias in non-randomized studies of interventions/exposures

SRs:

Systematic reviews

WCRF:

World Cancer research Fund

References

  1. Global, regional, and national age-sex specific mortality for 264 causes of death, 1980–2016: a systematic analysis for the Global Burden of Disease Study 2016 (2017). Lancet 390 (10100):1151–1210. doi:https://doi.org/10.1016/s0140-6736(17)32152-9

  2. Afshin A, Sur PJ, Fay KA, Cornaby L, Ferrara G, Salama JS, Mullany EC, Abate KH, Abbafati C, Abebe Z, Afarideh M, Aggarwal A, Agrawal S, Akinyemiju T, Alahdab F, Bacha U, Bachman VF, Badali H, Badawi A, Bensenor IM, Bernabe E, Biadgilign SKK, Biryukov SH, Cahill LE, Carrero JJ, Cercy KM, Dandona L, Dandona R, Dang AK, Degefa MG, ElSayedZaki M, Esteghamati A, Esteghamati S, Fanzo J, Farinha CSES, Farvid MS, Farzadfar F, Feigin VL, Fernandes JC, Flor LS, Foigt NA, Forouzanfar MH, Ganji M, Geleijnse JM, Gillum RF, Goulart AC, Grosso G, Guessous I, Hamidi S, Hankey GJ, Harikrishnan S, Hassen HY, Hay SI, Hoang CL, Horino M, Islami F, Jackson MD, James SL, Johansson L, Jonas JB, Kasaeian A, Khader YS, Khalil IA, Khang YH, Kimokoti RW, Kokubo Y, Kumar GA, Lallukka T, Lopez AD, Lorkowski S, Lotufo PA, Lozano R, Malekzadeh R, März W, Meier T, Melaku YA, Mendoza W, Mensink GBM, Micha R, Miller TR, Mirarefin M, Mohan V, Mokdad AH, Mozaffarian D, Nagel G, Naghavi M, Nguyen CT, Nixon MR, Ong KL, Pereira DM, Poustchi H, Qorbani M, Rai RK, Razo-García C, Rehm CD, Rivera JA, Rodríguez-Ramírez S, Roshandel G, Roth GA, Sanabria J, Sánchez-Pimienta TG, Sartorius B, Schmidhuber J, Schutte AE, Sepanlou SG, Shin M-J, Sorensen RJD, Springmann M, Szponar L, Thorne-Lyman AL, Thrift AG, Touvier M, Tran BX, Tyrovolas S, Ukwaja KN, Ullah I, Uthman OA, Vaezghasemi M, Vasankari TJ, Vollset SE, Vos T, Vu GT, Vu LG, Weiderpass E, Werdecker A, Wijeratne T, Willett WC, Wu JH, Xu G, Yonemoto N, Yu C, Murray CJL (2017) Health effects of dietary risks in 195 countries, 1990–2013: a systematic analysis for the Global Burden of Disease Study 2017. Lancet. https://doi.org/10.1016/S0140-6736(19)30041-8

    Article  Google Scholar 

  3. Kromhout D, Spaaij CJ, de Goede J, Weggemans RM (2016) The 2015 Dutch food-based dietary guidelines. Eur J Clin Nutr 70(8):869–878. https://doi.org/10.1038/ejcn.2016.52

    CAS  Article  PubMed  PubMed Central  Google Scholar 

  4. Global, regional, and national comparative risk assessment of 84 behavioural, environmental and occupational, and metabolic risks or clusters of risks, 1990–2016: a systematic analysis for the Global Burden of Disease Study 2016 (2017). Lancet (London, England) 390 (10100):1345–1422. doi:https://doi.org/10.1016/s0140-6736(17)32366-8

  5. Hultcrantz M, Rind D, Akl EA, Treweek S, Mustafa RA, Iorio A, Alper BS, Meerpohl JJ, Murad MH, Ansari MT, Katikireddi SV, Östlund P, Tranæus S, Christensen R, Gartlehner G, Brozek J, Izcovich A, Schünemann H, Guyatt G (2017) The GRADE Working Group clarifies the construct of certainty of evidence. J Clin Epidemiol 87:4–13. https://doi.org/10.1016/j.jclinepi.2017.05.006

    Article  PubMed  PubMed Central  Google Scholar 

  6. Nestle M (2018) Perspective: challenges and controversial issues in the dietary guidelines for Americans, 1980–2015. Adv Nutr 9(2):148–150. https://doi.org/10.1093/advances/nmx022

    Article  PubMed  PubMed Central  Google Scholar 

  7. US Department of Health and Human Services and US Department of Agriculture. 2015–2020 Dietary Guidelines for Americans. 8th Edition. December 2015. https://health.gov/dietaryguidelines/2015/guidelines/. Accessed 09 Jan 2020.

  8. World Cancer Research Fund International (2018) Diet, nutrition, physical activity and cancer: a global perspective—the third expert report. London: World Cancer Research Fund International. https://www.wcrf.org/dietandcancer.

  9. Pan A, Lin X, Hemler E, Hu FB (2018) Diet and cardiovascular disease: advances and challenges in population-based studies. Cell Metab 27(3):489–496. https://doi.org/10.1016/j.cmet.2018.02.017

    CAS  Article  PubMed  PubMed Central  Google Scholar 

  10. Hall DM, Most MM (2005) Dietary adherence in well-controlled feeding studies. J Acad Nutr Diet 105(8):1285–1288. https://doi.org/10.1016/j.jada.2005.05.009

    Article  Google Scholar 

  11. Larsen TM, Dalskov SM, van Baak M, Jebb SA, Papadaki A, Pfeiffer AF, Martinez JA, Handjieva-Darlenska T, Kunešová M, Pihlsgård M, Stender S, Holst C, Saris WH, Astrup A (2010) Diets with high or low protein content and glycemic index for weight-loss maintenance. N Engl J Med 363(22):2102–2113. https://doi.org/10.1056/NEJMoa1007137

    CAS  Article  PubMed  PubMed Central  Google Scholar 

  12. Poulsen SK, Due A, Jordy AB, Kiens B, Stark KD, Stender S, Holst C, Astrup A, Larsen TM (2014) Health effect of the New Nordic Diet in adults with increased waist circumference: a 6-mo randomized controlled trial. Am J Clin Nutr 99(1):35–45. https://doi.org/10.3945/ajcn.113.069393

    CAS  Article  PubMed  Google Scholar 

  13. Schwingshackl L, Schlesinger S, Devleesschauwer B, Hoffmann G, Bechthold A, Schwedhelm C, Iqbal K, Knüppel S, Boeing H (2018) Generating the evidence for risk reduction: a contribution to the future of food-based dietary guidelines. Proc Nutr Soc 77(4):432–444. https://doi.org/10.1017/S0029665118000125

    Article  PubMed  Google Scholar 

  14. Ioannidis JA (2018) The challenge of reforming nutritional epidemiologic research. JAMA 320(10):969–970. https://doi.org/10.1001/jama.2018.11025

    Article  PubMed  Google Scholar 

  15. Koushik A, Hunter DJ, Spiegelman D, Anderson KE, Buring JE, Freudenheim JL, Goldbohm RA, Hankinson SE, Larsson SC, Leitzmann M, Marshall JR, McCullough ML, Miller AB, Rodriguez C, Rohan TE, Ross JA, Schatzkin A, Schouten LJ, Willett WC, Wolk A, Zhang SM, Smith-Warner SA (2006) Intake of the major carotenoids and the risk of epithelial ovarian cancer in a pooled analysis of 10 cohort studies. Int J Cancer 119(9):2148–2154. https://doi.org/10.1002/ijc.22076

    CAS  Article  PubMed  Google Scholar 

  16. Humphrey LL, Fu R, Rogers K, Freeman M, Helfand M (2008) Homocysteine level and coronary heart disease incidence: a systematic review and meta-analysis. Mayo Clin Proc 83(11):1203–1212. https://doi.org/10.4065/83.11.1203

    Article  PubMed  Google Scholar 

  17. Albert CM, Cook NR, Gaziano JM, Zaharris E, MacFadyen J, Danielson E, Buring JE, Manson JE (2008) Effect of folic acid and B vitamins on risk of cardiovascular events and total mortality among women at high risk for cardiovascular disease: a randomized trial. JAMA 299(17):2027–2036. https://doi.org/10.1001/jama.299.17.2027

    CAS  Article  PubMed  PubMed Central  Google Scholar 

  18. Stampfer MJ, Hennekens CH, Manson JE, Colditz GA, Rosner B, Willett WC (1993) Vitamin E consumption and the risk of coronary disease in women. N Engl J Med 328(20):1444–1449. https://doi.org/10.1056/nejm199305203282003

    CAS  Article  PubMed  Google Scholar 

  19. Rapola JM, Virtamo J, Ripatti S, Huttunen JK, Albanes D, Taylor PR, Heinonen OP (1997) Randomised trial of alpha-tocopherol and beta-carotene supplements on incidence of major coronary events in men with previous myocardial infarction. Lancet 349(9067):1715–1720. https://doi.org/10.1016/s0140-6736(97)01234-8

    CAS  Article  PubMed  Google Scholar 

  20. Schatzkin A, Lanza E, Corle D, Lance P, Iber F, Caan B, Shike M, Weissfeld J, Burt R, Cooper MR, Kikendall JW, Cahill J (2000) Lack of effect of a low-fat, high-fiber diet on the recurrence of colorectal adenomas Polyp Prevention Trial Study Group. N Engl J Med 342(16):1149–1155. https://doi.org/10.1056/nejm200004203421601

    CAS  Article  PubMed  Google Scholar 

  21. Yusuf S, Dagenais G, Pogue J, Bosch J, Sleight P (2000) Vitamin E supplementation and cardiovascular events in high-risk patients. N Engl J Med 342(3):154–160. https://doi.org/10.1056/nejm200001203420302

    CAS  Article  PubMed  Google Scholar 

  22. Satija A, Stampfer MJ, Rimm EB, Willett W, Hu FB (2018) Perspective: are large, simple trials the solution for nutrition research? Adv Nutr 9(4):378–387. https://doi.org/10.1093/advances/nmy030

    Article  PubMed  PubMed Central  Google Scholar 

  23. Schwingshackl L, Knuppel S, Schwedhelm C, Hoffmann G, Missbach B, Stelmach-Mardas M, Dietrich S, Eichelmann F, Kontopantelis E, Iqbal K, Aleksandrova K, Lorkowski S, Leitzmann MF, Kroke A, Boeing H (2016) Perspective: nutrigrade: a scoring system to assess and judge the meta-evidence of randomized controlled trials and cohort studies in nutrition research. Adv Nutr 7(6):994–1004. https://doi.org/10.3945/an.116.013052

    Article  PubMed  PubMed Central  Google Scholar 

  24. Maki KC, Slavin JL, Rains TM, Kris-Etherton PM (2014) Limitations of observational evidence: implications for evidence-based dietary recommendations. Adv Nutr 5(1):7–15. https://doi.org/10.3945/an.113.004929

    CAS  Article  PubMed  PubMed Central  Google Scholar 

  25. Caan B, BallardBarbash R, Slattery ML, Pinsky JL, Iber FL, Mateski DJ, Marshall JR, Paskett ED, Shike M, Weissfeld JL, Schatzkin A, Lanza E (2004) Low energy reporting may increase in intervention participants enrolled in dietary intervention trials. J Am Diet Assoc 104(3):357–366. https://doi.org/10.1016/j.jada.2003.12.023.[quiz491]

    Article  PubMed  Google Scholar 

  26. Rebro SM, Patterson RE, Kristal AR, Cheney CL (1998) The effect of keeping food records on eating patterns. J Am Diet Assoc 98(10):1163–1165. https://doi.org/10.1016/s0002-8223(98)00269-7

    CAS  Article  PubMed  Google Scholar 

  27. Shim JS, Oh K, Kim HC (2014) Dietary assessment methods in epidemiologic studies. Epidemiol Health 36:e2014009. https://doi.org/10.4178/epih/e2014009

    Article  PubMed  PubMed Central  Google Scholar 

  28. Ioannidis JP (2016) The mass production of redundant, misleading, and conflicted systematic reviews and meta-analyses. Milbank Q 94(3):485–514. https://doi.org/10.1111/1468-0009.12210

    Article  PubMed  PubMed Central  Google Scholar 

  29. Higgins JP, Altman DG, Gotzsche PC, Juni P, Moher D, Oxman AD (2011) The Cochrane Collaboration’s tool for assessing risk of bias in randomised trials. BMJ. https://doi.org/10.1136/bmj.d5928

    Article  PubMed  PubMed Central  Google Scholar 

  30. Jadad AR, Moore RA, Carroll D, Jenkinson C, Reynolds DJ, Gavaghan DJ, McQuay HJ (1996) Assessing the quality of reports of randomized clinical trials: is blinding necessary? Control Clin Trials 17(1):1–12

    CAS  Article  PubMed  Google Scholar 

  31. Sterne JAC, Savovic J, Page MJ, Elbers RG, Blencowe NS, Boutron I, Cates CJ, Cheng HY, Corbett MS, Eldridge SM, Emberson JR, Hernan MA, Hopewell S, Hrobjartsson A, Junqueira DR, Juni P, Kirkham JJ, Lasserson T, Li T, McAleenan A, Reeves BC, Shepperd S, Shrier I, Stewart LA, Tilling K, White IR, Whiting PF, Higgins JPT (2019) RoB 2: a revised tool for assessing risk of bias in randomised trials. BMJ 366:l4898. https://doi.org/10.1136/bmj.l4898

    Article  PubMed  Google Scholar 

  32. Sanderson S, Tatt ID, Higgins JP (2007) Tools for assessing quality and susceptibility to bias in observational studies in epidemiology: a systematic review and annotated bibliography. Int J Epidemiol 36(3):666–676. https://doi.org/10.1093/ije/dym018

    Article  PubMed  Google Scholar 

  33. Stang A (2010) Critical evaluation of the Newcastle-Ottawa scale for the assessment of the quality of nonrandomized studies in meta-analyses. Eur J Epidemiol 25(9):603–605. https://doi.org/10.1007/s10654-010-9491-z

    Article  PubMed  Google Scholar 

  34. Bae J-M (2016) A suggestion for quality assessment in systematic reviews of observational studies in nutritional epidemiology. Epidemiol Health 38:e2016014–e2016014. https://doi.org/10.4178/epih.e2016014

    Article  PubMed  PubMed Central  Google Scholar 

  35. Higgins JPT, Green S eds. (2011) Cochrane handbook for systematic reviews of interventions version 5.1.0 [updated March 2011]. The Cochrane Collaboration. www.handbook.cochrane.org.

  36. Moher D, Hopewell S, Schulz KF, Montori V, Gøtzsche PC, Devereaux PJ, Elbourne D, Egger M, Altman DG (2010) CONSORT 2010 explanation and elaboration: updated guidelines for reporting parallel group randomised trials. BMJ 340:c869. https://doi.org/10.1136/bmj.c869

    Article  PubMed  PubMed Central  Google Scholar 

  37. von Elm E, Altman DG, Egger M, Pocock SJ, Gøtzsche PC, Vandenbroucke JP (2007) The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: guidelines for reporting observational studies. PLoS Med 4(10):e296. https://doi.org/10.1371/journal.pmed.0040296

    Article  Google Scholar 

  38. Sterne JA, Hernan MA, Reeves BC, Savovic J, Berkman ND, Viswanathan M, Henry D, Altman DG, Ansari MT, Boutron I, Carpenter JR, Chan AW, Churchill R, Deeks JJ, Hrobjartsson A, Kirkham J, Juni P, Loke YK, Pigott TD, Ramsay CR, Regidor D, Rothstein HR, Sandhu L, Santaguida PL, Schunemann HJ, Shea B, Shrier I, Tugwell P, Turner L, Valentine JC, Waddington H, Waters E, Wells GA, Whiting PF, Higgins JP (2016) ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions. BMJ (Clinical research ed) 355:i4919. https://doi.org/10.1136/bmj.i4919

    Article  Google Scholar 

  39. https://www.bristol.ac.uk/population-health-sciences/centres/cresyda/barr/riskofbias/robins-e/ Accessed 09 Jan 2020.

  40. Morgan RL, Thayer KA, Santesso N, Holloway AC, Blain R, Eftim SE, Goldstone AE, Ross P, Ansari M, Akl EA, Filippini T, Hansell A, Meerpohl JJ, Mustafa RA, Verbeek J, Vinceti M, Whaley P, Schünemann HJ (2019) A risk of bias instrument for non-randomized studies of exposures: A users’ guide to its application in the context of GRADE. Environ Int 122:168–184. https://doi.org/10.1016/j.envint.2018.11.004

    Article  PubMed  Google Scholar 

  41. González-Luis GE, van Westering-Kroon E, Villamor-Martinez E, Huizing MJ, Kilani MA, Kramer BW, Villamor E (2020) Tobacco smoking during pregnancy is associated with increased risk of moderate/severe bronchopulmonary dysplasia: a systematic review and Meta-Analysis. Front Pediatr 8:160. https://doi.org/10.3389/fped.2020.00160

    Article  PubMed  PubMed Central  Google Scholar 

  42. Morgan RL, Thayer KA, Santesso N, Holloway AC, Blain R, Eftim SE, Goldstone AE, Ross P, Guyatt G, Schünemann HJ (2018) Evaluation of the risk of bias in non-randomized studies of interventions (ROBINS-I) and the ‘target experiment’ concept in studies of exposures: Rationale and preliminary instrument development. Environ Int 120:382–387. https://doi.org/10.1016/j.envint.2018.08.018

    Article  PubMed  Google Scholar 

  43. Dekkers OM, Vandenbroucke JP, Cevallos M, Renehan AG, Altman DG, Egger M (2019) COSMOS-E: Guidance on conducting systematic reviews and meta-analyses of observational studies of etiology. PLOS Medicine 16(2):e1002742. https://doi.org/10.1371/journal.pmed.1002742

    Article  PubMed  PubMed Central  Google Scholar 

  44. Anand R, Norrie J, Bradley JM, McAuley DF, Clarke M (2020) Fool’s gold? Why blinded trials are not always best. BMJ 368:l6228. https://doi.org/10.1136/bmj.l6228

    Article  PubMed  Google Scholar 

  45. Rees K, Takeda A, Martin N, Ellis L, Wijesekara D, Vepa A, Das A, Hartley L, Stranges S (2019) Mediterranean-style diet for the primary and secondary prevention of cardiovascular disease. Cochrane Database Syst Rev. https://doi.org/10.1002/14651858.CD009825.pub3

    Article  PubMed  PubMed Central  Google Scholar 

  46. Sacks FM, Bray GA, Carey VJ, Smith SR, Ryan DH, Anton SD, McManus K, Champagne CM, Bishop LM, Laranjo N, Leboff MS, Rood JC, de Jonge L, Greenway FL, Loria CM, Obarzanek E, Williamson DA (2009) Comparison of weight-loss diets with different compositions of fat, protein, and carbohydrates. N Engl J Med 360(9):859–873. https://doi.org/10.1056/NEJMoa0804748

    CAS  Article  PubMed  PubMed Central  Google Scholar 

  47. Dansinger ML, Gleason JA, Griffith JL, Selker HP, Schaefer EJ (2005) Comparison of the Atkins, Ornish, Weight Watchers, and Zone diets for weight loss and heart disease risk reduction: a randomized trial. JAMA 293(1):43–53. https://doi.org/10.1001/jama.293.1.43

    CAS  Article  PubMed  Google Scholar 

  48. Crichton GE, Howe PRC, Buckley JD, Coates AM, Murphy KJ, Bryan J (2012) Long-term dietary intervention trials: critical issues and challenges. Trials 13:111–111. https://doi.org/10.1186/1745-6215-13-111

    Article  PubMed  PubMed Central  Google Scholar 

  49. Katikireddi SV, Egan M, Petticrew M (2015) How do systematic reviews incorporate risk of bias assessments into the synthesis of evidence? A methodological study. J Epidemiol Community Health 69(2):189–195. https://doi.org/10.1136/jech-2014-204711

    Article  PubMed  Google Scholar 

  50. Bjelakovic G, Nikolova D, Gluud LL, Simonetti RG, Gluud C (2012) Antioxidant supplements for prevention of mortality in healthy participants and patients with various diseases. Cochrane Database Syst Rev 3:CD001176. https://doi.org/10.1002/14651858.CD007176.pub2

    Article  Google Scholar 

  51. Chen GC, Neelakantan N, Martin-Calvo N, Koh WP, Yuan JM, Bonaccio M, Iacoviello L, Martinez-Gonzalez MA, Qin LQ, van Dam RM (2019) Adherence to the Mediterranean diet and risk of stroke and stroke subtypes. Eur J Epidemiol 34(4):337–349. https://doi.org/10.1007/s10654-019-00504-7

    Article  PubMed  Google Scholar 

  52. Atkins D, Best D, Briss PA, Eccles M, Falck-Ytter Y, Flottorp S, Guyatt GH, Harbour RT, Haugh MC, Henry D, Hill S, Jaeschke R, Leng G, Liberati A, Magrini N, Mason J, Middleton P, Mrukowicz J, O’Connell D, Oxman AD, Phillips B, Schunemann HJ, Edejer T, Varonen H, Vist GE, Williams JW Jr, Zaza S (2004) Grading quality of evidence and strength of recommendations. BMJ (Clin Res Ed) 328(7454):1490. https://doi.org/10.1136/bmj.328.7454.1490

    Article  Google Scholar 

  53. de Souza RJ, Mente A, Maroleanu A, Cozma AI, Ha V, Kishibe T, Uleryk E, Budylowski P, Schunemann H, Beyene J, Anand SS (2015) Intake of saturated and trans unsaturated fatty acids and risk of all cause mortality, cardiovascular disease, and type 2 diabetes: systematic review and meta-analysis of observational studies. BMJ 351:h3978. https://doi.org/10.1136/bmj.h3978

    CAS  Article  PubMed  PubMed Central  Google Scholar 

  54. Balshem H, Helfand M, Schunemann HJ, Oxman AD, Kunz R, Brozek J, Vist GE, Falck-Ytter Y, Meerpohl J, Norris S, Guyatt GH (2011) GRADE guidelines: 3. Rating the quality of evidence. J Clin Epidemiol 64(4):401–406. https://doi.org/10.1016/j.jclinepi.2010.07.015

    Article  PubMed  Google Scholar 

  55. Meerpohl JJ, Naude CE, Garner P, Mustafa RA, Schünemann HJ (2017) Comment on “Perspective: NutriGrade: a scoring system to assess and judge the meta-evidence of randomized controlled trials and cohort studies in nutrition research.” Adv Nutr 8(5):789–790. https://doi.org/10.3945/an.117.016188

    Article  PubMed  PubMed Central  Google Scholar 

  56. Schwingshackl L, Knüppel S, Schwedhelm C, Hoffmann G, Missbach B, Stelmach-Mardas M, Dietrich S, Eichelmann F, Kontopantelis E, Iqbal K, Aleksandrova K, Lorkowski S, Leitzmann MF, Kroke A, Boeing H (2017) Reply to JJ Meerpohl et al. Adv Nutr 8(5):790–791. https://doi.org/10.3945/an.117.016469

    Article  PubMed  PubMed Central  Google Scholar 

  57. Schunemann HJ, Cuello C, Akl EA, Mustafa RA, Meerpohl JJ, Thayer K, Morgan RL, Gartlehner G, Kunz R, Katikireddi SV, Sterne J, Higgins JP, Guyatt G (2018) GRADE guidelines: 18. How ROBINS-I and other tools to assess risk of bias in nonrandomized studies should be used to rate the certainty of a body of evidence. J Clin Epidemiol. https://doi.org/10.1016/j.jclinepi.2018.01.012

    Article  PubMed  PubMed Central  Google Scholar 

  58. Werner SS, Binder N, Toews I, Schunemann HJ, Meerpohl JJ, Schwingshackl L (2020) Use of the GRADE approach for rating the certainty of evidence in evidence syntheses published in high impact factor nutrition journals: a methodological survey. J Clin epidemiol (under review)

  59. Guyatt G, Oxman AD, Akl EA, Kunz R, Vist G, Brozek J (2011) GRADE guidelines: 1. Introduction-GRADE evidence profiles and summary of findings tables. J Clin Epidemiol. https://doi.org/10.1016/j.jclinepi.2010.04.026

    Article  PubMed  Google Scholar 

  60. Alonso-Coello P, Schünemann HJ, Moberg J, Brignardello-Petersen R, Akl EA, Davoli M, Treweek S, Mustafa RA, Rada G, Rosenbaum S, Morelli A, Guyatt GH, Oxman AD (2016) GRADE Evidence to Decision (EtD) frameworks: a systematic and transparent approach to making well informed healthcare choices. 1: Introduction. BMJ 353:i2016. https://doi.org/10.1136/bmj.i2016

    Article  PubMed  Google Scholar 

  61. Moberg J, Oxman AD, Rosenbaum S, Schünemann HJ, Guyatt G, Flottorp S, Glenton C, Lewin S, Morelli A, Rada G, Alonso-Coello P, Moberg J, Oxman A, Coello PA, Schünemann H, Guyatt G, Rosenbaum S, Morelli A, Akl E, Glenton C, Gulmezoglu M, Flottorp S, Lewin S, Mustafa RA, Rada G, Singh J, von Elm E, Vogel J, Watine J, Group ftGW (2018) The GRADE Evidence to Decision (EtD) framework for health system and public health decisions. Health Res Policy Syst 16(1):45. https://doi.org/10.1186/s12961-018-0320-2

    Article  PubMed  PubMed Central  Google Scholar 

  62. Springmann M, Spajic L, Clark MA, Poore J, Herforth A, Webb P, Rayner M, Scarborough P (2020) The healthiness and sustainability of national and global food based dietary guidelines: modelling study. BMJ 370:m2322. https://doi.org/10.1136/bmj.m2322

    Article  PubMed  PubMed Central  Google Scholar 

  63. Schwingshackl L, Watzl B, Meerpohl JJ (2020) The healthiness and sustainability of food based dietary guidelines. BMJ 370:m2417. https://doi.org/10.1136/bmj.m2417

    Article  PubMed  Google Scholar 

  64. Schünemann H, Hill S, Guyatt G, Akl EA, Ahmed F (2011) The GRADE approach and Bradford Hill’s criteria for causation. J Epidemiol Community Health 65(5):392–395. https://doi.org/10.1136/jech.2010.119933

    Article  PubMed  Google Scholar 

Download references

Funding

Open Access funding enabled and organized by Projekt DEAL.

Author information

Authors and Affiliations

  1. Faculty of Medicine, Institute for Evidence in Medicine, Medical Center, University of Freiburg, Freiburg, Germany

    Lukas Schwingshackl & Joerg J. Meerpohl

  2. Department of Health Research Methods, Evidence and Impact, Department of Medicine, McMaster University, Hamilton, Canada

    Holger J. Schünemann

  3. Cochrane Germany, Cochrane Germany Foundation, Freiburg, Germany

    Joerg J. Meerpohl

Authors
  1. Lukas Schwingshackl
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Holger J. Schünemann
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Joerg J. Meerpohl
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

LS, HJS, JJM contributed to the conception and design of the paper. LS, HJS, JJM drafted this manuscript. LS, HJS, JJM provided critical revisions and all authors have read and approved the final manuscript.

Corresponding author

Correspondence to Lukas Schwingshackl.

Ethics declarations

Competing interests

Lukas Schwingshackl is a member of the Editorial Board of Advances in Nutrition, and a member of the GRADE working group. Holger J Schünemann: is co-chair of the GRADE working group. Joerg J Meerpohl: is a member of the GRADE working group.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Schwingshackl, L., Schünemann, H.J. & Meerpohl, J.J. Improving the trustworthiness of findings from nutrition evidence syntheses: assessing risk of bias and rating the certainty of evidence. Eur J Nutr 60, 2893–2903 (2021). https://doi.org/10.1007/s00394-020-02464-1

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00394-020-02464-1

Keywords

  • Trustworthiness
  • Nutrition evidence
  • Dietary guidelines
  • Systematic reviews
  • Meta-analysis
  • Risk of bias
  • GRADE