Background

To reach the targets set for Millennium Development Goal (MDG) 1, which is to reduce extreme poverty and hunger by half, many countries are on-track in reducing the income poverty while less than a quarter of the developing countries are on-track for achieving the goal of halving undernutrition [1]. The global burden of undernutrition remains high with little evidence of change in many countries despite economic growth. Millions of people are faced with starvation and malnutrition; the State of Food Insecurity (SOFI) in the World 2012 estimated that about 870 million people have been undernourished (in terms of dietary energy supply) in the period 2010–2012, mostly from low-middle-income countries (LMIC) [2]. About one third of deaths in children under 5 years of age are due to underlying undernutrition, which includes stunting, severe wasting, deficiencies of vitamin A and zinc and suboptimum breastfeeding [3]. The majority of an estimated 178 million children under five suffering from stunting are in sub-Saharan Africa and South-central Asia [4]. Around 55 million children are wasted, 19 million of these are severely affected and at high risk of premature death [4]. Altogether, more than 3.5 million mothers and children under five die unnecessarily each year in LMICs due to the underlying cause of undernutrition [5]. With the existing burden of undernutrition, the problem of overweight and obesity among children is also on the rise. In 2011, globally, an estimated 43 million (7%) children younger than 5 years were overweight, an increase from an estimated 28 million in 1990. This trend is expected to continue and reach a prevalence of 9.9% (64 million) in 2025 [6]. Many of the LMICs are said to have the double burden of malnutrition—continued stunting of growth and deficiencies of essential nutrients along with the emerging issue of obesity.

Although there have been several attempts at developing consensus on actions and interventions for nutrition, there is lack of coordination between various academic groups, United Nations (UN) and development agencies, and these are working in parallel silos. The Lancet Nutrition series in 2008 [7-11] and more recently in 2013 [6,12-14] are important attempts at bringing the field together; however, progress by countries has been slow and also hampered by the global increase in food and oil prices; climate change; unprecedented draughts and increased number of countries affected by fragility, conflict and emergencies. As per the World Bank in 2011, 33 countries fall in the fragile situations category as per the World Bank definition [15]. In addition, conflict and fragility also occur at the sub-national level within some strongly performing countries.

Systematic reviews and meta-analysis have become increasingly popular in evidence-based healthcare over the past two decades [16], although these have been evaluated for reliability since quite a long time [17-20]. Rigorous and transparent systematic reviews are recognized internationally as a credible source for evidence of effects and as the basis for evidence-informed policy and decisions [16,21,22]. Within the field of nutrition, systematic reviews have been used to answer many complex questions. A description of systematic review methodology with particular regard to the steps for performing systematic reviews and areas unique to the discipline of nutrition is already available, published by the US Agency for Healthcare Research and Quality (AHRQ) [23]. Furthermore, guidance on assessing equity for users and authors of systematic reviews of interventions has also been highlighted recently to improve the relevance of systematic reviews for both clinical practice and public policy-making [24]. However, several gaps pertaining to the conduct and methodological approaches still exists particularly with reference to nutrition interventions implemented through sectors other than health, such as agriculture, social welfare and trade.

Across nutrition-related systematic reviews, there is multiplicity of reviews in the same subject area with variability in methods such as intervention, target population comparison, outcomes and consideration of contextual variables related to nutritional interventions and outcomes. Additionally, with the increasing attention to social determinants, there is a need to bring in additional qualitative aspects to such reviews. The World Health Organization (WHO) has led the field with a series of activities geared towards development of nutrition policy guidelines based on the best evidence. In 2011, a new online resource, the electronic Library of Evidence on Nutrition Actions (eLENA), was launched that provides access to available sources of evidence and to WHO guidelines, as well as to ongoing clinical trials in the field of nutrition [25].

This overview of systematic reviews from the selected areas in nutrition was produced as a background document for a WHO-convened meeting on improving the quality of systematic reviews of nutrition [26]. The aim of this paper is to assess the process and conduct of systematic reviews (including systematic reviews and systematic reviews + meta-analysis) published in the past 5 years to make recommendations on improving process and methodology of systematic reviews in the field of nutrition.

Methods

We consulted the major stakeholders including leading academic groups engaged in systematic reviews, editors of the Cochrane and Campbell collaborations; International initiative for impact evaluation (3ie); UN agencies including the WHO, United Nations International Children’s Emergency Fund (UNICEF), World Food Program (WFP) and Food and Agriculture Organization (FAO); International Food Policy Research Institute (IFPRI); and interested bilateral agencies including the Department for International Development (DFID), United States Agency for International Development (USAID), Norwegian agency for International Development (NORAD) and representatives from the Bill & Melinda Gates Foundation to identify nine focused interventions under four umbrella areas for our review. These areas were selected based on consultations and WHO interests and included preventive (interventions to prevent undernutrition and micronutrient deficiencies), therapeutic (interventions to treat undernutrition and micronutrient deficiencies), food fortification strategies and delivery platforms (Table 1).

Table 1 Topic areas of landscape study
Full size table

We defined systematic review according to the ‘Cochrane Handbook of Systematic Reviews’ [27] as ‘A systematic review attempts to collate all empirical evidence that fits prespecified eligibility criteria in order to answer a specific research question’. Meta-analysis was defined as ‘the use of statistical methods to summarize the results of independent studies’. An overview of systematic reviews was defined as ‘Systematic and explicit methods to identify select and critically appraise relevant findings from systematic reviews and/ or meta-analysis of effectiveness of interventions’. We included systematic reviews of trials as well as observational studies. We considered all available systematic reviews and overviews of systematic reviews on the predefined interventions. A separate search strategy was developed for each intervention using preidentified Medical Subject Headings (MeSH) and key search terms. A search was conducted in the Cochrane Library, Campbell Collaboration, PubMed and Google Scholar to identify systematic reviews in each of the nine intervention areas. Additional reviews were identified by hand searching references from included reviews. A priori inclusion criteria were outlined stated as below:

  • Systematic reviews pertaining to effectiveness or efficacy of nutrition-related interventions, assessed by prospective studies (including both trials and observational studies), published between 2005 and 2011 were included.

  • Overviews of systematic reviews were also included if they were found to be relevant.

  • Reviews of all languages with English translations were considered eligible for inclusion.

  • Inclusion was not restricted to any particular reported outcome.

Reviews that met the inclusion criteria were selected and double data abstracted on a standardized abstraction sheet for source journal, funding, authors, year, title, target population, intervention, study setting (high-income countries or low-middle-income countries) and comparison, outcomes, types of included studies (randomized trials, quasi trials, before-after, etc.), gaps in knowledge base (as identified by the author), and conclusions. Any disagreements on selection and abstraction of reviews between the two primary abstractors were resolved by the third reviewer. We also assessed each systematic review according to the components from the Cochrane Public Health Review Group guidance including study design; searching non-health databases; assessing risk of bias, ethics and inequality; context; sustainability and applicability [28]. For each systematic review, we assessed the quality of the review by the Assessment of Multiple Systematic Reviews (AMSTAR) rating consisting of 11 items which has been demonstrated to have good face and content validity for measuring the methodological quality of systematic reviews [29]. We then compared the reviews within the same domain with respect to their journals, funding, research question, included study designs and meta-analysis, reporting tools, and definition and assessment of outcomes.

Results

Our search identified 10,926 titles, of which 217 full texts were reviewed and 90 systematic reviews across the nine content areas were included (Figure 1). All eligible texts were available in English, and there was no exclusions based on language. We did not include any unpublished or grey literature review. A wide range of groups were responsible for commissioning and funding the reviews including governments, non-governmental organizations (NGOs), various institutions, UN agencies and private foundations (Table 2). Sixty-eight percent (53/78) of the systematic reviews from the selected nutrition domains were produced from authors based in the high-income countries (HIC) with major contributions from USA, UK, Canada and Australia; thirty-two percent (25/78) of the reviews had authors from the LMICs with Pakistan (13/25) being the major contributing site (Figure 2). From the illustrative areas reviewed, the journal publishing the most systematic reviews in the last 5 years in these nine intervention areas was the Cochrane Database of Systematic Reviews (35/90), followed by BioMed Central (7/90), ‘Journal of Nutrition’ (4/90), ‘The Lancet’ (4/90), ‘American Journal of Clinical Nutrition’ (4/90), ‘Food and Nutrition Bulletin’ (3/90) and ‘Journal of American Medical Association’ (3/90) (Figure 3). The included reviews, and overviews of reviews are summarized using the Cochrane public health guideline grid (Table 3). The assessment of the quality of systematic reviews using the AMSTAR rating, and further description of the attributes of the systematic reviews and overviews, are provided below.

Figure 1
figure 1

Search flow.

Full size image
Table 2 Commissioning and funding of the reviews
Full size table
Figure 2
figure 2

Countries of the lead authors producing reviews.

Full size image
Figure 3
figure 3

Journals publishing systematic reviews in last 5 years.

Full size image
Table 3 Cochrane public health guideline grid
Full size table

Methodological quality-AMSTAR rating

The median overall AMSTAR score was 8 (range 2–11) with methodological quality of Cochrane reviews being fairly consistent with a median AMSTAR score of 10 (range 8–11), while for the non-Cochrane reviews, it ranged from 2 to 11 with a median of 7.5 (Figure 4). From the 11-point AMSTAR criteria, 91% (82/90) of the reviews followed an a priori design, 81% (73/90) did duplicate screening and data extraction, 88% (79/90) conducted comprehensive literature search, 64% (58/90) used status of publication as an inclusion criteria, 44% (40/90) provided the list of included and excluded studies, 34% (31/90) provided the list of included studies only, 74% (67/90) provided characteristics of the included studies, 60% (54/90) assessed and documented the scientific quality of the included studies and used it in interpreting the results, 61% (55/90) used appropriate methods to combine the results, 40% (36/90) assessed publication bias and 82% (74/90) stated the conflict of interest.

Figure 4
figure 4

AMSTAR ratings for the included Cochrane and non-Cochrane reviews.

Full size image

Research question

The research question guides the systematic review and is probably the most important step that defines the population, the intervention, the comparison and the outcomes of interest. We compared research questions for reviews identified on the same intervention to assess for overlap of questions. Although we found several reviews for the same intervention, each review was different because of differences in aspects of the population, intervention, outcome or comparison. For preventive zinc supplementation, two of the included reviews targeted pregnant women, seven targeted children under 5 years of age, one targeted adults while one targeted general population. Besides differences in the target population, reviews evaluated various outcomes; five reviews evaluated childhood infections [30-34], of which three focused on pneumonia alone [31-33]; one on respiratory illness and diarrhoea [30] and one on diarrhoea, pneumonia and malaria altogether [34]. Similarly reviews on multiple micronutrient (MMN) supplementation targeted a variety of age groups including pregnant and women of reproductive age, children, healthy adults, and HIV-infected women and children. Table 4 reports the population, topics and outcomes reported in the included reviews.

Table 4 Population, topics and outcomes reported in the included reviews
Full size table

Included study designs

Good quality randomized controlled trials (RCTs) are considered as the highest quality evidence because they minimize the influence of bias in their results; however, RCTs cannot always be relied upon for conducting systematic reviews as such robust study designs account for only 10% of the healthcare literature [35,36]. Within the selected nutrition interventions, we found that 53% (48/90) of the reviews included RCTs while the others included quasi and observational studies along with RCTs. Most of the Cochrane reviews (73%) included RCTs and quasi randomized studies; however, for most of the non-Cochrane reviews and some specific interventions like breastfeeding, fortification and management of malnutrition, data from pre-post designs were also included.

Quality assessment of the included studies

Table 3 specifies the quality assessment tools used by the reviews in each domain. All of the Cochrane reviews used the Cochrane risk of bias tool to assess the methodological quality of included studies, while some non-Cochrane reviews used various quality assessment scales including the CONSORT checklist, Jadad scale, GRADE quality assessment, US Preventive Task Force Quality Rating and Criteria by Glasziou et al. For fortification reviews, methodological quality was assessed using The Food Standards Australia New Zealand (FSAN) framework for assessing evidence when substantiating nutrition, health and related claims on foods. Some reviews mentioned conducting a quality assessment; however, the methodology used was not clear, while information on methodological quality assessment was found missing or unclear in a few reviews. The overview of the reviews used an informal method of quality assessment that is the Oxman et al. 1991 criteria.

Outcomes standardization and assessment

Standardized, valid outcome measures are pivotal in interpreting findings of systematic reviews, as intervention effectiveness may vary depending on the outcome measurement tools. For example, the specificity of case definitions for acute respiratory infections (ARI) substantially influenced inferences for the effectiveness of preventive and therapeutic interventions [33,37]. Various definitions have been used for defining ARI; however, in some reviews, we found that standardization measures were taken by restricting the inclusion criteria to studies in which ARI case definition aimed to distinguish lower from upper tract respiratory disease; various definitions were also scored for standardization purposes in one of the reviews [33]. Furthermore, despite showing an overall effectiveness of preventive zinc supplementation on pneumonia prevention, the impact changed from statistically significant to non-significant when the definition of pneumonia changed from ‘pneumonia defined by specific clinical criteria’ to ‘lower specificity pneumonia case definition (i.e. age specific fast breathing with or without lower chest in-drawing)’ [31,33]. Few other reviews reported considerable variation in outcome definition across studies, for example definition of exclusive breastfeeding and mixed breast feeding [38], definitions of diarrhoea [30,34] and the definition of health care workers delivering the interventions [39]. In areas where WHO standard definitions exist like undernutrition, reviews have used standard WHO definitions [40-42], while for some reviews, reviewers have self-defined the outcome for standardization [43,44]. Forty three of the included reviews considered contextual factors and conducted sensitivity and subgroup analysis to adjust the outcomes according to these factors.

Meta-analysis and heterogeneity

Nutrition interventions are inherently complex due to multifaceted interventions, diverse settings and outcomes; hence, it is challenging to pool data for meta-analysis which leads to a lack of consensus on effectiveness and contradicting findings. Almost all the reviews reported heterogeneity in the data. Out of 90 included reviews, 62 were systematic reviews with a meta-analysis, 25 were systematic reviews without meta-analysis, while almost 11% (10/90) of the systematic reviews reported heterogeneity as a reason for not conducting meta-analysis. The heterogeneity reported was both clinical as well as statistical due to diverse interventions, target population and outcome definitions. None of the reviews conducted meta-analysis using individual patient data (IPD).

Discussion

Our overview findings pertain to the process and conduct of systematic reviews and not individual studies since we did not assess the primary studies. Our findings suggest that from the 90 nutrition-related reviews, we found considerable variation in methodological quality as assessed by the AMSTAR (ranging from 2 to 11), lack of standardization of outcomes, lack of standardized assessment of risk of bias of included studies, variation in study designs included and variation in how heterogeneity was handled. Each of these methodological choices influence the findings of the reviews, and lack of standardization across these domains increases the complexity for users of systematic reviews in interpreting results—sometimes of conflicting reviews. None of the included meta-analyses used IPD, which provides more detailed analyses and a broader methodological scope due to the presence of raw data. Our analysis shows that these nutrition reviews are comparable to a random sample of systematic reviews across all topics. However, the sample of nutrition reviews had considerably more reviews led by an LMIC author, 32% compared to the 2004 sample where <20% of all reviews had the contact author outside of USA, UK, Canada, New Zealand and Germany [45]. Systematic reviews and meta-analysis have become increasingly popular in evidence-based healthcare over the past two decades [16-20]. The selected systematic reviews on nutrition came from varying sources as the nutrition-related interventions are not solely implemented by health sector. There is an evident gap of quantitative data in some major areas concerning nutrition like undernutrition that direly needs policy guidance. We found considerable overlap between the topics and questions addressed in multiple reviews; however, the outcomes reported varied.

Overall, the methodological quality of the Cochrane reviews was found to be better, which is not surprising since they are conducted according to standardized guidelines published in the ‘Cochrane Handbook’. However, within the Cochrane Collaboration, several editorial groups have undertaken nutrition-related reviews, and while the broad methods are consistent, there is variability in nutrition-specific decisions about the methods such as eligibility criteria for population, intervention, comparison and outcomes and consideration of contextual variables. Furthermore, in some areas like conditional cash transfers (CCTs), the scope is very broad and thus fails to focus on nutrition-specific indicators. There is a need for improved justification of methodological choices in nutrition systematic reviews. We propose the need to follow standards for conducting systematic reviews such as the ‘Cochrane Handbook’ [46], as well as the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines for reporting systematic reviews and meta-analyses [47]. With respect to public health nutrition, there is a need for standardization in defining of the population, intervention, comparison and outcomes, as well as a need for attention to aspects of context, appropriate study designs, sustainability and applicability. For outcomes, there is a need to standardize the measurement and definition of outcomes. The Core Outcome Measures in Effectiveness Trials (COMET) initiative is a major opportunity to explore this for nutrition. The differences in methods used are consistently observed in both Cochrane and non-Cochrane reviews, suggesting a need for improved guidance for nutrition reviews.

Existing evidence related to systematic review conduct in the field of nutrition highlight the fact that systematic reviews in nutrition are unique and challenging as it encompasses various nutrition-related considerations including baseline nutrient exposure, nutrient status, bioequivalence of bioactive compounds, bioavailability, multiple and interrelated biological functions, undefined nature of some interventions and uncertainties in assessment [23]. Developing capacity-building programmes, searching the primary literature for research gaps and extending reporting tools such as the PRISMA Statement to the field of nutrition are recommended to improve the conduct of systematic reviews in nutrition [48,49].

One strength of this paper is the assessment of methods used in systematic reviews of nutrition using accepted tools including the AMSTAR and the ‘Cochrane Handbook of Public Health Guidance’. One limitation is that we used a non-validated search for nutrition topics and thus may have missed some systematic reviews. However, we think this limitation is unlikely to affect the findings of this study since we intended to identify a purposive sample, rather than an exhaustive sample of systematic reviews in nine intervention areas. Furthermore, our assessment of quality of the included systematic reviews is limited to how they were reported.

Given the growth in the field, there is a clear need to develop a consensus on methodologies for nutrition reviews, criteria for assessing the evidence and possibly facilitating development and collation of the evidence in the subject area. There also needs to be clarity on the purpose of the reviews and on the questions that they are addressing, for them to be useful in the context of programme and policy guidance. The diverse methods involved in such systematic evidence syntheses and conclusions thereof have a clear bearing on impact on policy and practice. Given the differing needs of information syntheses by practitioners and public sector policymakers, a consensus on the quality indicators for such reviews and methods is an important prerequisite. While the Cochrane and Campbell collaborations have a due process with published guidelines within the ‘Cochrane Handbook’, other journals and peer review processes may not follow a standard set of criteria. Cochrane and Campbell are well-positioned to take the lead in improving quality as our analysis also suggests that the quality ratings for the Cochrane reviews were found to be fairly consistent as compared to the non-Cochrane reviews. The Campbell International Development Nutrition group is actively working on these priority areas of methodological quality, outcome standardization and avoiding duplication. Engagement with a range of stakeholders could lead to wider uptake and endorsement of such standards and priorities. This could then expand to include guidance to other journals and agencies evaluating and publishing nutrition-related systematic reviews.

Conclusions

To conclude, there is a need to develop a consensus on methodologies for nutrition reviews, criteria for assessing the evidence and possibly facilitating development and collation of the evidence in the subject area. We therefore propose the development of a potential nutrition review editorial process and a group inclusive of major stakeholders in the field to help bring the field together.