What are the impacts of urban agriculture programs on food security in low and middle-income countries?
Issues of food security and nutrition have wide reaching implications for people and their environments, particularly in low and middle-income countries. One proposed solution is urban agriculture, which has been widely upheld as a solution to the food-crisis facing increasingly metropolitan populations. It is believed to provide the urban poor with food and a source of potential income, whilst improving the urban environment and reducing pressure on finite farmland. Although it faded from many development agendas in the 1990’s, urban agriculture has seen a resurgence since a peak in global food prices in the late 2000’s. There are, however, potential disadvantages to this increasing drive for urban agriculture including associated urban health risks and implications for the environment. The usage of waste-water, for example, may contaminate produced food and intensive irrigation might lead to the spread of malaria and water borne diseases, as well as threatening already limited water supplies. Soil erosion and the intensive use of fertilizers and pesticides might also present health risks to urban populations and damage the environment. Despite the potential benefits and harms of urban agriculture, the evidence-base is not well understood. Given the current policy drive to promote urban agriculture, there is an urgent need to understand its effects on urban populations and their environments.
This review will seek out, select, appraise and synthesise evidence on the impacts of urban agriculture on food security and nutrition. We will employ systematic review methodology to ensure that our review of the evidence is comprehensive, transparent and replicable. In addition to searching electronic databases, we will examine websites and contact academics, practitioners and policy-makers for relevant research. All potentially relevant literature will be screened against pre-specified criteria and assessed for risk of bias using established critical appraisal tools. This is to ensure that we only include the evidence in which we have confidence. Depending on the nature of the available data, we will then synthesise the available evidence using statistical meta-analysis and/or narrative synthesis. Our findings will be disseminated in a variety of ways to ensure that the evidence is available for policy-makers and practitioners.
KeywordsUrban agriculture Food security Nutrition Impact Systematic review Urbanisation
The emergence of urban agriculture
The twenty-first century has often been described as ‘the first urban century’. Unprecedented rural–urban migration has led to rapid urban growth. Whilst in 1900 a mere 13 per cent of the world’s population lived in urban areas, the UN-Habitat  estimates that by 2030 this level will have risen to 60 per cent. Furthermore, virtually all of this population growth over the next few decades will be absorbed by cities in low and middle-income countries, thus increasing the pressure on urban resources and administrations that are often already exhausted.
Among the most pressing needs of any urban agglomeration is the question of urban food security and ensuring the right to food. Urban populations depend on the reliable and stable availability of food products, as well as affordable and convenient access to them. High levels of urban income poverty paired with rising food prices, however, often make the formal urban food supply system unaffordable and inaccessible to the urban poor. An informal supply system, consisting of street vendors, informal markets, home-based enterprises as well as urban agriculture (UA), exists alongside formal interventions. These informal networks predominantly satisfy the urban poor’s demand for easily accessible – though not often cheaper – foodstuffs.
Approaches to urban agriculture
Whilst urban decision makers and academics alike have identified UA as the most beneficial and promising pillar of informal food supply systems [2–7], the evidence for such claims is unclear. Although UA has been an integral part of urban livelihoods throughout human history , the concept only came to the fore in the late 1980s/early 1990s, evoking interest among international donors and development practitioners. A United Nations Development Program (UNDP) report compiled by Smit et al.  estimated that it reached, in the early 1990s, some 800 million urban dwellers who used UA as a livelihood strategy. Some scholars [8, 9] even argue that UA presented some kind of a ‘magic bullet’ or panacea to eradicate hunger and poverty in urban areas. A number of studies with promising titles such as ‘hunger-proof cities’, ‘Agropolis’ and ‘Cities feeding people’  indicate the potential associated with UA. Critics nevertheless quickly pointed to the weak empirical evidence of some of these studies and the low overall scale of UA amongst urban poor . During the first years of the urban century, UA had therefore slipped past the focus of the international development community. Yet the peak of global food prices in 2008 shed a new light on the idea of locally produced food products and households’ subsistence production. UA subsequently once more was portrayed as a major intervention to improve urban food security .
Renewed interest in UA amongst scholars and policy makers is a positive development since local and international environments have changed greatly since the 1980s and 1990s, when most of the initial research on the concept was conducted . In its Growth and Development Strategy (GDS) 2040, the City of Johannesburg, for example, identifies UA as its main intervention to address food security within the city . On a global scale, the UN High Level Task Force on the Global Food Crisis  identified UA as a strategy to alleviate urban food insecurity and build cities that are more resilient to crisis. A joint World Bank (WB)/Food and Agricultural Organisation (FAO) paper of the same year ( : 5) also expressed that “the World Bank and FAO, … will promote [urban poverty alleviation] related programs and projects in the context of the MDGs and more specifically MDG1 ‘Eradicate extreme poverty and hunger’ and MDG7 ‘Ensure environmental sustainability’.” These two MDGs seem interrelated because food insecurity is associated with poor environmental management in terms of, for example, over-fishing in lakes, bushmeat and killing of wild herds (Bow year), and also with an exploited environment. This makes achieving food security difficult. The FAO, moreover, published the Urban producers resource book (2007) as an outcome of its ‘Food for the city’ program. This program forms part of a wider network of organisations, consisting of the UNDP/UN-Habitat ‘Sustainable Cities Program’, the IDRC’s ‘Urban Poverty and Environment’ program, and the Resource Centre on Urban Agriculture and Food Security (RUAF), which all strongly advocate the recommendation of UA as a tool to address urban food insecurity. In light of this strong support for UA one must note, however, that some fundamental questions regarding this intervention remain unanswered. Renewed interest in the topic did not necessarily converge with new knowledge about UA; little is known about the true extent and impact of UA in urban livelihoods.
Definitions of urban agriculture
Urban agriculture is not easily defined, as a large variety of urban farming systems exist internationally, with varying characteristics depending on local socio-economic, geographic and political conditions. The most widely used definition of UA was developed by Luc Mougeot . Using technical criteria of UA he explained that,
(u)rban agriculture is an industry located within (intra-urban) or on the fringe (peri-urban) of a town, a city or a metropolis, which grows and raises, processes and distributes a diversity of food and non-food products, (re-)using largely human and material resources, products and services found in and around that urban area, and in turn supplying human and material resources, products and services largely to that urban area. 
Urban agriculture, food security and nutrition
UA is thought to increase food security through two main pathways: improved access to food, and increased income . Home-grown foodstuffs increase the total amount of food available to a household and thus can prevent hunger and malnutrition. At the same time the availability of fresh, home grown food products, in particular fruits and vegetables, advances the nutritional status of household members and thereby improves health. Direct access to food often allows particularly poor households to consume a more diverse diet than they would otherwise be able to afford. Especially, animal husbandry is believed to provide an important source of animal protein, which is commonly limited in poor households’ diets due to income constraints.
Recent studies on UA and its impact on nutrition focus on dietary diversity and kilocalorie consumption as two main aspects which influence the outcome of improved nutrition . In his analysis of child nutrition and UA in Kampala, Maxwell  also connected the aspect of maternal care to UA, arguing that mothers engaged in UA, as opposed to other forms of non-farm employment away from home, have an increased ability to care for their children. This was in return believed to positively impact levels of child nutrition. However, even proponents of UA highlight the fact that there is currently no detailed, empirical evidence for UA’s impact on nutrition levels [21, 22]. This is also brought up for discussion in the International Food Policy and Research Institute’s discussions on the links between UA and nutrition (see ).
Secondly, UA is assumed to create an ‘opportunity cost’ – domestic producers can either save income, via the consumption of home-produced foodstuffs that are cheaper to produce than to buy from the market, and/or increase income by selling or trading their products. Addressing urban food insecurity therefore requires a balancing act between urban agriculture (the opportunity cost of producing your own food) and more efficient urban food markets (making the food you buy cheaper). The latter may be a result of urban or peri-urban farmers’ contributions to the markets. Higher cash income at the household level is then positively linked to food security as households are believed to have greater access to food products, both in terms of quantity and quality. This relationship however, to large extent, depends on the calorie elasticity of income – that is, the extent to which a change in income leads to a meaningful change in calorie intake. Households with low-income elasticity, for example, might not experience improved levels of nutrition linked to an increase in income. Given the low input costs of UA, most scholars nevertheless believe it to have great potential in addressing urban poverty and food insecurity .
Critics of urban agriculture
Although Mougeot  claims that “little could be found in the academic literature which would condemn UA at large and advocate its ban under any form”, and despite presenting hostile urban planners and policy makers as the main critics of UA, the concept is not as universally supported as Mougeot wishes the reader to believe. The absence of reliable empirical data on the scale and impact of UA must be named as one of the main limitations of UA and may also explain urban planner’s reluctance to embrace the concept. The above cited number of 800 million people involved in UA globally provides good insight into the quality of research regarding UA. Even though this number is widely cited in background sections and fact sheets on UA, it is merely based on the personal estimation of Jac Smit, an outspoken proponent of UA. His estimations came as result of a number of different national surveys and UNDP-funded tours through major urban centres in the developing world during the early 1990s. Accurate data on the scale of UA is further limited by different survey designs and definitions of the concept. While scholars (e.g. [25, 26]) once identified Lusaka as the UA capital, a recent baseline study by the African Food Security Urban Network  found that only three percent of households in Lusaka currently use UA as a livelihood strategy. The same problem arises with regards to believed impact of UA and urban livelihoods. Ellis and Sumberg  point to the absence of control groups in research regarding UA’ s impact, and criticise that “(UA) claims too much by equating all food production in towns with improved food security for poor people”.
Apart from the perceived absence of empirical evidence on UA, scholars have also pointed to urban health risks associated with UA and the implications of these for the environment. The usage of waste water, for example, is feared to contaminate produced food and intensive irrigation might lead to the spread of malaria and water borne diseases, as well as lowering the water table . Soil erosion and the intensive use of fertilizers and pesticides might also present health risks to urban populations, as well as cause environmental harm. Furthermore, both large- and small-scale food production is associated with deforestation, draining of water reservoirs, etc. (e.g., [28–30]). In addition to that, Tevera  believes that UA often excludes the most vulnerable groups in society through a lack of legal tenure, capital and policy support. Land use planning in urban areas often implies that once land that has been used for urban agriculture rises in value, the land is removed from agricultural use . Only if and when city administrations set aside dedicated pieces of land for urban agriculture with low opportunity costs to those intending to use it, will potential users be able to use it for improved access to food or in order to generate income. Furthermore, evaluations of the value of urban agriculture must consider the value of alternative uses of this land. Some go so far as to question whether UA is an oxymoron , and although they conclude otherwise - that urban agriculture has the potential to improve urban lives – the evidence presented is limited and not systematically reviewed. As a result there is very little reliable knowledge on the actual scale of UA and its impact on urban livelihoods. It is also uncertain as to whether UA has a positive impact on urban livelihoods, which types of households are able to benefit from it, and whether UA’s negative side effects outnumber its positive contributions.
Urban agriculture and the need for a systematic review
There is thus an evident need for a systematic approach to gather and integrate available data on UA. Cofie et al.  underline that urban planners and decision makers cannot be expected to base their policy recommendation on what Zezza and Tascotti  call “qualitative if not anecdotal evidence”. The methodology of a systematic review provides the means of for identifying, synthesising and assessing the findings of various rigorous studies (both quantitative and qualitative) to answer a focused question. Petrosino and colleagues describe systematic reviews as “the most reliable and comprehensive statement about what works” (in ).
Lastly, a systematic review by the Dutch Ministry of Foreign Affair’s Policy and Operations Evaluation Department  focused on the question of which agricultural interventions improve food security. It identified increased production, improved value chains, markets regulations and safe legal tenure as the four main pillars in increasing food security. Although the review claims that it considers urban food security, it neither differentiates between rural and urban food security impacts in its synthesis, nor does it account for such distinction in the reviewed impact studies. According to the review, improved irrigation and the use of genetic crop modification have been found to increase agricultural production in the most significant ways. Value chain interventions benefited farmers through improved income from the sale of cash crops. Market deregulations had an ambivalent impact on food security. Whilst the decrease of monopolies and the reduction of government involvement were believed to increase food security during crisis situations, the long-term impact differed greatly amongst the studies. Lastly, improved land tenure benefited food security in all reviewed cases. This finding might be important in the context of UA, as it supports proponents’ calls to give urban farmers access to public land and change discriminative urban policies to legalise and support agricultural interventions in urban areas. Since we do not know to what extent urban famers engage in UA for commercial reasons, it is difficult to assess the scope of the impact of interventions that focus on value chains and market regulations. Individual and household farmers – who represent the majority of urban farmers and practice agriculture on mainly small plots within the city – might not be able to make use of such techniques to increase their agricultural output. The related systematic reviews cited above have shown that increased food production does not directly link to improved nutrition.
Therefore, the impact of UA on food security and nutrition levels remains unanswered. Much has been published on the topic assuming a positive relationship, and the concept enjoys the outspoken support of international development agencies such as IDRC and UNDP. Few studies, however, have generated reliable data on the scale and impact of UA. There is also an absence of a coherent approach to define and measure the scale and impact of agricultural interventions in urban areas. This systematic review is therefore set to address these shortcomings by providing an evidence-based review of all relevant data on the topic. It will map out the available evidence and seek to draw out implications for those involved in UA initiatives on all possible levels by providing indications on if and/or how to use agricultural interventions to improve levels of urban food security and nutrition.
Objectives of this review
This review aims to address the gap in our knowledge about UA in low and middle-income countries. It seeks to provide a solid evidence base for policy makers, practitioners and members of the international donor community on the feasibility, benefits and cost of urban food cultivation. Specifically, this review will answer the question: what is known about the impacts of urban agriculture programs on food security and nutrition in low and middle-income countries? In addressing this question, we will focus on outcomes that measure levels of food security at the individual, household and/or community levels.
According to the official definition adopted at the 1996 World Food Summit in Rome “food security exists when all people, at all times, have physical and economic access to sufficient, safe and nutritious food to meet their dietary needs and food preferences for an active and healthy life” (quoted in : 7). This reflects a broad approach to food security, including the elements of food availability, food accessibility, food reliability, food quality and food preference . Food security is most commonly defined in terms of the three pillars of: availability (including consistency of that availability), access (with the specification of not just access to food, but access to sufficient food for a nutritious diet), and use . The FAO add a fourth pillar, that of stability and apply it to all three of the others i.e. the stability of availability, of access and of use .
Levels of food security and nutrition are interdependent with households’ socio-economic status. The economic impact of the usage of UA as an income-generating scheme in order to purchase more or different food is therefore also considered in this review. Sales of domestically produced food stuffs can be measured either in terms of quantity or with regard to their monetary value.
identify which assessments have been conducted on the impact of UA on food security in low and middle-income countries;
synthesise what existing available evidence tells us about the extent to which UA increases food security and improves nutrition;
depending on the available evidence, use the understanding we gain from the literature to test the causal chain outlined in Figure 2 above, and to develop it further; and,
draw out implications for policy, practice and research.
We will employ systematic review methods as promoted by the Collaboration for Environmental Evidence. In doing so, we seek to identify all the relevant and reliable research on UA and its impact on food security and nutrition, and to synthesise it in meaningful ways. Whilst we will combine the most reliable evidence using structured synthesis, we will not discard any evidence rated ‘medium reliability’ from which we feel we can learn, but will report their findings with a ‘health warning’ as to their potential for bias.
This systematic review process includes searching comprehensively for all available potentially relevant evidence, and then filtering it firstly, for evidence and secondly, for risk of bias. In doing so, we will use structured approaches to describe and critique the available research. All data will be recorded on specialist systematic review software (EPPI-Reviewer 4) to enable transparent and accurate analysis.
We will search for relevant literature using the search terms presented below. Searches will be conducted in English, Spanish and Portuguese. This is grounded in the findings of preliminary searches, conducted in June 2012, that much of the evidence base is found in sub-Saharan Africa and Latin America. Title and abstracts of any literature identified in other languages will be translated using Google Translate, and any studies which appear to be relevant will then be translated and considered for inclusion in the review.
‘Urban’ AND ‘agriculture’
(1 OR 2) AND 3
The above terms have been identified and tested using Science Direct. They will be adapted as necessary and applied to the following electronic databases and websites:
African Journals Online
Asia Journals online
Centre for Agricultural Bioscience International
Campbell Collaboration Library
Cochrane Collaboration Library
Collaboration for Environmental Evidence Library
DAC Evaluation Abstracts
Database of impact evaluations (3ie)
Emergency Events Database
Internet library sub-Saharan Africa
Isidore: Open Access Portal (French)
Online Access to Research in the Environment
Resource Centres on Urban Agriculture & Food Security RUAF
Social Science Research Network
United Nations Economic Commission for Africa (UNECA) institutional repository
Searches will be limited using specific ‘human’ filters, and exclusively include literature since 1980 as only isolated UA initiatives existed before then.
Citation searches for the following key publications will be conducted using Google Scholar.
● Atkinson, S.J. (1992). Food for the cities: Urban nutrition in developing countries, Public Health and Policy Publication No. 5, London School of Hygiene and Tropical Medicine, London.
● Binns, T. and Lynch, K. (1998). Feeding Africa’s growing cities into the 21st century: The potential of urban agriculture, Journal of International Development, Vol. 10: 777–793.
● Bishwapriya, S. (1985). Urban agriculture: Who cultivates and why? A case-study of Lusaka, Zambia, Food and Nutrition Bulletin, Vol. 7(3), 15–24.
● Cofie, O.O., Van Veenhuizen, R. and Drechsel, P. (2003). Contribution of urban and peri-urban agriculture to food security in sub-Saharan Africa. RUAF paper presented at the 3rd WWF in Kyoto.
● Crush, J., Hovorka, A. and Tevera, D. (2011). Food security in Southern African cities: The place of urban agriculture, Progress in Development Studies, Vol. 11 (4), 285–305.
● Drescher, A.W. (2004). Food for the cities: Urban agriculture in developing countries. SHS Acta Horticulturae 643: International Conference on Urban Horticulture.
● Ellis, F. and Sumber, J. (1998). Food production, urban areas and policy responses, World Development, Vol. 26 (2), 213–225.
● Maxwell, D.G. (1995). Alternative food security strategy: A household analysis of urban agriculture in Kampala, World Development, Vol. 23, 1669–1681.
● Maxwell, D.G. (1998). Does urban agriculture help prevent malnutrition? Evidence from Kampala, Food Policy, Vol. 23 (5), 411 – 424.
● Mbiaba, B. (1995). Classification and description of urban agriculture in Harare, Development Southern Africa, Vol. 12 (1), 75–86.
● Rakodi, C. (1985). Self-reliance or survival? Food production in African cities with particular reference to Zambia, African Urban Studies, Vol. 21, 53–63.
● Rogerson, C.M. (1992). Feeding Africa’s cities: The role and potential for urban agriculture, Africa Insight, Vol. 22 (4), 220–234.
● Sawio, C. (1993). Feeding the urban masses? Towards an understanding of the dynamics of urban agriculture in Dar es Salaam, Tanzania, Ph.D. Thesis, Clark University, Worcester.
● Tevera, D. (1999). Urban agriculture in Africa: A comparative analysis of findings from Zimbabwe, Kenya and Zambia, African Urban Quarterly 11 (2/3), 181–7.
● Simatele, D., M. and Binns, T. (2008). Motivation and marginalization in African urban agriculture: The case of Lusaka, Zambia, Urban Forum, Vol. 19 (2), 1–21.
● Zezza, A. and Tascotti, L. (2010). Urban agriculture, poverty and food security: Empirical evidence from a sample of developing countries, Food Policy, Vol. 35, 265–273.
In addition, reference lists of all the included studies will be checked for further relevant papers, and citation searches will be conducted to identify other applicable literature.
Inclusion and exclusion criteria/screening search results
We will employ a two stage screening process, screening all search hits’ abstracts and titles according to the following criteria. Full texts of all potentially relevant studies will then be obtained. (If in doubt of their relevance, we will collect full texts). In the second stage, all full texts will be screened according to the same criteria as below. Two reviewers will work independently on this task and any disagreements will be discussed and resolved with a third reviewer. A flow chart showing the flow of references through these stages, and the following stage of critical appraisal, will be produced for readers.
Studies which meet all of the following criteria will be included in this review.
Only research conducted in countries classified as Low or Middle-income countries (LICs/MICs) by the World Bank  will qualify to be included in the review. Research that focuses on both LICs/MICs and High Income countries (HICs) will be considered as long as it is possible to isolate the impact of UA on the former.
Only impact evaluations which set out to measure the effectiveness of UA interventions on urban food security at the household, individual and/or community level, as compared to the effects of not engaging in UA, will be included in this review. Studies that do not provide a comparison group will be excluded. Previous reviews (if available) will not be included, but their reference lists will be searched and relevant studies will be included in our review.
This review will include UA in all its forms when used as a livelihood strategy. This can include growing plants to eat or sell (for example, herbs, fruit, vegetables or flowers) and animal husbandry. Urban Agriculture when purely used as a leisure activity, such as home or roof top gardens that are not intended to contribute to either food or income in the household, will be excluded from the review.
The review focuses on people in urban and peri-urban contexts within LMICs, who use forms of UA. We will not exclude any group of people on age or socio-economic group, but will classify studies according to the population and conduct sub-group analyses, if appropriate.
This review will focus on food security, including changes in access to, and quality of, food. Studies that do not include either one of the two will be excluded. Nutrition refers to both access and quality of food; we will therefore also include studies which assess the impact of UA on nutrition levelsa.
As UA can lead to a change in income levels, which in turn can have an effect on food security, we will include studies that assess impacts of UA on income when the study also relates to food security. Studies which address impacts on income with no link made to food security will be included, but discussed separately. Studies that only focus on the environmental and social aspects of UA will be excluded.
Studies will not be excluded from this review on the basis of language. The review team has the scope to translate studies published in English, French, Spanish, Portuguese, German, Dutch, Afrikaans, Zulu and Sotho languages. Abstracts of identified papers in other languages will first be translated using Google Translate and, if deemed relevant for inclusion in the review, will be translated fully.
Date of publication: All studies published since 1980 will be included in this review.
Potential effect modifiers and reasons for heterogeneity
Potential effect modifiers and reasons for heterogeneity in this review have been identified from the literature and in consultation with experts in our informal advisory groupb.
As suggested above, there are a number of factors that are likely to shape how UA impacts on the food security and nutrition of individuals and groups of research participants, i.e. effect modifiers. Food security may depend on their ability to store food for future consumption, and their knowledge of alternative forms of food stuffs (some groups may not view certain plants as food, whilst others consume them regularly). Access to food may also vary according to age, gender and status within the community. An individual’s nutritional status will also vary considerably depending on their previous levels of nutrition, their age, gender, and health status, as well as the size of their families and the age of their children. Research participants’ success in generating income from UA will also vary. We know from studies of microfinance that soft factors, such as entrepreneurial ambition and ability, vary and are difficult to measure. Yet these are likely to influence beneficiaries’ levels of food security.
A number of these influencing factors are time- and context-bound, and are therefore likely to vary across studies within our review. We therefore anticipate dissimilarity in the results on the basis of variation in climates, cultural expectations, and the level of food security within the country/city as a whole. Food security is both seasonal and varies from year to year, with previous years’ harvests impacting on the next, and a population’s capacity to cope with shortages determined in part by their nutritional history. Access to food is further shaped by conflict, absolute and relative poverty, and in some communities, to the relative status of particular ‘ethnic’ groups, women, and children. Where possible, we will code for these factors in our review and consider their impact on the findings of individual studies as well as on our review results.
Sub-group analyses of these effect modifiers are described under section 3.7.
descriptors of the study, including when it was conducted, where, by whom, and how it was reported;
descriptors of the intervention assessed (the nature of the UA intervention assessed, the theory of change used, and the desired outcomes)
descriptors of the methodology employed, including the study design, population, data collection, and data analysis; and,
the outcomes assessed and the measures used, including an indication of whether the measures were objective, using recognised scales, and/or self-reported data.
These codes will be applied by two researchers independently, i.e. blind to one another’s coding. The results of this will be compared until inter-researcher reliability is over 90%. Once this level is achieved, two researchers will continue to be involved, but they will work independently of each other – one person will code and the second will check their coding for accuracy. Any disagreements will be resolved through discussion with a third researcher.
By coding the available research in this way, we will be able to map out the evidence of UA and ensure the relevance of the research we have collated. We will then apply the following pre-determined critical appraisal criteria to our pool of relevant research as explained below.
Assessment of the risk of bias within studies
We will draw on published critical appraisal tools used by the Cochrane Collaboration in their health care systematic reviews, namely the Risk of Bias Tool , and the Grades of Recommendation, Assessment, Development and Evaluation (GRADE) approach . Used in combination, these are designed to assess potential biases within a study and to help decide how much confidence we can have in its findings when integrating these into the review synthesis. The tools include consideration of selection bias, performance bias, detection bias, attrition bias and reporting bias. These are then combined within GRADE, which focuses clearly on each research question addressed in the review and on each outcome of interest, the available evidence to address each question and outcome, and its associated ‘risk of bias’, and a structured evidence trail from the data to our review’s conclusions.
These Cochrane tools have been largely, although not exclusively, designed for critically appraising randomised controlled trials. However, as noted above, we anticipate identifying studies that employ other study designs for assessing the impact of UA on food security and nutrition. We will none-the-less apply the above Cochrane tools to all relevant evidence, making some adjustments for study design. In doing so, we will draw on our experiences of conducting systematic reviews in the area of microfinance . For these systematic reviews, we critically appraised a wider range of study designs, including non-randomised controlled trials, retrospective assessment of panels of data using multiple regression and independent variables (statistical analysis of large longitudinal data sets), and simple with-without studies (with comparison groups, but no ‘before’ data).
By critically appraising the relevant research in these ways we will be able to determine which studies have a ‘high’ risk of bias and which of those are of ‘medium’ or ‘low’ risk of bias. All studies will be included and their findings extracted. However, findings of high-risk bias studies will be reported separately and will not be included in statistical meta-analysis. Critical appraisal will be conducted by at least two reviewers, and any disagreements will be resolved through discussion with a third reviewer.
We will employ, as far as is possible, Cochrane’s Risk of Bias and GRADE tools. In translating these for use in a broader range of literature, we will draw on our own, and the EPPI-Centre’s experience of critically appraising international development literature. In doing so we will draw on the tools used in two completed EPPI-Centre reviews ( ; 2012), a Campbell Review (Tripney et al. – under peer review) and one ongoing 3ie/Campbell Review (Stewart et al. undergoing peer review).
Data extraction of findings
Having narrowed our pool of research to those studies that are of ‘medium’ or ‘low’ risk of bias, we will move on to extract the findings relevant to the impact of UA on food security and nutrition. Findings will be extracted from the primary report of each study, as well as from any identified linked reports on the same data. We will extract findings and record these on our EPPI Reviewer database. As with the coding process described above, two researchers will be involved in this data extraction process, initially working entirely independently and eventually working together, but always with a second person checking the extracted data to avoid any error or bias in the extracted findings.
Data synthesis and presentation
Findings will be synthesised using framework synthesis, which applies pre-determined categories to the data and enables structured comparison and synthesis. As described below, if available and comparable, quantitative results from comparative studies will be combined statistically and other findings synthesised using structured matrices.
Where data or information on the methodology of primary studies are missing or reported inconclusively, we will contact the study investigators. We will not replace missing data.
For studies reporting dichotomous outcomes, both the risk ratios (RRs) and standard mean difference (SMD) effect sizes (Cohen’s d) will be calculated. We anticipate encountering the use of a continuous outcome by some authors, and a dichotomous outcome by others. For each outcome category, we will determine the number of coded effect sizes in the different metrics. Where more than one type occurs in a given outcome category, we will transform the effect size metric with the smaller proportion into the metric with the larger proportion using the Cox transform. This will allow all the effect sizes for that outcome category to be analysed together. We have described this in the protocol.
Studies will be sorted into the matrix below. We will then focus initially on synthesising the findings of:
● Comparative outcome evaluations which measure the impact of urban agriculture on food security and nutritional status directly (1a, 1b &1c below)
● Comparative outcome evaluations which measure the impact of urban agriculture on food security and nutritional status via the generation of income (2a, 2b & 2c below).
RCT evidence only (i.e. 1a & 2a only)
Trial evidence only (i.e. 1a, 2a, & 1b, 2b, only)
All comparative outcome evaluations (i.e. all cells below).
Combined results reporting in reliable studies
Assessing impact on food security and nutrition directly
Assessing impact on food security and nutrition via the generation of income
Randomised control trials
Non-randomised control trials
Other comparative outcome evaluations
Where data are available from studies with comparable interventions, populations and contexts, the findings of comparative study designs will be synthesised using statistical meta-analysis. Specifically we will combine, using statistical meta-analyses, the results of those interventions where all of the following statements are true:
● The intervention evaluated incorporates the same dimensions of urban agriculture (e.g. food production or animal husbandry).
● The study design for evaluating impact is the same.
● The risk of bias within the study is rated as medium or low in our critical appraisal (see above).
We will calculate effect sizes where possible. Studies will be grouped according to the intervention evaluated. Where possible, effect sizes will be standardised and analysed using funnel plots to assess publication bias and to consider the extent of variation between study results. Forest plots and Chi-Squared tests will be used to assess the size and significance of heterogeneity and possible reasons for such differences will be explored. Depending on these analyses, findings may be synthesised using pooled effect sizes illustrated on forest plots, or may be limited to framework synthesis, which applies pre-determined categories to the data and enables structured comparison using matrices.
If the group of studies draws from heterogeneous samples, we will use a random effects model, and if drawn from homogeneous samples, then a fixed effects model will be used. If studies use the same continuous outcomes measure, we will calculate the weighted mean difference score and if studies use different outcome measures, we will pool effects using the standardised mean differences of each study.
In all other cases, we will synthesise findings by using a matrix, describing:
● The direction of the effect;
● The size of the effect (if available).
This is an approach often used by the EPPI-Centre. Examples can be found in our recent reviews of microfinance ( , Stewart et al. 2012).
In order to explore not only whether UA impacts on food security and nutrition, but also when and in what circumstances, we will focus specifically on the following sub-groups of interventions:
● Different types of urban agriculture;
● Interventions targeting specific populations;
● Interventions within low-income countries;
● Interventions with middle-income countries.
By doing this, where the data is available, we will be able to compare and contrast the implications of these different characteristics. This will inform our understanding of how urban agriculture interventions might be working, and help us to think through, in more detail, revisions to our original proposed causal pathway (Figure 2). Sub-group analysis will be interpreted with caution (between study variation can be due to a number of observable and unobservable characteristics). Furthermore care will be taken to ensure that the sub-group characteristics are interpreted with caution, exploring the logic behind how each modifier may be working.
We do not anticipate having sufficiently complete data to conduct mega-regression. However, if this is available, we will conduct a meta-regression as it can be valuable in providing a picture of the relative interactions of variables. Our meta-regression findings will be interpreted with caution and acknowledged as providing an incomplete picture; however findings will inform our revisions of the causal pathway (Figure 2) on how urban agriculture impacts on food security.
The review team will meet to synthesise their findings and discuss the implications for policy, practice and research.
Initial conclusions and implications will be circulated to our virtual advisory group for their input. Amendments will be made in light of any feedback. This will allow consideration of wider forms of policy and practice knowledge, and provide an opportunity for researchers to inform us of any relevant research. We will hold further meetings following the formal peer review to decide on our ultimate conclusions and implications and write our final report.
We intend to write up the findings and methods of this review for publication, and to present our findings at relevant conferences.
a As noted by Masset et al.  ‘Nutritional outcomes in particular are objectively measurable with considerable precision. The measurement of micronutrient intake can be complex if based on blood samples or on detailed expenditure surveys, but anthropometric measurements … require a very simple technology and only moderately skilled staff to perform the measurements.’ They might include mid-upper-arm circumference measures of children, or Body Mass Index measures in adults calculated using simple height and weight measurements.
b The development of this protocol has been informed by discussions with our virtual advisory group which includes a number of policy and research experts in the field. It has been further strengthened by the peer-review process of publication in this journal.
Thanks to our ‘virtual advisory group’ for their input at key stages in the development of this protocol. Dr Tsakani Ngomane, University of Pretoria, South Africa Dr Michael Aliber, Department of Agriculture, Forestry and Fisheries, South Africa Prof Gordana Kranjac-Berisabljevic, University for Development Studies, Ghana Dr Elaine Tweneboah Lawrence, University of Ghana, Ghana Prof Chris Gordon, University of Ghana, Ghana.
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.