FormalPara Key Points

Many reviews about built environments and physical activity have methodological limitations.

The label ‘systematic review’ has mostly been used inappropriately.

The PRISMA statement is useful to standardize the reporting of methodology and limit bias in systematic reviews of effectiveness.

1 Introduction

Over the last decade there has been a marked increase in publications about built environments and physical activity [1]. There has also been a substantial policy response with governmental and non-governmental agencies recommending environmental strategies to counter the epidemics of low physical activity, sedentariness and obesity while addressing common cross-sectoral goals including traffic congestion and safety, air pollution and climate change [2–5]. As the number of publications in this field is growing rapidly, it is becoming increasingly challenging for researchers and policy makers to keep track of the evidence base. Literature reviews play an important role in identifying primary studies and in summarizing and synthesizing their findings. Moreover, the rapidly growing evidence base requires that reviews be regularly updated [6].

Just as the quality of primary studies can vary, the methodological quality of review articles can vary substantially and affect their conclusions [7, 8]. A 2007 article critically appraised methodological aspects of literature reviews about built environments and physical activity. The main findings were that reviews inappropriately claimed to be systematic, did not provide important methodological information, omitted large numbers of relevant studies, and reported some study results incorrectly [9]. Since then, hundreds of primary studies and dozens of literature reviews about environments and physical activity have been published [10–16]. Many of the new literature reviews continue to demonstrate methodological limitations that might lead to an inaccurate summary of the evidence. This commentary re-emphasizes the importance of methodological rigor in literature reviews of effectiveness by (i) describing common problems in recent literature reviews, and (ii) providing guidance for future reviews on environments and physical activity.

2 Strength of Evidence

It is increasingly accepted that reviews of the effectiveness of public health interventions should start with a categorization of the evidence, ranking studies based on the strength of the evidence, starting with the strongest research designs and studies that minimized selection, measurement and confounding bias [17]. So far, only a few studies on the health effects of built environments managed to randomly assign individuals [18–20] or settings [21]. As randomization is virtually impossible to achieve in this research field, there have been calls for more opportunistic evaluations of environmental interventions [22, 23]. In a framework for evidence-based public health regarding built environments and physical activity, controlled prospective evaluations of environmental interventions and evaluations of people’s activity level before and after they relocate between neighborhoods of different urban forms provide the highest level of evidence available. Only prospective evaluations of environmental interventions and relocation studies in which the same individuals are exposed to different environments can establish the temporal sequence of cause and effect between exposure to environments and subsequent physical activity. Cross-sectional studies that adjust for residential preferences represent the next level of evidence, followed by usual cross-sectional studies. Editorials, opinion pieces and similar articles on the potential association between the environment and physical activity provide the lowest level of evidence [24].

There are various methodological challenges in conducting prospective evaluations of environmental interventions and relocation studies, and some of the evaluations of changes to the environment have had methodological limitations [25, 26]. Despite these challenges and limitations, in recent years some studies that examined the effects of changes to the environment have been published [27]. One recent literature review specifically focused on causality in the relationship between the built environment and physical activity by only including quasi-experimental studies and cross-sectional studies that controlled for residential preferences [27]. Surprisingly, some literature reviews a priori excluded intervention studies and focused solely on cross-sectional studies, mentioning in the discussion that environmental interventions are needed to provide causal evidence [13, 28–31]. As prospective evaluations of environmental interventions and relocations provide the most robust information on effectiveness, it is essential to include them in reviews that aim to synthesize the literature. The findings of such intervention or longitudinal studies should be summarized separately from cross-sectional studies [24, 32].

3 Criteria for Systematic Reviews

Systematic reviews are considered to represent the highest level of evidence of effectiveness. While not every review of effectiveness needs to be a systematic review, those reviews that are used to inform policies and guidelines on physical activity and public health should have high methodological standards to minimize sources of bias. Moreover, when claims are made by authors that a review is ‘systematic’, then it should be expected that it meets current widely accepted systematic review standards. In contrast to traditional literature reviews, systematic reviews are more rigorous and need to meet specific quality criteria to minimize sources of bias [33]. The PRISMA (Preferred Reporting Items for Systematic reviews and Meta-Analyses) statement, which replaces the previous QUORUM statement [34], provides a checklist with 27 items to include when reporting systematic reviews [35] and is required for this kind of research by more than 170 medical and public health journals [36]. While the PRISMA statement was primarily designed for the synthesis of randomized trials, it can also be adopted for other types of studies, including those that examine the relationship between environments and physical activity [37]. However, with few exceptions [12, 38], literature reviews about environments and physical activity did not refer to or follow the QUORUM or PRISMA statements. Protocols and final results of systematic reviews with health-related outcomes should be registered with PROSPERO (established in 2011), the international database for systematic reviews in this field of research (http://www.crd.york.ac.uk/PROSPERO/) [39]. To avoid the risk of wasteful duplication of efforts and resources, those who are planning to conduct a review can ascertain from the database if other similar reviews are already underway. Prospective registration of reviews also improves their methodological quality and reporting of final results.

To reduce the risk of bias, one of the major criteria in guidelines for systematic reviews is that the quality of all included primary studies needs to be assessed, and this should be reflected in the synthesis, presentation and interpretation of results [35, 40, 41]. An assessment of study quality allows the reader to examine results stratified by their quality and their effects. Though many literature reviews about built environments and physical activity have used the label ‘systematic review’ [11–15, 27, 29–31, 42–46], few assessed the quality of the primary studies [12, 15, 38, 43]. Tools for the assessment of the quality and validity of intervention studies have been suggested [41]. A quality checklist applicable to cross-sectional studies could be adapted from previous reviews of observational studies [47–49].

4 Publication Bias

The Prague Definition states that “grey literature stands for manifold document types produced on all levels of government, academics, business and industry in print and electronic formats that are protected by intellectual property rights, of sufficient quality to be collected and preserved by library holdings or institutional repositories, but not controlled by commercial publishers i.e., where publishing is not the primary activity of the producing body” [50]. Grey literature includes sources such as reports, repositories, clinical trial data, conference proceedings, and doctoral theses. Studies in peer-reviewed journals are more likely to report significant results than those in the grey literature [51–54]. Excluding the grey literature in a review could introduce publication bias and be a concern to the validity of the findings of the review. It is reasonable to have concerns about the quality of grey literature, but this could be managed by rating the methodological quality of all studies in the review. Therefore, another criterion in guidelines for systematic reviews is that authors of literature reviews should search for grey literature [33, 41, 55]. However, with few exceptions [14, 15, 38, 56, 57], reviews on environments and physical activity only included articles from peer-reviewed journals. Sources of grey literature could be found through databases such as PsycEXTRA (http://www.apa.org/psycextra), Open Grey (http://www.opengrey.eu) and NTIS (http://www.ntis.gov) and internet searches targeting academic, government, and non-government organizations’ publication or report lists.

5 Conceptual Match of Environments and Physical Activity Domains

Physical activity occurs in multiple domains including occupation, household, transportation, and leisure time [58]. As ecological models particularly emphasize domain-specific and context-specific environmental influences [59], Giles-Corti and colleagues recommended that environmental attributes and domain-specific physical activity should be conceptually matched [60]. However, the current understanding of ‘conceptual match’ is limited by the not yet well developed models of environmental influences. For example, land-use mix, a construct that was previously hypothesized to be a correlate of transport walking, was later found to be associated with leisure walking as well [61]. Literature reviews provide unique opportunities to explore the conceptual match between environmental attributes and physical activity to improve current ecological models. Specifically, reviews can stratify based on domains of physical activity and/or compare the consistency of associations between conceptually matched pairs with conceptually unmatched pairs. Compared with applying a correlation matrix and summarizing all associations at once [13], which would dilute the association between the environment and physical activity, stratifying studies by environment and physical activity conceptual match has the potential to improve ecological models and guide more specific and better designed studies.

6 Stratification by Measurement Method and Population Subgroups

One often-cited suggestion for future research in this area is to include both objective and self-reported measures of the built environment [10]. Both modes of measurement are important because studies have identified discrepancies between self-reported and objectively measured environmental attributes and their differential associations with physical activity outcomes [62–64]. Similarly, differences in associations were found depending on whether physical activity was measured objectively or by self-report [65]. Therefore, it is important for literature reviews to stratify the synthesis by environmental and physical activity measurement mode to ascertain which combination of measurements gives the most consistent associations.

Ecological models postulate that environmental attributes may interact with sociodemographic characteristics in influencing one’s behavior, which may lead to differential associations between environments and physical activity among different population subgroups [61, 65]. Whenever possible, literature reviews should stratify the synthesis by population subgroups to improve the specificity of findings [66] and to test and improve theoretical models. Such review processes provide opportunities for creating a clearer picture of the evidence base which could lead to more specific scientific conclusions and policy recommendations.

7 Conclusion

Literature reviews are key sources of information for researchers, practitioners, policy makers, and other stakeholders. However, methodological limitations of the reviews can lead to inaccurate portrayals of the evidence. The label ‘systematic review’ has mostly been used inappropriately in the built environment and physical activity literature. This finding indicates common misperceptions among authors, peer reviewers and members of editorial boards about the criteria for true systematic reviews that might be due to a lack of training in review methodology. Moreover, if non-systematic literature reviews that inappropriately use the label ‘systematic review’ get published in peer-reviewed journals, then this reinforces misperceptions of the methodological rigor that is required. The PRISMA statement is a useful tool to standardize the methodology of systematic reviews of effectiveness and limit sources of bias in them. Applying this tool and the further recommendations from this article can help in moving forward the synthesis of evidence of effectiveness not only in the area of built environments and physical activity, but also more broadly in public health.