The benefits of systematic mapping to evidence-based environmental management

Systematic maps were first developed and used to synthesise social science research (Bates et al. 2007; Clapton et al. 2009). Systematic mapping has now been adopted in a range of disciplines beyond the social sciences, and is the subject of considerable coordinated research effort within specific topics (e.g. spinal and brain injury research; Bragge et al. 2011). The first SM in environmental management research was published in 2012 (Randall and James 2012) and catalogued literature relating to the impacts of integrated farm management, organic farming and agri-environment schemes on biodiversity in temperate ecosystems. To date, 9 SMs have been published in EEJ (Randall and James 2012; Randall et al. 2015; Haddaway et al. 2014, 2015b; Roe et al. 2014; Bernes et al. 2015; Macura et al. 2015; Neaves et al. 2015). Interest in SM methods is expanding (Fig. 1): 15 SM protocols have been published since 2012 (as of September 2015; see Appendix S1), and a CEE Methods Group concerned with SMs has been recently formed (http://www.environmentalevidence.org/wp-content/uploads/2015/02/The-CEE-Systematic-Mapping-Methods-Group.docx).

The objectives of SMs and SRs are fundamentally similar; to collate and describe all of the available published research evidence on a topic in an objective, repeatable and transparent manner (CEE 2013). These syntheses aim to be comprehensive and should be undertaken according to an a priori peer-reviewed method (a SM/SR protocol). Publication of a protocol that sets out the planned methodology before the review commences has a number of important benefits. Firstly, this essentially ‘registers’ the reviewers’ intent to complete a full review, which reduces chances of duplication of effort and can allow interested parties to contact the review team to provide advice, comments, evidence and critique for the review whilst it can still be considered and integrated into the review. Secondly, publication of protocols involves peer-review by subject and methodology experts, ensuring that the procedures to be used are as reliable as possible and that susceptibility to bias, such as publication bias and selection bias, are minimised. Finally, the publicly available protocol helps to ensure that the actual conduct of the review proceeds according to the protocol and does not deviate from the initial plans.

Generally, SMs are appropriate for broad topics that are often too expansive for an individual SR, and typically answer questions such as: “what evidence exists concerning…?”, “how much research is available regarding…?”, and “what is the current state of knowledge about…?”. Systematic maps do not aim to provide a quantitative or qualitative answer to a question of impacts (i.e. a summary effect estimate) or test a hypothesis, but rather an overview of the evidence base, or more specifically, what research has been undertaken, where and how.

Translating systematic mapping and associated SR methodology from the social sciences and medicine, respectively, to environmental sciences requires some careful consideration of the relevant analogous systems that we deal with. However, these considerations are not as challenging as often perceived. For example, the PICO/PECO (population, intervention/exposure, comparator, outcome) model used in SRs and SMs translates well onto environmental subjects: population refers to the specific system investigated, rather than a human population (e.g. farms in boreo-temperate systems); intervention/exposure refers to either a management practice or some other environmental factor (e.g. soil tillage using mouldboard ploughing); comparator refers to the factor with which an intervention/exposure is compared (e.g. before tillage or untilled control group), although SMs may not always require the presence of a comparator; outcome refers to the variable being measured (e.g. soil carbon concentration). Not all review questions have a PICO/PECO structure, of course, but this is not particular to environmental science. There is a mistaken belief that SR methods are only appropriate for randomised control trials (RCTs) and quantitative, experimental research. In fact, SRs and SMs can be used to synthesise any form of research evidence, including observation and qualitative studies (Haddaway and Bilotta 2015).

Whilst key SR outputs include quantitative or qualitative summary effect estimates for the interventions or exposures of interest, the main output from a SM is the description of the evidence base. This description should take the form of the SM report and a searchable database of relevant studies. The report may include a discussion of the following key aspects of the evidence base: (i) general patterns in study methods and settings, (ii) knowledge gluts, where substantial numbers of studies have investigated a similar subtopic, (iii) knowledge gaps, where there is a significant lack of research on a subtopic, and (iv) deficiencies or best practices in research methodology (although this latter point is typically the result of critical appraisal, which is an optional stage in SMs). The SM database includes a range of descriptive information for all included studies, including: (i) citation information, (ii) study setting descriptors, (iii) methodology details, and (iv) summaries of the nature and location of quantitative or qualitative study findings (but, importantly, not the findings themselves). The SM database is intended to be a readily usable resource for researchers and decision-makers when looking for evidence, and usability is a main consideration when compiling the SM and building the database.

In addition to the report and map database, reviewers may also produce other outputs from the systematic mapping process. In particular, geographical information systems (GISs) allow for information from within the SM database to be displayed across a cartographic map. This approach can be particularly useful for environmental topics that are global or wide scale in nature.

Systematic maps have a variety of uses across research, policy and practice. Researchers, research funders and decision-makers can benefit from learning about knowledge gaps, subtopics that are underrepresented in the evidence base. Such gaps may warrant novel primary research in order to provide a future evidence base, particularly where prevailing policy or practice is controversial or perceived to require change. Researchers, research funders and decision-makers can also benefit from highlighted knowledge gluts that allow full synthesis in the form of SR. Furthermore, areas that have received substantial research effort may be deemed by research funders to be sufficiently well understood that research funding could be more effectively directed elsewhere. Researchers and research funders can also benefit from the identification of deficiencies and best practices across the evidence base, which may be used to increase consistency across studies. Environmental managers may also benefit from using the SM database as a library, from which they can identify a subset of studies that are most relevant to their situation. Finally, since SMs published with CEE are Gold Open Access (i.e. freely, immediately accessible in full to everyone), researchers, practitioners and policy-makers can use the database as a source of detailed descriptive information regarding studies that may not be individually accessible in full text. Practitioners and policy-makers often cite limited accessibility as a barrier to evidence use (e.g. Haynes and Haines 1998; Oliver et al. 2014), and the provision of accessible, reliable summaries may prove particularly important for environmental decision-makers.

Besides the CEE journal (EEJ), other options exist for publication of SMs. However, since SRs and SMs are new methods, publishing them without using the expertise of a coordinating body (such as the CEE and its associated journal EEJ) can have serious disadvantages. Firstly, no other formal academic publication as yet accepts SM or SR protocols. Secondly, other journals may not have the methodological expertise necessary to identify flaws or missing information within review reports. This is evident in a recent example, a SM on the subject of European agroforestry ecosystem services that was published in the journal Ecological Indicators (Fagerholm et al. 2016). Whilst this SM purports to be undertaken according to CEE guidelines, it lacks a peer-reviewed protocol. In this case, the authors have chosen not to include grey literature, and their search strategy involves the use of country names within the search string. The former practice opens the review to publication bias [the inclusion of only academic research is well-proven to be more likely to show positive and significant findings (Rothstein et al. 2006)]. The latter practice compromises comprehensiveness, since there is a significant risk of missing research that does not mention the study country within its title, abstract or keywords. It is possible to self-publish a SM protocol, however, particularly where independent peer-review by subject and methodology experts can be transparently demonstrated. Some generalist journals are also increasingly able to peer-review and publish SRs and SMs, such as PLoS One, which requires review authors to submit a PRISMA checklist (a set list of descriptors that confirm the review has undertaken specific required aspects of formal review methodology). Such checklists are not a definitive indication of reliability on their own, however (O’Leary et al. 2015).

Review questions initiated by external commissioners (e.g. Defra; Randall et al. 2015) are often very broad in nature and may initially be suitable for review by SM. Furthermore, since SMs can highlight knowledge gaps and knowledge gluts, they may be a useful first step before full SR where the volume of evidence on a topic is poorly understood. Completing a full SR following on from a SM may then be a much more rapid process than for a totally novel topic, since many of the SR processes have already been undertaken. Furthermore, SMs may provide a resource from which a number of full SR topics can be identified. Although a useful first step, SMs are not necessarily any less resource intensive than SRs. Despite not involving full data extraction, critical appraisal or quantitative/qualitative synthesis, SMs typically cover broader evidence bases and may assimilate a larger number of studies.

SMs also prove useful additions to full SR. Tables of relevant studies and their descriptive meta-data are likely to be produced by the vast majority of reviewers as part of the SR process. Providing these resources as supplementary, interactive, searchable databases would be a valuable output for all reviews, requiring minimal additional effort. Furthermore, reviewers may choose to produce a map database that is broader in scope than the subset of studies taken on to full synthesis, increasing the relevance to end-users.

In summary, SMs have a plethora of uses to researchers, research funders, policy-makers and practitioners alike: from acting as a library for finding single relevant studies to providing recommendations of knowledge gaps that may warrant further research.

Three new SMs have recently been completed on the subjects of agricultural impacts on soil organic carbon (SOC) (Haddaway et al. 2015b), management of protected forests (Bernes et al. 2015) and on-farm mitigation measures for improving water quality (Randall et al. 2015). These maps have used state-of-the-art methodology in systematic mapping, including in-depth stakeholder engagement from the outset of the review projects, comprehensive assessment of all relevant review bibliographies, the use of GIS to visually display the contents of the map databases. These projects demonstrate the utility of SMs.