What is a Systematic Review?

Policy and practice decisions should always be based on the best available evidence.Footnote 1 A lack of high quality evidence in decision making can result in detrimental changes or a lack of necessary change. However, attempting to collate all of the evidence on a topic and obtain a reliable conclusion can sometimes seem an insurmountable task. Systematic reviews (SRs) are an important tool for obtaining and appraising evidence in a reliable, transparent and objective way. They are useful because they encourage the development of focused, well-defined questions within a topic of interest. As a secondary research technique, they can be used to address questions relating to the efficacy of an intervention or the impact of an exposure. They can also address concerns about a lack of consensus or doubt about the reliability of a body of evidence and can identify factors that cause heterogeneity in results amongst studies. Finally, SRs are increasingly used to synthesise qualitative data (see [11]).

SRs should be carried out according to established standards that ensure they minimise bias and include all available information. Various organisations set and govern the standards of SRs in their respective subject areas. The Cochrane Collaboration (www.cochrane.org/) works in health sector, the Campbell Collaboration (www.campbellcollaboration.org/) in the social sector, 3ie (www.3ie.org) in international development and the Collaboration for Environmental Evidence (CEE) (www.environmentalevidence.org) in the environmental sector. SR libraries are typically open access, improving the accessibility of the results of large bodies of evidence to managers and policy makers.

The need for a SR is typically identified and developed by a Commissioner who requests the review and may fund the resources needed, but may also be curiosity driven and used to plan future research priorities. The SR itself is undertaken by a Review Team who undertakes the practical aspects of the review. The final key group in any SR is the group of stakeholders that may be affected by the review’s results.

SRs comprise a number of distinct stages, with each being specifically adapted to ensure that all relevant available evidence is synthesised into reliable conclusions. These stages are formulation of an appropriate question, writing of the protocol, searching for studies, screening for relevance, critical appraisal, data extraction and synthesis in a final review document. Each of these stages must be conducted in a systematic and transparent manner. In this way, both published and grey literature is identified and all relevant information is included. Systematic critical appraisal ensures that all individual studies are objectively assessed for internal validity (reliability) and external validity (generalisability). Since SR methods themselves can differ, for example, with external validity sometimes not conducted for qualitative evidence as for quantitative, the synthesis methods to be used in a SR should be stated a priori.

How SR Differs from Other Review Techniques

SRs differ from other review techniques in a number of key ways. Other techniques include traditional reviews (i.e. reviews that do not conform to a defined standard), meta-analyses and synopses. Traditional reviews are a useful method of collating relevant references of published research, but they often fail to include potentially influential unpublished evidence, such as organisational reports or unpublished theses. Furthermore, the methodology for identifying the relevant research is rarely described in much detail. Finally, traditional reviews rarely perform an objective assessment of individual study validity. As a result, the susceptibility of these reviews to biases such as publication bias or prevailing dogma may be high. Meta-analysis is a powerful tool for combining statistically the results of individual studies, increasing the sample size and the precision of the individual studies. However, meta-analyses are susceptible to the same issues as traditional reviews, and their results are only as reliable as the data included. Finally, synopses are compilations of evidence on a specific management intervention or group of interventions. In the area of conservation science, these are published in the journal Conservation Evidence (http://www.conservationevidence.org). Since synopses focus on management interventions, they are valuable resources for decision makers. However, they do not aim to appraise the validity of the included studies and risk vote counting, simply noting the number of studies either supporting or refuting the benefit of an action and ignoring the magnitude of the effect or the contributing factors that may alter its effect. SRs contain aspects of all of these review techniques, but their strengths lie in the inclusion of all available evidence, transparent documentation of the review methodology involved and objective assessment of relevance and validity of all included studies.

Brief History of SR in Medicine and the Social Sciences

The Cochrane Collaboration was the first formal body to set guidelines for the conduct of SRs. Established in 1993, it is a partnership of almost 30,000 experts across more than 100 countries [1] and was established in response to Archie Cochrane’s call for more SRs for assessing the results of medical randomised control trials. SRs and the meta-analyses they enabled provided invaluable syntheses of studies that individually did not identify significant patterns but that were able to when viewed together. One notable example is the use of streptokinase in the treatment of myocardial infarction (heart attack), which was only widely recommended for prescription in the 1990s after four decades of randomised control trials. If it had been the subject of a meta-analysis through SR, however, a significant positive impact would have been identified in the early 1970s, potentially saving thousands of lives [7]. Similarly, the Campbell Collaboration was subsequently established in 1999 to facilitate the production of SRs relating to social interventions (education, crime and justice, social welfare and international development).

History of SR in Environmental Management (EM)

The Centre for Evidence-Based Conservation was established by Andrew Pullin and Teri Knight in 2003 in order to support decision making in conservation, and thereafter expanded its remit to environmental management. CEBC promotes and disseminates SRs in order to provide decision makers with sound evidence on the effectiveness of management and policy interventions and on the impact of human activities on the natural environment. CEBC undertakes SRs, acts as a source of advice and training in SR methodology and is a coordinating centre for CEE.

CEE was formed as a charitable organisation and a community of managers and researchers. The Collaboration is currently formed of five Centres in the UK, Sweden, South Africa, Canada and Australia. CEE provides and monitors adherence to the Guidelines for SR in Environmental Management through registration, peer-review and publication of SR protocols and completed reviews in the EEJ and the CEE Library. The CEE Library currently holds over 60 completed SRs and over 20 protocols for reviews currently underway. The number of active CEE SRs is expanding as knowledge of their benefits and applications is spreading. Two large research groups in Europe are in the planning stages of an additional 30 SRs on a wide variety of topics. Since 2012, newly published protocols and SRs are hosted in the open-access journal Environmental Evidence.

The breadth of topics covered by CEE SRs is also rapidly expanding to include more divergent themes, for example, socio-ecological interventions such as community-based resource management [3] and arsenic contamination of groundwater [8]. Just as primary research in environmental management is becoming more interdisciplinary, so too are SRs.

To date, governing bodies for SR exist only in environmental management and the social, medical and health sciences. As the number of publications and journals increases, and as the open access movement increases access to novel research, there may be a need to establish organisations to oversee SRs in other subjects.

Criticisms of SR

Whilst SR methods have strong support across disciplines and the science-policy/practice interface, it should be noted that there have been criticisms of SRs (e.g. [5, 6, 12]). One such criticism is that SRs favour certain forms of evidence over others [6]. SR advocates argue that any evidence is admissible, providing that it is documented and its reliability can be assessed. Whilst it is true that early SR guidance did favour some ‘higher quality’ study types over others (i.e. randomised control trials, RCTs), the adaptation of SR methods into other disciplines has necessitated a relaxation of this requirement that has also been adopted in the health sciences in order to make better use of the available evidence.

Why Undertake a SR as an Early Career Researcher?

SR is a rapidly expanding research technique and is increasingly being recognised as a standard tool in informing decision-making and policy-making processes. SR can form a useful overview of an applied and/or policy-relevant topic, from question formulation to management/policy-relevant conclusions. Undertaking a SR is a valuable experience for early career researchers, particularly those interested in evidence-informed practice, management or policy, providing many transferable skills, for example in project management, literature searching, database management, critical appraisal and report writing. Working as part of a team that may be interdisciplinary and producing a number of publications (the protocol, the completed review, associated traditional publications, policy briefs and press releases) are other benefits that are likely to result from a SR. A SR is a very serious undertaking, demanding substantial resources to ensure an appropriate execution. However, SRs have been completed successfully and effectively by competent postgraduate students and are frequently undertaken by researchers within the timeframe of a postdoctoral contract [2]. Postgraduate supervisory teams often make good review teams, although the addition of one or more experts in SR methodology is extremely useful.

How SRs Can Inform Policy and Practice

SRs have become an industry standard in evidence assessment. Many SRs are conducted in response to specific evidence requirements of organisations needing to make well-informed, evidence-based decisions. Groups or individuals who request the conduct of a SR are known as ‘commissioners’. Commissioners may also fund SRs. It is worth taking time to consider the cost of SRs here. As experts in SR, we are often asked what the resource requirements are for a SR in terms of time and money. While every review is different, a typical SR may take between 9 and 12 months to complete and could involve one full-time researcher throughout, depending on the breadth of the topic (see the literature on rapid evidence assessments/rapid reviews for comparative methods). This may seem like a large investment, but it is vital to put this in the context of policy and management practices. For example, if a SR enables the identification of a widely used management practice that is unsupported by a reliable assessment of the evidence, then the sums of money saved across organisations or nations could be several orders of magnitude greater than the cost of the SR. Furthermore, SRs are particularly attractive to commissioners when evidence from different sources conflicts and can provide transparent and objective assessments where subjects are controversial or high profile.

SRs carried out according to any of the various guidelines produced by a SR organisation, such as CEE, should engage stakeholders in the review process from a very early stage. Many research groups undertaking SRs employ formalised stakeholder engagement to ensure their outputs are as relevant as possible to policy and practice.

The outputs of a SR are disseminated through a variety of different media. As stated above, protocols and completed reviews are published in open access formats on the internet. Transparency, objectivity and reliability are key strengths of SR, but these can often lead to very detailed, lengthy documents, and often this information needs to be summarised further. Funding bodies often request summary reports, and traditional research publications, policy briefs and press releases are also common. More recently, social media has been used to disseminate the results of SRs to decision makers in a rapid, concise and effective manner.

How SR Can Inform Policy

Policy makers rarely have the resources necessary to undertake an appropriate assessment of the evidence relating to a specific concern described above. SRs perform this job in an objective way and when summarised in accessible policy-briefs, written in lay terms, they can become a key source of information for policy makers and other decision makers alike. A recent SR assessing the impact of fully and partially protected marine reserves on commercially exploited and non-target fish species identified strong support in the evidence base for marine reserves in terms of the biomass and number of fish present, a pattern that was not universally clear in many studies [9].

Another SR published in 2010 examined the efficacy of ‘greening’ of urban areas in reducing human exposure to ground-level ozone and ultraviolet light, and minimising the ‘urban heat island effect’ [4]. The results of this SR directly informed the policy-making process behind the Heatwave Plan for England (https://www.gov.uk/government/publications/heatwave-plan-for-england-protecting-health-and-reducing-harm-from-extreme-heat-and-heatwaves).

It should also be noted that there have been recent calls in the social sciences to ensure that SRs be made more policy relevant (e.g. [10]) to increase their potential impact on policy. Whilst the body of environmental management SRs is still relatively small, policy relevance should be a major concern to all SR review groups. Suggestions of ways that this may be achieved include a move from effectiveness-style reviews to those with mixed methods; the use of programme theories; novel means of synthesising diverse forms of data.

The Future of SR

SRs are continuing to become standard practice in collating and synthesising evidence to aid decision making, and SR will almost certainly continue to be used to this effect in the future. In the short term, there is a clear current demand for SRs to tackle a wider range of topics and concerns, including multi-arm reviews (e.g. both qualitative and quantitative syntheses) and configurative (model forming) as well as aggregative (question answering) reviews.

As more SRs are published, SR guidance will develop both to better support reviewers and continue to improve standards. SR methodology is a newly emerging discipline that aims to critically assess the efficiency of SR guidance in order to provide best practice guidance in a way that is more efficient with respect to time and resources. This work involves making the most of recent advances in computing and empirically testing the impact of certain aspects of SR methodology on reliability. As SR efficiency increases, their usability will likewise increase, and SR methods will be more applicable for reviews with necessarily shorter timescales. As SR methods have been adapted from medicine to the social and environmental sciences, so too will methods be adapted to other disciplines.

Other Roles of SRs

Systematic maps catalogue the location and nature of relevant literature on a topic of interest. They follow the SR process but often focus on broader questions and typically stop at the full text assessment stage, although they may continue with critical appraisal and synthesis of a subset of the literature in answer to a specific secondary review question. The key outputs of systematic maps are searchable databases of literature on a topic along with information regarding the population, intervention/exposure, comparator, measured outcomes and study design and methodology (collectively referred to as coding). In addition, systematic maps highlight subject areas where little evidence exists, also known as knowledge gaps, or where full SR may be particularly fruitful because of a critical mass of data. Identification of these gaps can directly feed back into proposals for primary research by scientists and into calls for grant applications by funders.

SRs may also highlight issues with the evidence base as it stands, which restrict a synthesis across studies or affect the quality of individual research results. For example, a generally low standard of study quality or poor level of methodological detail may be identified across studies, identifying a need to standardise approaches in primary studies to improve the validity and generalizability of future research.