The policy contexts identified in the Frankfurt Workshop and described in the following list are not exhaustive but they are examples of factors that might influence choice of synthesis methodology. We recognise that they are potentially overlapping and interrelated:
Time constraints
The timeframe over which policy decisions need to be made (the policy window) can sometimes be very short (days to weeks). This places limits on the knowledge synthesis that can be achieved or encourages forms of synthesis that can be conducted and updated rapidly.
Financial resource constraints
Alongside time constraints there are always financial constraints and knowledge-synthesis methods may be confined to low cost options.
Controversy caused by conflicts of evidence
Knowledge needs may arise in relation to a disagreement over the interpretation and implications of the current evidence, or its robustness in terms of the variability of existing results. This may require transparent, rigorous, independently conducted (by actors perceived by stakeholders in the conflict to have no vested interest in the outcome) and inclusive synthesis methods that minimise susceptibility to bias by engaging key actors in designing research questions and discussing conclusions on the basis of evidence.
Controversy caused by conflicts of values and/or interests
Knowledge needs may differ according to vested interests (legitimate or otherwise) in the outcome and/or a fundamental difference in values and beliefs on the part of two or more groups. This may require substantial stakeholder engagement to generate an acceptable question (or questions) as well as transparent, rigorous and inclusive synthesis methods that provide reliable evidence regarded as legitimate by the parties involved.
Serious and/or unacceptable consequences of making the wrong decision
Where the consequences of making a wrong decision are regarded as a high risk to a decision maker they may require transparent, rigorous and independently conducted synthesis methods that minimise susceptibility to bias and/or clearly state what the potential biases are, thus providing a clear audit trail to justify the decision.
Diversity of knowledge and information
Where the question demands the synthesis of a high diversity of different types of knowledge and/or many different perspectives need to be included, it may require inter- and transdisciplinary methods and approaches that are able to structure diversity, identify commonalities and differences, or rank alternatives.
High uncertainty
In situations where there is significant uncertainty or variability in knowledge, methods may be required that seek to examine sources of uncertainty and variability in results, and synthesise knowledge taking such uncertainty and/or variability into account. Such methods would provide a best estimate of the truth together with statements of confidence in that estimate.
The identified methodologies are drawn from the natural and social sciences and all have been applied to some extent to support decision making in environmental and other related sectors. Definitions are provided in Table 1 together with explanations of their suitability to explain why they might be chosen for a particular combination of question and policy context.
Table 1 Description of knowledge synthesis methods considered in this paper (in no particular order)
Table 2 presents an example matrix of methodologies that might be suitable for different question types in different policy contexts. It is not exhaustive and represents the initiation of what could be a more extensive effort to provide guidance in this area. We detected that for many combinations of questions and policy constraints there is more than one possible method. The matrix suggests appropriate methods according to the most prominent constraint characterizing a particular decision setting. For some of these, e.g. very restricted time available, this limits the possible choice of method to expert consultation or causal chain analysis for practically all types of knowledge needs (first row). In the case of controversy of evidence, the choice of appropriate method depends much more on the type of knowledge need (third row). In practice, it is likely that several of these characteristics or constraints will apply in any given situation so that further elaboration of the context (e.g. the nature of evidence sought, qualitative, quantitative and/or the estimated amount of evidence available) would be necessary to make a final decision. For example, in the context of controversy or conflicts of evidence, there may also be secondary contexts, such as time and resource constraints, that would shift the balance of choice toward more rapid methods.
Table 2 A decision matrix highlighting evidence synthesis methodologies that may be helpful (based on authors’ consensus opinion) for informing environmental decision making for different combinations of questions and key policy contexts (N/A indicates combinations considered unlikely to arise)
The selection of synthesis methods will have to consider the requirements and constraints of each method highlighted in Table 1 above in relation to the policy context. For example, while systematic review is a very robust knowledge synthesis methodology and can provide highly transparent and reliable results, for many specific knowledge requests there will not be sufficient evidence available to justify a systematic review. Constraints such as insufficient or disputed evidence (e.g. determined by a ‘quick scoping’ of the literature, Defra 2015) might in many cases make it necessary to resort to joint fact finding or double-sided critique while time and resource constraints and short policy window could justify expert consultation or focus groups.