Introduction

Invasive alien species (IAS) can severely impact biodiversity, ecosystem services and human livelihood and health. As a consequence, the Convention on Biological Diversity states that by 2020, IAS and invasion pathways should be identified, prioritized and measures to manage priority species (i.e. control or eradication) and pathways should be put in place (Aichi Target 9; https://www.cbd.int/sp/targets). Legal instruments have been established to meet this target, such as the recent European Union IAS Regulation 1143/2014 (European Union 2014; Genovesi et al. 2014; Tollington et al. 2015), that targets IAS through trade restrictions, border controls, targeted surveillance, rapid response, management and restoration obligations. These measures rely heavily on listing of high impact IAS underpinned by comprehensive risk assessments. The quality of risk assessments is thus a pivotal element to promote action on a sound basis given the potential legal and financial implications (e.g. trade limitations and management obligations) for the parties that have to adopt these policies. More generally, good quality risk assessments are an essential part of conservation efforts, since their outcome may lead to costly and sometimes controversial eradication or control actions. Transparent and sound risk assessments are needed to ensure efficient allocation of usually scarce conservation resources (Bottrill et al. 2008).

The Food and Agriculture Organization (FAO, for plant health), the World Organisation for Animal Health (OiE, for animal health) and the World Health Organisation (WHO, for human health) have issued common standards for risk analysis frameworks. Risk analysis consists of risk assessment, risk management and risk communication (European Food Safety Authority 2012; IPPC 1997; Maijala 2006; OiE 2015). Risk assessments (the evaluation of entry, exposure and consequence) are the cornerstone of risk analyses. They can ensure transparency and traceability during the process of listing pests, weeds, diseases and their pathways of introduction since they follow a formalised procedure including documentation requirements and quality assurance (Roy et al. 2014a). Risk assessments performed by experts can be based on quantitative statistical models, semi-quantitative scoring or qualitative assessment. To do this, a wealth of qualitative and semi-quantitative protocols have emerged in recent years (e.g. Baker et al. 2008; Blackburn et al. 2014; Bomford 2008; D’hondt et al. 2015; Essl et al. 2011; EFSA Panel on Plant Health 2011; Kenis et al. 2012; Nentwig et al. 2010; Sandvik et al. 2013). These schemes range from classification systems to more elaborate scoring systems and are widely used in IAS management. Several studies have reviewed and compared these tools. Protocols consider different subsets of the major invasion stages (transport, introduction, establishment, spread), impact categories (environmental, health, socio-economic) and domains of impact (human, animal, plant) (Heikkilä 2011; Kumschick and Richardson 2013; Leung et al. 2012; Verbrugge et al. 2010; Roy et al. 2014a; McGeoch et al. 2016). They also differ in the possibilities for weighting different components of impact and the way they cover uncertainty (Heikkilä 2011). In a comprehensive review of risk assessment schemes, McGeoch et al. (2016) stated that there is currently no broadly adopted, standard approach to prioritizing invasions. The method and approach that is most relevant for any given risk assessment will differ depending on the objective and scope.

All current risk analysis tools and practices are developed to enable proper use of available data to inform policy. As such, they are valuable tools in addressing the risk of IAS and should be acknowledged on those merits. There will probably never be a one-size-fits-all solution to risk assessment for alien species. However, here, we illustrate several drawbacks of the current expert-based risk analysis practice that represent a hindrance to effective prioritization and smart resource allocation. Acknowledging the difficulties of resolving all of those, we provide two suggestions for improvement.

Limitations of current risk analysis practice for IAS

Protocol specificity and scope

Species-based risk assessment protocols were usually developed for specific environments (e.g. Molnar et al. (2008) for the marine environment) or taxa (e.g. Copp (2013) for fish). Nonetheless, several protocols claim to be generic across regions or taxonomic groups (reviewed by Leung et al. 2012), while others are developed for very specific purposes. For instance, some less elaborate protocols have been developed for quick screening only, such as the EPPO prioritisation tool for invasive plants (Brunel et al. 2010) and the Invasive Species Environmental Impact Assessment (ISEIA; Vanderhoeven et al. 2015). These tools lack essential components of risk analysis (e.g. transport, introduction) and represent impact-based prioritization tools rather than full risk assessments, taking into account all components involved in the invasion process (transport, introduction, establishment, spread). They are well suited to weed out species of low risk and pre-select species for detailed risk assessment (e.g. horizon scanning (Gallardo et al. 2016; Roy et al. 2014b, 2015) but cannot be applied for pre-border screening. However, because they are easy to apply and require relatively little information and time to perform, they are often used as a substitute for full risk assessments although they are not fit for that purpose (e.g. Gyimesi and Lensink 2010; Schiphouwer et al. 2014; Van de Koppel et al. 2012). In addition, despite the common standards for risk analysis issued by FAO, OiE and WHO (European Food Safety Authority 2012; Maijala 2006), some protocols include elements of both risk assessment and risk management rather than separating them (e.g. Baker et al. 2008). This raises the question whether comparing results from different protocols is relevant or at all feasible (Turbé et al. 2017).

Information demands

At the very heart of species-based risk assessment is the question: “What information do we need to accurately classify an alien species as invasive?”. This requires baseline data on priority species, their pathways of introduction and susceptible and sensitive sites (McGeoch et al. 2016). High resolution distribution data as well as survey data on establishment and spread are often lacking or inadequate which can lead to suboptimal conservation investment. The lack of information on species impacts on native biota and ecosystems represents a particularly problematic source of error in species listing (McGeoch et al. 2012). As an illustration, some high profile IAS in Europe, which are well known invaders in many countries and cause major socio-economic impacts, suffer from a notorious lack of published evidence on their ecological impact (Hulme et al. 2013). For example, this is true for invasive bird species like the greater Canada goose, Branta canadensis or the Egyptian goose, Alopochen aegyptiacus (Strubbe et al. 2011). Also, for several aliens in Europe, the risks to human health are better understood than their ecological impacts (Hulme 2014). The Siberian ground squirrel, Tamias sibericus, for example, has been shown a competent reservoir of lyme borrelliosis (Marsot et al. 2011, 2013), but the potential ecological impact of the species is insufficiently known and there is a lack of evidence on its invasiveness and impact. For the sacred ibis, Threskiornis aethiopicus, the evidence for impact is ambiguous as a result of conflicting evidence (Clergeau and Yésou 2006; Marion 2013; Yésou and Clergeau 2005). These examples illustrate how a lack of evidence can lead to conflicting views regarding impact. Drafting and conducting detailed risk assessments is a resource-demanding activity and because of limited research resources, these cannot be allocated to more fundamental ecological studies on IAS impacts. If species listing is not based on adequate data on occurrence, establishment, spread capacity and impact, this represents a risk of losing policy and public support, as well as discrediting the risk assessment protocols used.

Addressing manageability

Furthermore, although some protocols claim to prioritize species for management (Roy et al. 2014a), most of them do not adequately cover potential management options. Therefore, they cannot help choose the best management strategy and may lead to a bad return on investment. Such choices should explicitly include the socio-economic context and multiple stakeholder trade-offs. Typically, the choice between non-action, containment, eradication or long term control, is a function of the severity and persistence of any impact, the area of occupancy of an invader, the probability of reinvasion and, more pragmatically, the type of habitat and the availability of effective management methods and resources (Adriaens et al. 2015; Pluess et al. 2012). Some recently emerging techniques, such as eradication probability modelling which involves inferring a quantitative probability of success of a planned project and allows weighting and comparing management options (Drolet et al. 2014, 2015), offer interesting tools to guide decisions on IAS management, although a lack of published data still impedes their widespread use.

Addressing uncertainty and variability

Addressing uncertainty is an important requirement to ensure scientific rigour of assessments. Uncertainty in IAS listing processes can be associated with lack of information, conflicting evidence, context dependence or unclear formulations (Leung et al. 2012; McGeoch et al. 2012). Currently, protocols address uncertainty by asking assessors to provide a confidence score for each of their answers. An overall confidence score is derived by calculating a mean of confidence scores across questions and/or modules, which provides a general level of confidence for an IAS risk assessment. Some protocols require the assessor to reiterate their uncertainty evaluation in the overall risk appraisal, without relying on a quantitative approach. Although quantifying uncertainty indicates whether the outcome is sufficiently robust, it does not allow to identify the type of uncertainty. The reasons for low levels of confidence in the output of risk assessments remain mostly unexplained, and this can be critical to judge the content quality of risk assessments. For example, several protocols do provide explicit guidance and/or values (e.g. low/medium/high levels of confidence) for scoring uncertainty on the assessor’s answers (Baker et al. 2008; Blackburn et al. 2014; D’hondt et al. 2015). However, they do not require the assessor to justify their confidence scores. Moreover, when a species is risk assessed by more than one assessor, differences in individual assessments may represent important information, which are not taken into account when calculating the risk scores.

Disregarding uncertainty may lead to suboptimal decisions. Risk assessment schemes should consider uncertainty in input information and output ranking to explicitly integrate uncertainty in decision making (Heikkilä 2011). In practice, relative risk scores of species lead to high-, medium- or low-risk categorisation, which then promptly results in listing high or medium risk species, regardless of the confidence on the risk scores. Yet, many species suffer from lack of evidence, conflicting evidence or context dependent variability which complicates a decision on actions to take. This does not justify ignoring uncertainty which is intrinsically associated to risk.

Reviewing risk assessments

Although risk assessments are sometimes subject to a period of public consultation, the reviewing process is often not transparent and the rules and procedures—type of peer review, number of reviewers, feedback mechanisms—are not sufficiently set out by protocol developers/users. Typically, risk assessments are performed by one assessor and review, if any, is performed by peers. Yet, the authorship of risk assessments is sometimes unclear and the same is true for the selection of potential peers for review. Without proper review, assessments may solely represent the opinion of a single expert instead of building on all available evidence. As new information is obtained or after a number of years, risk assessments need to be updated with the same requirements in terms of transparency and clear rules and procedures.

Recommendations for improved risk assessment practice

All alien species risk assessment protocols have their advocates, but regardless of the type of protocol selected, the use and the way the scores are handled can be improved. We provide a number of recommendations to support an expert-based risk analysis workflow to underpin IAS policies (Table 1).

Table 1 Recommendations for expert-based risk analysis workflow underpinning invasive alien species policy
Full size table

Quality control is key in any species or pathway based risk assessment procedure. For example, the legislation underpinning the EU black list mandates quality control as a minimum standard for risk assessment and an independent scientific forum to check this requirement (Tollington et al. 2015). Two approaches could potentially improve the way risk assessments are commonly conducted for IAS. Each approach results in an overall risk score and related confidence score, includes mechanisms for quality control, and capitalizes on available data and expertise to maximize the evidential basis in decision making. Here, we primarily consider species-based prioritization processes. Yet, the suggestions to improve transparency and quality also apply to pathway-level assessments (e.g. Brunel 2009; Madsen et al. 2014; NOBANIS 2015; Pfeiffenschneider 2016).

A first approach is to subject risk assessments performed by a single assessor or group of assessors to a peer review process (Fig. 1a). Here, peer review should be performed independently of the assessor(s). Rules and procedures should be clearly set out, including the selection of relevant reviewers (e.g. based on species, geographic areas, impact domains) and clear feedback mechanisms between authors and reviewers. The GB non-native species risk analysis panel which guides this process for Great Britain might provide a suitable model for guiding risk assessment in that manner, with experts drafting risk assessments, review by peers, a consultation period and final adoption of the documents by a risk assessment panel (GB Non-Native Species Secretariat 2008).

Fig. 1
figure 1

Schematic representation of potential steps for an envisaged improved integration of invasive alien species risk assessments in decision making (black based on data; grey procedural aspects). a Peer review option. b Consensus option

Full size image

Peer review of risk assessment can be implemented in various ways. For example, the Environmental Impact Classification for Alien Taxa (EICAT) (Blackburn et al. 2014) recommends assessments to be independently reviewed by at least one appropriate reviewer. In this context, the reviewers need to be familiar with the applied protocol rather than the species. They should not have been involved in the initial assessment process, and their role is to check that the data and questions have been interpreted correctly and consistently, categories and uncertainty handled appropriately (Hawkins et al. 2015). The EICAT process then proposes ratification by a unit consisting of members of the IUCN Invasive Species Specialist Group and suggests risk assessments are made available on a website to relevant stakeholders.

Approved assessments as well as their review history could be disclosed to stakeholders and the public. As an example, this way of working was applied in the consultation phase on the potential application of augmentative biocontrol for the invasive Japanese knotweed Fallopia spp. in the UK, with key data also being published in a peer-reviewed journal (Shaw et al. 2009, 2011; Waage 2009). Making risk assessments publicly available can be a good option for controversial species where multiple interests may be in conflict, but such a process is time demanding. For the knotweed this process took several years. Such a long process is not the best option when a rapid policy response to invasion is required. Alternatively, a more classical editorial process could also be applied to ensure that reviewing can be performed independently and that authors are obliged to react to comments. Expert engagement could also be increased by publishing risk assessments in a dedicated journal, such as the monographs dedicated to invasive plants in Europe (Fried et al. 2016). However, introducing such scrutiny in performing risk assessments will always be time- and resource-demanding.

The peer review option clearly provides benefits in terms of traceability and transparency. However, as any disagreement is usually not resolved, such risk assessments are based on a single perspective and remain the opinion of one (or a limited number of) experts. Peer review approaches do not quantitatively address contradicting evidence or differences in expert opinions. Risk assessment for invasive alien species listing has not kept up with recent progress in the field of expert elicitation in support of environmental decision making (Martin et al. 2012; Drescher et al. 2013; Morgan 2014; Sutherland and Burgman 2015). To overcome this, we suggest a second approach including a method of consensus building (Fig. 1b). Consensus building is a form of expert elicitation where the weighted assessments of all experts are pooled to provide a consensus judgement. Consensus is mostly obtained based on percent agreement in a moderated participatory process. Various participatory techniques can be applied to reach consensus, for example through Delphi technique, an iterative structured group communication process used for gathering and evaluating expert knowledge (Mukherjee et al. 2015). This usually comprises several rounds of structured questionnaires, followed by aggregation of responses and feedback to the experts. Some issues need to be addressed to adequately carry out the Delphi method. For instance, a balanced selection of experts, defining the consensus level, allowing anonymity for contributing experts, how to analyse each consensus ‘round’, and how to provide feedback to experts (Keeney et al. 2006). Whatever the elicitation method used, the key elements are to use a transparent elicitation process and to establish a representative pooling of judgments (Hsu and Sandford 2007; Martin et al. 2012; Morgan 2014; Sutherland and Burgman 2015). Different sources of subjective biases can also be minimised through a well-designed elicitation process, to ensure that remaining differences reflect true differences in opinions among experts. For example, unclear questions, due to linguistic uncertainty, can be explained and resolved during the process of consensus building. Lack of evidence can be flagged and contradicting information can easily be tracked and discussed.

Compared to the peer review option, a multi-expert risk assessment based on individual assessments with consensus building allows to track different types of uncertainty and to calculate metrics of expert agreement. The different assessments can be treated like survey results and different indices can be used to calculate inter-assessor reliability in addition to overall risk scores (Krippendorff 2012). Importantly, if the predefined level of consensus was not reached, it also identifies the extent of disagreement which is important for decision making.

We argue that compared to peer-review approaches, the consensus option provides additional benefits when gathering and evaluating expert knowledge. First, the process elicits different expert opinions, which can be differently valued or weighted by a decision maker according to the domain of impact considered (human, animal, plant). The approach also stimulates mutual learning among assessors (Kolfschoten 2013). Interestingly, computer-based group support systems are available to support in consensus building (Gnatzy et al. 2011). Shared assessments also offer broader consensus, which decreases the risk of poor policy or stakeholder support. Performing participatory multi-expert assessments is also more efficient in terms of avoiding redundancy (the same work is not done independently in different countries) and mobilizing expert time. Moreover, from a practical point of view, performing shared assessments can provide a basis for management at an international level as different governments conclude on similar risk score for the same target species and are thus likely to manage invasions at a similar level of urgency. Indeed, acquiring a more streamlined approach towards cross-border management of IAS is an important objective of the EU IAS Regulation (Tollington et al. 2015).

Despite these advantages, it must be recognized that consensus building also has its limitations. First, depending on the level of agreement, the process of including different rounds of consensus building can take considerable time (Hsu and Sandford 2007). Second, depending on the type of expert elicitation, combining judgments can move views towards the average and miss extreme outcomes and outliers (Martin et al. 2012) which might be of importance to risk assessment of IAS. Clearly, expert elicitation does not simply represent a low-cost, nor low-effort alternative to extensive research and analysis (Morgan 2014).

In the end, invasive species risk assessments must face the axiom that “Uncertainty exists, but regardless, decisions must be made” (Leung et al. 2012). Assuming that appropriate elicitation and quality control methodologies have been used to reduce uncertainty related to question formulation and conflicting evidence, the main sources of uncertainty should be related to lack of data, or differences in interpretation of an uneven evidence base. In these cases, some advocate the use of the precautionary principle and consider the species a higher risk, until proven differently. For instance, the Australian Weed Risk Assessment has been effectively used for many years. The protocol uses the answers to 49 questions concerning the species’ biology, biogeography, and behaviour elsewhere, to classify a plant species according to its risk of becoming invasive (Hulme 2012). Based on the precautionary principle it is using species scores to classify species as permitted, rejected, or prohibited, without recourse to expert discussion (Gordon et al. 2008; Pheloung et al. 1999). Such a straightforward approach is useful for species which are not yet or only locally established, so as to effectively prevent entry of any potentially harmful organism. However, in the case of already established and widespread species, further evaluation should also focus on management feasibility (manageability) issues. This is especially true when dealing with so-called charismatic invasive species, such as for example parakeets or squirrels, as parts of the general public may passionately advocate or oppose management of such invaders (Crowley et al. 2016).

Tackling risk management and risk communication

Risk assessment is only one pillar of the decision process. Prioritizing actions should also be based on the feasibility of management. Hence, different species and/or management options might come into play considering the resources available for prevention, eradication, containment or control. Cost-benefit analysis of invasive species management is often complicated and information demanding (Panzacchi et al. 2007; Pearce et al. 2006; Yokomizo et al. 2012) and there is a paucity of literature documenting the outcome of specific management actions (Simberloff 2003, 2009). Invasion scientists and practitioners are therefore developing robust scoring protocols to broadly assess the manageability of species in relation to various management options (Booy 2015). These protocols are mostly based on expert knowledge of the species distribution and abundance, the probability of reinvasion, the effectiveness of management options, the prevailing legislation and public acceptance of the eradication or management measures. These tools are only starting to be applied across species and regions. Like risk assessment, these expert opinion-based risk management evaluation will have to consider proper quality control. Here, risk analysis should link with methods from other disciplines fit to deal with the inherent complexity and uncertainty of biodiversity management problems such as structured decision making (Gregory et al. 2012), scenario planning and adaptive management strategies (Hulme 2012).

Also, a critical yet often overlooked area of risk analysis is the dissemination of information on the risk of new introductions and the consequent management choices made through risk communication. We argue that integrating transparency as much as possible both in risk assessment and risk management will effectively increase understanding of the risk and facilitate decision making. In parallel, innovative solutions should be explored to get the message across to a variety of stakeholders and the public. These include tools to better visualize risk and related uncertainty (Holt et al. 2012).

Conclusions

Implementing IAS legislation and management commonly relies on alien species listing processes. Such listings potentially affect multiple stakeholders. Therefore, comprehensive risk assessments that build on all available evidence and expertise, and that apply schemes that are fit to purpose are needed. Quality control on the content of assessments is essential but is currently typically lacking. The way assessments are performed should also be subject to clear procedures to ensure scientific rigour and repeatability. This could be achieved through a peer review process with clear feedback mechanisms between assessors and reviewers in order to improve assessment quality. Alternatively, a consensus approach offers multiple advantages, including additional indices on both the confidence regarding the risk posed by an IAS and on inter-rater reliability. Better documenting reasons for low confidence on risk scores also has the potential to greatly increase the quality of the risk assessment process. It should be acknowledged that the predictive value of risk assessments is constrained since they might not fully capture the complexity and contextuality of invasion processes. Since risk predictions using expert judgement inherently suffer from cognitive bias (Hulme 2012), it is important to stress that even a perfect risk assessment protocol and risk analysis process might not represent the holy grail for decision making. The peer-review and consensus building approaches put forward here will not automatically solve all of these issues, but will at the very least help to increase invasive species risk assessment transparency and will facilitate identifying the fundamental sources of uncertainty. Beyond the risk assessment per se, it is not made explicit how the outcome of the assessments should be handled and used, often leading to ambiguity in how to link assessments with subsequent policy decisions. Transparent translation of risk assessment outcomes in structured decision making is however a prerequisite to ensure reliability, credibility and endorsement of the outcome by stakeholders and the public (Hattingh 2011; McGeoch et al. 2012). It is therefore crucial to make better informed decisions in order to ensure efficient allocation of available conservation budget to counteract the growing challenge of invasive alien species.