Introduction

Classical biological control involves the introduction of natural enemies (NEs) to new environments to control unwanted pests or weeds. Formal applications to introduce biological control agents (BCAs) are increasingly required throughout the world as a result of global concerns about adverse environmental impacts (Follett and Duan 2000; Lockwood et al. 2001; Barratt 2011). Applications usually require risk assessments and there are several high-level guidance documents to assist biological control practitioners (FAO 2005; OECD 2004; EPPO 2014), and the regulators in several countries have developed their own standards and information requirements (Pyle and Gough 1991; Barratt and Ehlers 2017).

Risk can be defined as the probability and consequence of some (usually adverse) outcome. Risk is inherently influenced by a combination of environmental, economic, social/cultural, and technical factors which contribute uncertainty to the outcomes of decisions. Failing to acknowledge these uncertainties during the risk assessment process increases the chance of unanticipated outcomes (Meenken et al. 2021). In extreme cases unacknowledged and/or unknown uncertainties can lead to an extreme and low probability event sometimes called a ‘black swan’ event (Aven 2015).

There has been considerable discussion in the literature on risk, and how it pertains to biological control e.g., (Heimpel and Mills 2017). The aim of risk assessment is to provide information about possible adverse effects that might result from a decision to release a biological control agent into a new environment (Pyle and Gough 1991).

The consequences of uncertainty in risk assessment for biological control differs from many other environmental risk assessments. In biological control, the BCA is introduced from outside the ecosystem and has been specifically selected to successfully establish and spread. Hence the process is essentially irreversible and not realistically possible to further manage. This contrasts with, for example, uncertainty about the risk of a point source of water pollution that can be managed if adverse effects are detected. Furthermore, accurate prediction of the organism's wider impact on the ecosystem is notoriously incomplete and uncertain because of the enormous complexity of ecosystems (Darbra et al. 2008), leading to multiple, often interacting, risk factors for which there may be little or no data (McCoy and Franks 2010).

Uncertainty can arise from unpredictable random probabilistic variability (often known as aleatory uncertainty) or as result of imperfect knowledge or expertise (epistemic uncertainty). The latter can be reduced by acquiring more knowledge or information, whereas the former, sometimes called irreducible uncertainty, cannot (Fox and Ulkumen 2011; Zio and Pedroni 2012). So, if epistemic uncertainty is attributed to imperfect knowledge, then it should be possible in some cases to reduce uncertainty by searching for existing information or carrying out research that will allow more accurate prediction of outcomes (Marcot 2021).

Uncertainty can also be categorised as 'known’ versus ‘unknown’ unknowns. Aven (2015) defines: ‘unknown unknowns’ as events completely unknown to the scientific environment, ‘unknown knowns’ as events not identified from the perspective of those who carried out a risk analysis (or another stakeholder), and ‘known unknowns’ are those events that can be identified as contributing to and helping to quantify uncertainty. The latter can in some cases be reduced by further investigation as for epistemic uncertainties, but in many cases are irreducible. In the biological control application system in New Zealand significant effort is made to incorporate unknown knowns via processes that enable a different point of view to be incorporated, for example by seeking advice from experts and the community (Pyle and Gough 1991). Identifying the various dimensions and variables that contribute to uncertainties and unknowns are key to identifying the set of factors and interactions that form a causal web (Marcot 2021). Identifying cause and effect is central to predicting biocontrol outcomes.

Applications to import BCAs into New Zealand are made under the Hazardous Substances and New Organisms Act (HSNO Act 1996). The HSNO Act is regulated and administered by the Environmental Protection Authority of New Zealand (EPA NZ), formerly known as the Environmental Risk Management Authority (ERMANZ). The HSNO Act requires a precautionary approach to be taken since it assumes that all new organisms introduced into the environment in New Zealand will have some effect on the environment. It requires decision makers to be “cautious when there is scientific and technical uncertainty” (ERMA New Zealand 2010). The Act also requires that the Authority makes decisions in accordance with a prescribed Methodology Order (ERMA New Zealand 1998) and with respect to uncertainty, the Methodology Order instructs the Authority to “determine the materiality and significance to the application of the uncertainty” in discussion with interested parties, and where this cannot be resolved, to exercise caution in managing adverse effects. If uncertainty has arisen because of insufficient information provided by the applicant, a request for additional information can be made. If the Authority considers there is uncertainty in relation to risks and benefits, it needs to establish the range of uncertainty and consider the probability of the risks and benefits being “either more or less than the levels presented in evidence” (ERMA New Zealand 1998).

Applications under the HSNO Act require an application form to be completed with information on: the full identification of the organism; a full analysis of risks, costs and benefits; whether any controls can be applied that might mitigate risk; and whether the organisms is an unwanted organism as defined by the Biosecurity Act 1993. It must also fulfil the criteria that it must not: displace or reduce a valued species; cause deterioration of natural habitats; be disease-causing or be a parasite; be a vector or reservoir for human, animal, or plant disease; have adverse effects on human health and safety or the environment. Applicants are required to consult with Māori (indigenous people of New Zealand) to determine whether the application to introduce a new organism for biological control presents any risk to their cultural values. To assist in fully incorporating the knowledge, values and philosophy of Māori into decision-making, a framework has recently been developed as a tool to guide a dual focus incorporating both Māori and conventional science-based evidence into the process (EPA 2020). Once an application is formally accepted by the EPA it is publicly notified, and stakeholders and the public are able to make submissions either supporting or opposing the application. After these are received, an EPA advisor prepares a detailed report based on their evaluation of the applicant’s assessments, and the submissions from stakeholders and the public. A committee appointed by the Minister for the Environment is charged with making a decision based on these documents. Submitters can request to be heard in support of their submission, and if this occurs a public hearing is convened. The time period from the application being formally accepted, and a decision being made is 100 working days unless further information needs to be sought. Since the HSNO Act came into force, very few applications have been declined.

The objective of this contribution is to identify and characterise areas of uncertainty, both for risk of adverse impacts, and benefit of a BCA introduction, that have been expressed (by applicants, EPA staff and decision-makers, submitters, and consulted stakeholders). In a subset of applications for classical biological control releases, we sought to: identify commonly expressed areas of uncertainty, determine whether there were any differences relating to the target (weed or insect), examine differences over time since HSNO was implemented (1999–2019), and whether applicants and the EPA were identifying different numbers and types of expressions of uncertainty (EoUs). It was anticipated that, for areas of epistemic uncertainty at least, pinpointing where the most frequently occurring areas of uncertainty occur could inform future research targeted at reducing some aspects of uncertainty in future applications. It is expected that our findings will apply widely to other jurisdictions where the introduction of biological control agents is regulated.

Materials and methods

The applications made for BCA introductions since the HSNO Act was enacted in 1998 are summarised in Table 1. To date there have been 52 applications made to release BCAs against 33 target species, of which 51 have been approved (Table 1). All documents pertaining to every application (the EPA staff assessment, public submissions and the decision) are available on the EPA website https://www.epa.govt.nz/.

Table 1 Number of applications processed by the New Zealand EPA under the Hazardous Substances and New Organisms Act for biological control agent release, and those approved for release, shown for weed and insect targets between 1999 and 2020
Full size table

Categorisation of the data

A subset of 20 applications for BCA release for which decisions have been made were selected as case studies to represent a range of target species (insects and weeds), using a range of BCA taxa and spread of application date (between 1999 and 2019) (Tables 1, 2). Three source documents relating to each application were examined, the application itself, the EPA staff report on the application, and the final decision by the appointed committee. As an indication of the amount of information examined, applications averaged 48 pages (range 18–92), the EPA staff reports averaged 49 pages (range 15–102) and the decision documents averaged 18 pages (range 7–30), all excluding appendices. These documents have been separated in the analysis to determine which items have been raised by the applicant, and which have been raised in addition to those noted by the applicant. The latter two, the EPA staff assessment report, which documented submitters concerns, and the decision document have been combined because of the restatement and overlap of EoUs in these two documents. The complete text of these documents was read and all comments expressing uncertainty either in an area of potential risk or benefit were highlighted. A brief description of each individual EoU was entered into a spreadsheet and then categorised by combining those that were considered to be referring to the same specific issue (see Table 3). In cases of doubt about interpretation, the specific comment was revisited to check the context and categorise accurately. To facilitate statistical analysis the categories of EoUs were then further grouped to allocate them to a more general (henceforth called ‘higher-level’) risk or benefit category (biological, social etc.) (Table 3).

Table 2 The target and biological control agents included as case studies in this analysis are shown in order of the year in which the application was formally received by the New Zealand EPA
Full size table
Table 3 Categories of uncertainty for risk and benefit ordered from the most to least frequently mentioned EoUs in applications
Full size table

Data analysis

The number of EoUs per application per higher-level risk or benefit category were analysed to explore whether the number or type of EoUs had changed over time (based on the date of the application), source of the EoU (from the application or EPA documents), and whether they varied with the target organisms (weeds vs. insects). Due to the structure of the data, two separate analyses were carried out for each of the risk and benefit datasets. First, the data were modelled via a generalised linear mixed model (GLMM) (Schall 1991), assuming a Poisson distribution and using a logarithm link function. GLMM models allow random effects, in this case ‘application ID’. The fixed effects were Risk or Benefit Category, target, source and their higher order interactions. Fixed effects were assessed using the F- statistic (VSN International 2020). Two groups were excluded from the analysis since they had zero observations leading to model instability. These were the EPA + /cultural/social category for insect targets and the application/environmental impact category for insect targets. Time was not included in this analysis. Rather, a simple linear regression model was fitted separately to understand the relationship between risk or benefit through time. All analyses were carried out using GenStat v. 20.1 (VSN International 2020).

Results

Categorising EoUs

Examination of the 20 selected BCA application documents identified EoUs related to risk and benefit that were grouped into 29 and 21 categories respectively (Table 3; Fig. 1). The percentage of case study applications in which each EoU risk or benefit category occurred is also shown in Table 3. The most common categories recorded for risk were direct non-target (NT) effects (85%), cultural risks (75%), and whether there were already existing (and possibly effective) NEs of the target present in the new range (70%) (Fig. 1a). Cultural risks included concerns for the safety of ‘taonga’ (species particularly treasured by Māori) and other indigenous species. Indirect (e.g., food web) effects and adequacy of host range testing were also mentioned in more than half of the case study applications and associated documents. Post-release monitoring was mentioned in 20% of case studies in the context of uncertainty that this would take place, or the strategy for this.

Fig. 1
figure 1

Expressions of uncertainty a for risk and b benefit for insect and weed biological control applications calculated per application listed in decreasing order of numbers for weed targets. T target species, NT non-target species, BCA biological control agent, NE natural enemy

Full size image

For uncertainty relating to benefits, 75% of case studies mentioned the uncertain efficacy of the proposed BCA, or if it would establish and spread (60%). Half of the case studies expressed uncertainty about the cost: benefit analyses that had been presented in the application (Fig. 1b).

Risks

There was strong evidence that the mean number of EoUs was not uniform across the five risk categories (F4,360 = 11.70, p < 0.001) (Fig. 2a, b). Further, there was evidence that this effect was not consistent between the two target groups (F1,360=14.21, p < 0.001). It appears that, overall, the largest number of EoUs noted were in the areas of ‘lack of background knowledge’ and ‘potential adverse impact’ (this latter effect was particularly pronounced for the weed target species, but the interaction was not significant (F4,360 = 1.08, p = 0.366). Similar numbers of EoUs were noted in the applications and the EPA report (including submitters) and decision documents. (Fig. 2a, b). A linear regression model showed no evidence of a significant trend of increasing or decreasing numbers of EoUs for risks during the period 2009–2019 over which the selected applications were received (F1,19 = 0.02, p = 0.878) (Fig. 3a).

Fig. 2
figure 2

Effect of source and higher-level risk category on mean number of expressions of uncertainty (EoUs) for: a EoUs for insect target risks, b EoUs for weed target risks, c EoUs for insect target benefits, d EoUs for weed target benefits. Bars represent back-transformed 95% confidence intervals using the average SE. Counts are back transformed from the log scale predictions from the GLMM analysis

Full size image
Fig. 3
figure 3

Total number of EoUs per application plotted for a risk and b benefit for each year in which applications were received between 1999 and 2019. The regression line is shown bounded by 95% confidence intervals

Full size image

Benefits

For benefits overall, there was strong evidence that the EoUs were not consistent between the benefit categories (F3,248 = 13.20, p < 0.001) (Fig. 2c, d). The number of EoUs was particularly high for the economic benefit category across both source and target types. Overall, there were more expressions of uncertainty for weed targets than for insect targets (F1,248 = 4.33, p = 0.039) (Fig. 2c, d). Similar numbers of EoUs were noted in the applications and the EPA report (including submitters) and decision documents (F1,248 = 0.36, p = 0.549) (Fig. 2c, d). A linear regression model showed no evidence of a significant trend of increasing or decreasing EoUs for benefits over the 20-year period in which applications used as case studies were received by the EPA (F1,19 = 0.45, p = 0.513) (Fig. 3b).

Discussion

Our analysis has highlighted areas of uncertainty which have been most frequently expressed during the process of assessing risks and benefits of proposed BCAs by the applicant and assessors of applications in New Zealand. Over the 20 or more years since the HSNO Act 1996 legislation was implemented (1998 for new organisms) there seems to have been no detectable change in the frequency of identification or reporting of EoUs for risk or benefit for reasons which are unclear. Had earlier applications regulated under different legislation been considered, some significant changes might have been identified. There were also no differences in the numbers of EoUs identified for weed or insect targets for any of the major risk categories. It also seems apparent that the applicants themselves raise most of the EoUs in their application, and the EPA staff (and submitters to the EPA) raise some further issues such as cultural and social aspects, and quarantine test results, which may have arisen from public submissions.

Risks and benefits associated with the introduction of BCAs can be considered from the early concept of a potential biological control programme, when a potential BCA is identified for the target before the organism has been imported, through to considering the results of biosafety tests undertaken in quarantine, engagement with stakeholders and indigenous people, and then to the submission of the application. Quarantine host range tests provide some of the most important information that can contribute to the decision-making process. Hill (1999) argued that no-choice tests carried out in quarantine are as important if not more so than choice tests where the BCA is offered test species alongside the target host. The no-choice test is a conservative test (Kaufman and Wright 2017), which if negative (showing no attack by the BCA on the test species), or failure to develop successfully can provide regulators with a degree of comfort that the organism is not a physiological host and so is unlikely to be subject to on-going or population level impacts in field conditions (Murray et al. 2010). If it is positive, then it means that the NT species is a physiological host indicating that further testing using choice tests should be conducted (Murray et al. 2010). It is often argued that a positive test in the artificial conditions of the laboratory overestimates the likely field host range and fails to consider ecological factors that might influence host range (e.g. Kaufman and Wright 2017). Choice tests can provide information on host preference of the BCA, but whether this test in quarantine is representative of field conditions is also debateable. BCAs use a variety of host-finding mechanisms in the environment and setting up realistic tests to simulate these conditions in quarantine is challenging. Complex modelling and statistical methods are increasingly being utilised to help overcome the issues of variable responses and outliers (Withers et al. 2013), but the more complicated these methods become the greater the challenge in assuring assessors of the applications about the interpretation of the results. McCoy and Franks (2010) suggested that risk presented by an introduced BCA to NT species could result from inadequacies in taxonomy, host specificity testing, and knowledge of basic ecology, a prediction that to some degree is consistent with our findings for categories of uncertainty.

Considerable progress has been made recently in improving host-specificity testing. Bayesian Network modelling using comprehensive ecologically based information can inform risk assessment and estimate the probability of a negative impact of an entomophagous biological control agent on non-target populations (Meurisse et al. 2021a; b). For weed biological control, a tool has been developed to guide the prioritization of cultivar for test species selection for host-range testing to reduce uncertainty in risk assessment (Lefoe et al. 2022). The value of integrating chemical–ecological methods into host specificity testing procedures has also been demonstrated (Avila et al. 2016a, b; Park et al. 2018; Saunders 2022).

The ‘authors’ of EoUs in our EPA documents (which included applicants, submitters, EPA staff, Māori cultural advisers, and decision-making committee members) were probably not aware that they were indeed commenting on an aspect of uncertainty per se, but they were commenting on unknowns or concerns that were not possible to resolve. The applicant would often argue that their testing of a number of potential NT species provides a degree of confidence that the BCA would not attack other species, but they often commented that they could not account for the possibility of undiscovered related species being present in the environment, or laboratory testing not being representative of what would happen in the field, or obscure indirect effects having an adverse impact. This is reflected in the similar amount of uncertainty around potential adverse impacts of the BCA being expressed in both applicant and EPA documents. Expressions of uncertainty do not necessarily translate to high risk, they are simply statements made, often by the applicants themselves, that they are unable to be definite about a particular aspect of their assessment of risk or benefit. For example, an applicant cannot be absolutely certain that a BCA will establish or spread in the environment if it is released.

It must be acknowledged that an application to import a biological control agent has been prepared by a committed research team that has invested many years in the identification of a potential agent, carried out biosafety evaluations, economic evaluations, stakeholder engagement, and then put an application together which is not a trivial exercise. Their objective in proposing to release a BCA is to mitigate the impact of a pest which is a severe limitation to primary producers, or negatively impacting the natural environment, threatening biodiversity or causing significant costs to land managers. We speculate therefore that applicants are unlikely to emphasise that more testing, more replication, different types of testing, etc., could help reduce uncertainty. Furthermore, the regulators job is to make a decision on the application as submitted, with the guidance and advice of stakeholders and submitters, and sometimes experts. It is not their role to send the applicant away to reduce levels of uncertainty. The regulator would rather ensure that the evidence presented in the application is in a suitable form for a decision to be possible before it is formally accepted. This is achieved by dialog between the applicant and the regulator from the early stages of the application being prepared.

Accepting an inevitable level of uncertainty can help interpret research results and thereby highlight key unknowns which have the most influence on a decision (Marcot 2021). It is important also to consider ‘value of information’ (VOI) which helps determine the degree to which reducing uncertainty (by carrying out more research for example) can provide more certain outcomes. In other words, a VOI analysis can help decide on best use of funds which will be most likely to improve outcomes from making a particular decision (Marcot 2021).

Post-release validation of risk associated with the release of biological control agents can provide valuable feedback to regulators about the outcome of a release and, in turn, areas of uncertainty in the predictions made prior to approving the release. Paynter et al. (2004) carried out a survey of 20 case studies of weed biological control agents released in New Zealand and found that most proved to be host-specific to their target plant, two were shown to cause very minor damage to native plants which had been predicted pre-release, and for two others, non-target attack had not been predicted, although in these cases, the quarantine testing was deemed to have been inadequate. An equivalent survey has not been carried out for insect biocontrol agents, but in four cases in New Zealand that we are aware of, one reported less non-target impact than was predicted (Gonzalo Avila pers. comm.) (Avila et al. 2023) one was predicted to have a very narrow host range which proved accurate (Goldson et al. 1992; Barratt et al. 1997), one has a much wider host range with the possibility of population impacts (Barlow et al. 2004) but quarantine testing was inadequate (Barratt et al. 2007), and one has a wider host range than predicted when the standard of quarantine testing was good (Goldson et al. 2005); Colin Ferguson (unpublished data).

The main objective of this study was to identify the most commonly occurring areas of uncertainty in risk assessment from application documents to inform future research. “At best, uncertainty should motivate curiosity and investigation. Knowledge may support power, but uncertainty should engender creativity.” (Marcot 2021). Below we consider a few of the most frequently occurring EoUs identified in this study and discuss recommendations for future research which might help reduce uncertainty in decision-making.

  1. 1.

    Unsurprisingly, uncertainty around the potential for NT impacts was raised in the majority of case studies and is well recognised in the literature (Howarth 1991; Follett and Duan 2000; Bigler et al. 2006; Simberloff 2012). There has already been a considerable amount of research conducted in this area, both in the development of tests that can be carried out in quarantine (Avila et al. 2014) and their interpretation (Withers et al. 2013) in addition to traditional exposure of carefully selected test species to the proposed BCA (Todd et al. 2015). This includes behavioural and physiological research on olfactory recognition of potential hosts (Avila et al. 2016a, b), and modelling to evaluate probability of risk (Paynter and Teulon 2019; Meurisse et al. 2021b). Adoption of these techniques is, of course, dependent upon resources available to the applicant.

  2. 2.

    Prediction of indirect NT impacts is a very difficult area for decision-makers because of the complexity of interactions between species in ecosystems. Constructing food webs can be helpful and some work has been done in this area using data collected post-release (Memmott 2000) and using quantitative food webs to try to predict indirect effects pre-release (Lopez-Nunez et al. 2017). Using qualitative food webs, Todd et al. (2021) found that showing all known connections between species in a food web diagram could help to reduce uncertainty around which species might be at risk. Kotula et al. (2021) tested machine-learning approaches for predicting indirect effects and found that while they were not able to predict indirect effects, they did have potential to rank hosts as having low or high risk of indirect effects. These authors suggested that validation of such predictions post-release would be of value to regulators in future decision-making. Clearly there is considerable potential for more research in this area.

  3. 3.

    The lack of background knowledge of various aspects of the taxonomic and ecological affinities of the pest, potential NT hosts and the BCA, were noted in 70% of case studies. Specifically, uncertainty about the existence of natural enemies of the target pest that might already be in the new range and hence able to reduce pest densities; the lack of information presented in applications on the potential NT fauna in the new environment and possible NEs of the proposed BCA were also frequently raised as areas where it was considered that insufficient information was available. The acquisition of data which could provide such information is likely to involve a substantial research effort and hence be highly resource dependent. In many cases applicants have raised these issues, however, exhaustive analyses of these aspects are rarely possible, and receives cursory discussion. It is unlikely that generic research to reduce uncertainty in these areas is possible, but a case-by-case approach using the modelling tools mentioned above (Meurisse et al. 2021a) could be informative.

  4. 4.

    Cultural and social risks were recorded in 75% of applications. Māori expressed a wide range of concerns mostly centred around the potential impacts on Māori values, indigenous species that are treasured (taonga), and the wider impact of non-native species on the environment. It was regularly noted that Māori cultural concerns were not adequately considered in applications for uncertainties relating to both risk and benefit of a proposed BCA release, and hence the biological control release was often opposed. There is considerable potential for research to assist with cultural concerns, particularly in presentation of Mātauranga Māori evidence (Māori knowledge: the body of knowledge originating from Māori ancestors, including the Māori world view and perspectives) alongside western scientific evidence. This would create a uniquely New Zealand way of addressing environmental issues such as these.

  5. 5.

    EoUs in the benefit categories were dominated by economic considerations, but also about uncertainty as to whether the BCA would establish and spread and provide effective reduction in pest damage (Environmental Impact Fig. 2). Applicants often attempt cost–benefit analyses and in some cases use professional resources for this. There are several studies which have very successfully demonstrated substantial economic benefits from successful biological control introductions (Jarvis et al. 2006; Molina-Ochoa and Foster 2011; Basse et al. 2015; Naranjo et al. 2015; Benjamin and Wesseler 2016; Ferguson et al. 2018; Valentea et al. 2018). It is acknowledged that it is more difficult to apply cost–benefit analysis to biocontrol for conservation benefit (Simberloff and Stiling 1996) because it is challenging to place a dollar value on species or ecosystems. Uncertainty was expressed about the economic value of the pest, and the impact that the biological control agent would have on this, and hence the beneficial value of the biological control introduction. Applicants often use costs of alternative (or currently used) control methods such as pesticides to offset the costs of the program. Research to develop a standardised framework or methodology for cost–benefit analysis for biocontrol which applicants are advised to use would be useful.

Using applications from the New Zealand regulatory system for introduction of new organisms, this contribution has identified some of the most common areas of uncertainty that are likely to apply in any system internationally. We have summarised the most frequently recurring areas of uncertainty in both risks and benefits for BCAs yet to be released. However, research that could assist in reducing uncertainty in the future should undoubtedly include post-release studies of introduced BCAs. Such studies enable investigations to compare direct NT impacts predicted from quarantine tests with realised field impacts. We have made some recommendations for research that could reduce uncertainty in some areas where there are known unknowns or epistemic uncertainty. Finding resources for this research will in some cases be challenging especially where they relate to complexity in the underlying ecological system. However, acknowledgement and consideration of uncertainty is essential if we want to improve our capacity for risk assessment in the future and to utilize biological control more fully as a tool for pest management.