Limits of Knowledge and Tipping Points in the Risk Assessment of Gene Drive Organisms

New challenges in the risk assessment of genetically engineered (GE) organisms are expected to emerge in the context of so-called ‘gene drives’. Based on a review of findings from current knowledge of GE organisms, it is concluded that the risk assessment of gene drive organisms intended for release into the environment will inevitably suffer from major uncertainties, ‘unknowns’ and methodological problems: subsequent generations of GE organisms might show effects that were not observed or intended in the first generation. Unintended effects can, for example, emerge from interaction with genetic backgrounds within natural populations or be triggered by changing environmental conditions. Due to the increasing spatio-temporal complexity associated with the long-term persistence and propagation of GE organisms, risk assessment can no longer be expected to produce sufficiently reliable results. It has to be assumed that at a certain point in the dissolution of spatio-temporal boundaries a tipping point will be reached, that will make reliable risk assessment impossible. Moreover, methodological problems need to be overcome: the comparative approach that is the starting point for current European Food Safety Authority (EFSA)’s environmental risk assessment might not be applicable due to the lack of suitable ‘comparators’. Despite increasing uncertainties, risk assessors and risk managers need to solve the problems of how to come to robust conclusions and make reliable decisions that take the precautionary principle (PP) into sufficient consideration. The introduction of a new step in the risk assessment of GE organisms has been suggested to solve these problems—which is the ‘spatio-temporal controllability’ and takes three criteria into account: 
 
(1) 
 
the biology of the target organisms, 
 
 
 
 
(2) 
 
their naturally occurring interactions with the environment (biotic and abiotic), 
 
 
 
 
(3) 
 
the intended biological characteristics (traits) of the GE organisms.


Introduction
People have selected and cross-bred plants (and animals) for thousands of years in order to establish beneficial and desirable traits. However, natural mechanisms such as gene regulation and heredity can now be circumvented with modern technical tools of genetic engineering. In consequence, experience gained from conventional plant breeding cannot simply be extrapolated to the risk assessment of GE plants. According to EU regulation (Directive 2001/18), all organisms derived from processes of genetic engineering require a risk assessment before they can be released.
New challenges have arisen with applications such as 'gene drives' that are intended to be introduced into natural populations and give rise to offspring that spread and propagate throughout those populations. Gene-drive mechanisms were, for example, successfully established in laboratory populations of Drosophila by using the nuclease CRISPR/Cas9 (Gantz and Bier 2015). Such organisms replicate the process of genetic engineering in a self-organised way: in every generation the nuclease is meant to copy and insert itself at a given location within the genome. This process is also named 'mutagenic chain reaction' (see Gantz and Bier 2015;Ledford 2015). As a result, the newly introduced DNA can spread through a population exponentially, and much more rapidly than could be expected under the Mendelian pattern of inheritance.
The risk assessment of a potential release of a gene drive organism into the environment needs to consider uncertainties and limits of knowledge on at least three levels: the technology, the target organism and the receiving environment, including abundant non-target organisms (NTOs). Moreover, methodological problems need to be overcome: the comparative approach that is the starting point for current EFSA (EFSA 2010) environmental risk assessment might not be applicable due to the lack of suitable 'comparators'. The following paragraphs therefore discuss criteria and methodologies that can be applied in the risk analysis of gene drive organisms in the face of substantial uncertainties.

The Production of Knowledge and Non-knowledge
There can be various reasons for non-knowledge. It can, for example, be due to unawareness of facts that are already known in a specific field of expertise. This kind of non-knowledge can be easily remedied. Other areas of non-knowledge and the production of non-knowledge can create large and systemic problems. In some circumstances, we might not even be aware of our limits of knowledge.

The Science of Non-knowledge in Upstream Technology Assessment
The science of non-knowledge already has a long tradition in the context of technology assessment. Some of the early debates were triggered by discussions around nuclear power and chemical pollution. This report discusses some selected aspects.
In 1992, Wynne developed a classification system based on the criteria "risk", "uncertainty", "ignorance" and "indeterminacy" as summarised in Table 8.1.
These criteria are explained by Wynne (1992) in more detail (for all quotes in bullet points see page 114): • The term risk can be applied when "the system behaviour is basically well known, and chances of different outcomes can be defined and quantified by structured analysis of mechanisms and probabilities." • The term uncertainty can be applied "if we know the important system parameters but not the probability distributions (…). These uncertainties are recognized, and explicitly included in analysis." • According to Wynne "a far more difficult problem is ignorance, which by definition escapes recognition. This is not so much a characteristic of knowledge itself as of the linkages between knowledge and commitments based on it-in effect, bets  (technological, social, economic) on the completeness and validity of that knowledge. (…) The conventional view is that scientific knowledge and method enthusiastically embrace uncertainties and exhaustively pursue them. This is seriously misleading. It is more accurate to say that scientific knowledge gives prominence to a restricted agenda of defined uncertainties-ones that are tractable-leaving invisible a range of other uncertainties, especially about the boundary conditions of applicability of the existing framework of knowledge to new situations. Thus ignorance is endemic to scientific knowledge, which has to reduce the framework of the known to that which is amenable to its own parochial methods and models." • Finally, "indeterminacy exists in the open-ended question of whether knowledge is adapted to fit the mismatched realities of application situations, or whether those (technical and social) situations are reshaped to 'validate' the knowledge." Böschen (2006Böschen ( , 2009 and his colleagues explored areas of non-knowledge, especially in the context of biotechnology. Böschen et al. (2006) put forward three dimensions of non-knowledge (p. 297):

The Exploration of Non-knowledge in the Field of Biotechnology
• "The first dimension refers to knowledge (or awareness) of non-knowledge, which spreads between full awareness of non-knowledge (we know what we don't know) and complete unawareness ('unknown unknowns'). • The second dimension, intentionality of non-knowledge, contrasts unintended nonknowledge with the conscious refusal of certain cognitions. • The third dimension, temporal stability (or reducibility) of non-knowledge, extends from what is not yet known, but (presumably) does not present any substantial difficulties to cognition, to the entirely 'unknowable' and therefore uncontrollable." Based on these criteria, the perspective of scientists in the field becomes important and allows exploration of what is known as 'cultures of non-knowledge'. Böschen et al. (2006) show that two interrelated scientific disciplines-molecular biology and ecology-"entail different types of non-knowledge and deal with non-knowledge differently (…) The scientific culture of non-knowledge in molecular biology can be described as control-oriented, while that of ecology can be described as uncertaintyoriented." (p. 295).
In regard to the boundaries of knowledge, Böschen (2009) argues that one decisive question remains to be answered in order to implement the PP as requested by EU regulations e.g. 178/2002 andDirective 2001/18 (p. 509): "Although there is an institutional solution, one all-decisive question is remaining. What is the actual evidence on which decisions about the applicability of the PP are to be taken? And which evidence is necessary to decide about different precautionary strategies? These questions are difficult to answer with respect to the debate about non-knowledge. But they have to be answered, because political decisions are always based on knowledge (…)." Adequate management of non-knowledge in implementing the PP cannot be established if Böschen's (2009) question cannot be answered. Consequently, within the regulatory processes, inherent non-knowledge might increase unnoticed to such an extent that sufficiently robust risk assessment is disabled. Böschen (2009) shows there are basically two options in regard to performing risk assessment in this context: a "restrictive evidential culture (e.g., molecular biology)" and a "holistic evidential culture (e.g., ecology)". These are accomplished by a third approach which could be seen as a pragmatic compromise: "evaluative evidential cultures (e.g., environmental medicine)".
Böschen (2009) summarises (p. 513): "Control-oriented epistemic cultures proceed in a restrictive-experimental way and are oriented towards an improvement of (technological) options for action. In contrast, complexity-oriented epistemic cultures structure their knowledge in a holistic-contextual way and enhance options for reflection. Finally, expertise-based epistemic cultures are marked by the combination of diagnostic knowledge and knowledge on problem-solving. There, epistemic strategies are related towards an improvement of options for decisions. All of these cultures generate knowledge relevant for making decisions, but they do not find a balanced attention in the risk policy of the GMOs. Therefore, a selection process of the knowledge resources relevant for the conflicts occurs." Building on arguments and criteria developed by Böschen andhis colleagues (2006/2009), one can conclude that it is necessary to integrate adequate management of non-knowledge into the field of genetic engineering and biotechnology in upstream prospective technology assessment, as well as within 'end of pipe' regulatory decision-making for specific products and organisms.

Some 'Known Unknowns' in Regard to Risk Assessment of GE Organisms and New Challenges Posed by Gene Drives
Decisions made by risk managers are always dependent on plausibility and knowledge and cannot be based on speculation. Risk assessment has to be organised in a way that the final decision-making of the risk manager is sufficiently informed as to whether a GE organism can be considered to be safe and therefore allowed for release. However, due to the complexity of the associated mechanisms and various interactions between the target organisms and their environment, substantial uncertainties and areas of non-knowledge have to be taken into account when it comes to the risk assessment of gene drive organisms.
In this context, two questions are of crucial relevance: What are the 'known unknowns' stemming from experience with already existing GE organisms that give 'reasons for concern'? What are the main challenges in risk assessment of gene drives that go beyond the experience with existing GE organisms?
To explore these questions, some conceptual challenges of GE organism risk assessment are given as a starting point. Subsequent sections set out a more detailed investigation of the differences between GE plants and plants derived from traditional breeding. This is followed by a discussion of questions arising from the risk assessment of gene drives.

Conceptual Challenges in Risk Assessment of GE Organisms
As a starting point for the discussion on the risk assessment of GE organisms, it is useful to consider what is often called the complexity of biology. A comparison to the risk assessment of chemicals might be quite useful in this context: while chemicals (in many cases) can be considered clearly defined entities, the characteristics of organisms are largely shaped by interactions and the mechanisms of self-reproduction, self-organisation and adaptability. Conceptual challenges for the risk assessment of GE organisms can be identified on several levels.

What is the Entity that Has to be Assessed?
To some extent, life forms can only be assessed in combination with their environment: for example, the well-established concept of the 'holobiont' (see for example Richardson 2017) shows that multicellular organisms such as plants, insects or mammals can hardly be separated from their associated microbiome. The organism and its associated microbiome interact very closely: it is known that the microbiome can extensively impact the biological characteristics and health status of humans, plants and animals (see for example Lynch and Pedersen 2016;da Silva et al. 2016).

How to Assess Complex Cause-Effect Relationships?
Well defined cause-effect relationships may frequently not be applicable in the case of life forms: as can be shown in GE plants (for some references see below in 3.3), the interaction of the inserted genes with the genetic background as well as the interactions of the organisms with their environment can play an important role. These interactions can create effects in a bi-directional and non-linear manner: it is not only the organisms that impact the environment, the various environmental conditions, abiotic and biotic stressors also impact the biological characteristics of organisms. Thus, risk assessment of GE organisms not only has to assess the impact of the organisms on the environment, but also vice versa. In addition, the resulting combinatorial effects also have to be taken into account.

Will the Characteristics of the Relevant Entities Remain Predictable in Future?
The characteristics of GE organisms might change from one generation to the next (for some references see below in 3.3). With self-organisation and self-reproduction and in interaction with changing environmental conditions, next generation effects may occur that cannot be predicted on the level of the previous generations. Even if DNA is transmitted to the next generation in a way that genetic stability is assumed on the genomic level, this does not mean that the intended function of the gene and the associated phenotype will be transmitted to the offspring as well. Thus, next generation effects have to be considered in all cases where GE organisms might be able to persist and propagate in the environment. This is especially relevant if gene flow occurs from the GE organism into wild populations.

How to Take Communication and Signalling Pathways Between Organisms into Account?
Life forms interact with the environment via multiple bio-chemical pathways. In plants, these pathways include signalling and communication with other plants, microorganisms and beneficial insects (see for example Schaefer and Ruxton 2011). There are various compounds involved such as volatile substances, other secondary metabolites and biologically active compounds. Environmental risk assessment of GE organisms should include the various ways in which organisms interact and communicate with their environment, and these might not be well defined in all cases.
The conceptual challenges listed above show that, given the terminology introduced by Böschen (2009), a "restrictive evidential" approach" is not sufficient to perform risk assessment on GE organisms, and a more "holistic evidential" approach" has to be applied.

Specific Characteristics of GE Organisms
Risks of GE organisms differ from those associated with organisms derived from natural evolutionary processes. In the overview we have used plants to show some relevant differences; this is because most of the experience we have has been obtained from the environmental risk assessment of GE organisms in regard to plants.
Conventional plant breeding can look back on long-standing experience. The mechanisms used are generally based on the methods and results of evolutionary processes, such as selection: plant breeding starts from a broad range of biodiversity that is used for selection and is newly combined through crossing. In addition, since ca. the 1950s, it has been possible to enhance genetic variation by technically inducing mutations. These methods are known as mutagenesis (Oladosu et al. 2016) and do not profoundly change the pattern of emerging genetic variations; they more or less simply speed up evolutionary processes that might also occur naturally.
In short, the methods and mechanisms used in what is known as 'conventional' breeding: • make use of huge genetic diversity as a starting point; • are applied to the whole cell or organisms; • do not insert or delete genetic information targeted by technical means.
Therefore, conventional breeding does not change the mechanisms of natural heredity and gene regulation. Furthermore, in many cases, each step might be relatively small in regard to the relevant plant characteristics. Thus, plant characteristics are improved by breeding processes that can take many years and involve many varieties. This is a constantly ongoing process that allows breeders to gain experience with each of the specific traits over a longer period of time. Nevertheless, some organisms resulting from conventional breeding might require risk assessment in regard to health and the environment. For example, it is possible to establish herbicide resistant crop plants by means of conventional breeding, which should be investigated in regard to their impact on weedy species and biodiversity (Burgos et al. 2014).
On the other hand, genetic engineering is based on "techniques involving the direct introduction into an organism of heritable material prepared outside the organism" (Directive 2001/18, Annex I A). These techniques and processes show technical characteristics that are distinct from those of conventional breeding: • The techniques applied in genetic engineering allow mechanisms of natural heredity and gene regulation to be by-passed. • Direct intervention on the level of the genome means that traits can be established that do not occur naturally e.g. plants which produce insecticidal proteins derived from Bacillus thuringiensis (Bt). • In many cases, the additionally inserted genes are not identical to those found in nature: for example, in the case of plants that produce Bt toxins, the DNA sequences are modified in the laboratory giving rise to truncated or chimeric Bt proteins that do not exist in nature (see Hilbeck and Otto 2015).
The biological changes in regard to plant characteristics can in many cases be extensive and might even be considered 'disruptive'. The resulting plants might be cultivated on large scale without any experience being gathered over a longer period of time.
In summary, experience gained from conventional plant breeding cannot simply be extrapolated to the risk assessment of GE plants. Thus, according to EU law (Directive 2001/18), all organisms derived from processes of genetic engineering generally require risk assessment before they are released into the environment.
Environmental risk assessment (ERA) of GE plants examines several aspects (EFSA 2010): it encompasses the trait (such as the Bt toxin), the organism and its genetic stability (including gene expression, stability of the gene functions) and the interactions of the organism with the receiving environment on various levels.

Specific Challenges in Risk Assessment of Gene Drives
There are several genetic engineering techniques that can be used to create a gene drive, many of them are based on applications of the nuclease CRISPR/Cas. There is some proof of principle that gene drive can be established in yeast (Di Carlo et al. 2015), mosquitoes (Gantz and Bier 2015;Hammond et al. 2015;Kyrou et al. 2018), flies (Champer et al. 2017;KaramiNejadRanjbar et al. 2018;Buchman et al. 2018) and mice (Grunwald et al. 2018). Each of these outcomes will require a case by case risk assessment in regard to its technical characteristics, the target species and the receiving environment (Akbari et al. 2014;Kuzma et al. 2017;Noble et al. 2017;Oye et al. 2014).
In addition, there are some general characteristics that can be used to distinguish gene drive organisms from other GE organisms that have been assessed by EU institutions so far: • Where gene drives are based on CRISPR/Cas, the process of genetic engineering becomes inherited and self-replicating in subsequent generations; established as a self-organising process, largely outside of efficient or ongoing control mechanisms. • Changes in the 'laws' of inheritance are so fundamental that in many cases it will hardly be possible to find suitable comparators, even though this is a requirement of the 'comparative risk assessment' approach. • Gene drives have been developed specifically to genetically engineer species that are non-domesticated. Consequently, the additional genetic information will be introduced into a wider range of genetic backgrounds which-especially in wild, natural populations-can be quite heterogeneous and give rise to a wide range of unexpected effects (see also Chandler et al. 2013;Mullis et al. 2018;Evangelou et al. 2018;Saltz et al. 2018). • Gene drives are intended to target wild, natural populations, therefore, a wider range of possible interactions with the receiving environment and the ecosystems has to be expected. • If gene drives are introduced into wild, natural populations, it can become much more difficult to intervene if adverse effects emerge than with crops grown in the fields.
Some of the differences in risk assessment between gene drive organisms and GE crop plants are summarised in Table 8.2.
In conclusion, the hypothesis discussed in the following chapters is that gene drives and also other GE organisms that can persist and propagate in the environment and/or enable gene flow to wild populations, pose new challenges for EU risk assessment. What can be expected is a substantial increase in spatio-temporal complexity and a decrease in the robustness of overall risk analysis (see also Simon et al. 2018). The majority of crop plants are cultivated for only one growing period. These plants are not meant to reproduce spontaneously Next generations will emerge spontaneously; the process of genetic engineering is a self-organised process replicating in each generation Due to previous breeding processes, plant varieties as used for genetic engineering, are stable and have defined characteristics, as well as reduced genetic diversity. Seed quality can be controlled by breeders (or farmers) before and during cultivation Wild populations very often inherit a broad spectrum of heterogeneous genetic backgrounds. As a result, gene drive organisms can introduce their new genetic information into heterogeneous genetic backgrounds without additional controls in place Crop plants of the same species are often cultivated under similar environmental conditions in a managed agricultural system Wild populations e.g. insects are often exposed to a wider range of environmental conditions due to their mobility. Further impact factors include e.g. seasonal changes Crop plants are often grown in an environment of agricultural systems with reduced biodiversity

Some Reasons for Concern Arising from Existing Evidence
To test the hypothesis, it has to be investigated whether there is any supporting evidence that.
• the process of spontaneous self-reproduction of GE organisms increases uncertainty regarding genetic stability (including gene expression and stability of the gene functions) in the offspring generations; • a higher range of genetic diversity within the target populations increases uncertainty regarding genetic stability (including gene expression and stability of the gene functions) in the offspring generations; • interaction with a more complex environment increases the likelihood of unexpected effects in GE organisms.
To answer these questions, research was conducted within peer reviewed publications on the risk assessment of existing GE crop plants in the EU. Some of this research was aimed at GE plants that had unintentionally escaped into natural populations or had started to become feral (see, for example, Bauer-Panskus et al. 2013). Other relevant publications were those dealing with the responses of GE plants to changes in environmental conditions (see, for example, Zeller et. al. 2010). Based on the existing publications, evidence can be established for relevant aspects. Some of these findings are summarised in Table 8.3. Some examples are explored in more detail below. It was concluded that existing experience was in line with the above hypothesis. These findings can be explained in more detail by exploring some of the examples, whereby the process and result of genetic engineering can be assumed to be a cause for the unintended effects: • Bollinedi et al. (2017) crossed lines of so-called "Golden Rice" with the Indian variety Swarna and observed growth disturbance since the gene constructs interfered with the plant's own gene for producing growth hormone. Further, the gene constructs were not, as intended, active solely in the kernels, but also in the leaves. This led to a substantial reduction in the content of chlorophyll that is essential for vital functions in the plants. This effect was not observed in other varieties. Genetic background interaction is a commonly observed phenomenon in many species (see, for example, Table 8.3 for further examples). • Fang et al. (2018) showed that higher fitness does occur in GE glyphosate resistant plants in a glyphosate-free environment. According to this research, the enzyme EPSPS (5-enolpyruvylshikimate-3-phosphate synthase) produced in the plants not only makes the plants resistant to glyphosate, it also interferes with plant metabolism for growth and fecundity. As a consequence, the offspring of the plant can produce more seeds and be more resistant to environmental stressors such as drought and heat. They also describe the interaction between the genome and the environment: for Arabidopsis producing additional EPSPS enzymes it was observed that seed germination ratios increased significantly when transgenic seeds were exposed to heat and drought stressors, although no differences were found in seed germination among different lines when seeds were exposed to normal temperatures. • Transgenic oilseed rape is known to have become established independently of cultivation in several regions of the world, such as Canada, the US, Japan, Australia and Switzerland (Bauer-Panskus et al. 2013). Interestingly, some populations seem to be self-sustaining and can persist without additional gene flow (spillage) at Japanese harbors (Katsuta et al. 2015). Similar findings were also reported from Canada (Warwick et al. 2008;Knispel and McLachlan 2010). This is a strong indication that fitness of the offspring of the transgenic plants was underestimated.
Next generations effects were also observed in rice, especially if crossed with weedy rice: Cao et al. (2009) describe crosses between insecticidal rice with weedy rice that causes taller plants, more tillers, panicles and spikelets per plant, as well as higher seed weight, compared with the weedy rice parents. Seeds from the F1 hybrids had higher germination rates and produced more seedlings than the weedy parents.
It was concluded that the risk assessment or technical characterisation of GE organisms (plants) established in the laboratory or under controlled conditions cannot be seen as sufficient to predict all the relevant effects that can emerge in the next generations, and in interaction with the receiving environments. Therefore, parallel to an increase in spatio-temporal complexity, a decrease in the robustness of overall risk analysis is very likely.

The EFSA Concept and the Problem of Spatio-Temporal Complexity
The EFSA is the EU regulatory authority responsible for assessing the risks of GE organisms in regard to health and the environment. There is existing experience to show the way in which EFSA deals with the reasons for concern presented here. Further, the question arises of whether relevant issues can escape the current system.

The Current EFSA System and Its Approach to Future Applications
The Commission Implementing Regulation 503/2013 is applied in EFSA risk assessment of GE plants for import and usage as food and feed. It foresees a number of investigations to be performed and several sets of data to be presented by the companies. The way in which EFSA puts the regulation into practice can be regarded as more "restrictive evidential" than "holistic evidential" (using the terminology of Böschen 2009): in most cases, EFSA does not discuss limits of knowledge, and where there are uncertainties, EFSA does not generally ask for further data. For example, very often several significant findings regarding changes in plant composition are identified (see for example, EFSA 2018). However, EFSA does not see the need to request further investigations as long as there is no evidence that the changes in composition can cause harm to health or the environment (Testbiotech 2018a). Thus, risk assessment currently performed by the EFSA is largely based on a restrictive paradigm of evidence, or more broadly speaking, works within categories and systems in which risks can be quantified by structured analysis of mechanisms and probabilities, without giving sufficient weight to uncertainties and limits of knowledge.
A key element in the risk assessment of GE organisms is the comparative approach which is integrated in Annex II of Directive 2001/18/EC (see Section 2.2). As applied by EFSA (EFSA 2010), it requires the identification of differences between the GE organisms and their adequate comparator(s), of both intended and unintended effects.
Clearly there are some problems with the comparative approach. Some relevant information can be gained from comparison with wild species. However, organisms inheriting a gene drive able to produce offspring that can spread and propagate further by overriding the pattern of Mendelian inheritance, can cause changes in population dynamics and interaction with the environment that go far beyond what can be observed in the wild species.
In this context, it is notable that EFSA developed specific guidance for the environmental risk assessment of GE animals (EFSA 2013) that also addresses the issue of gene drive. So far, this guidance has not been used because there have been no applications for GE animals. It can be concluded from the published guidance (EFSA 2013) that EFSA assumes the risk assessment of GE animals and insects does indeed lead to a higher level of uncertainties in comparison to the risk assessment currently established for GE crop plants (EFSA 2010). For example, EFSA requests the applicant to consider several degrees of uncertainty: The formal analysis should address three broad types of uncertainty: long-term effects may also occur due to increases in spatial and temporal complexity. (p.39).
Furthermore, relevant issues in regard to NTOs (and the environment) include the ecological functions of specific species and their complex biotic or abiotic interactions: i. The ecological functions of specific species and their complex biotic or abiotic interactions (…) are not always fully understood.
ii. The methodologies for testing potential effects on NTOs are limited. Field trials might not be feasible in all cases, as it might be impossible to eradicate the released GM insect population if an adverse effect is identified related to the release, in particular, applying replacement strategies.
iii. The fact that it is not feasible to simulate the complexity of the receiving environments in laboratory tests, semi-field tests or modelling. (…) Consequences of the decrease or eradication in population size of a certain species or the replacement of wild population by GM insect populations might not be predictable. (p. 103).
EFSA also addresses the problem finding an adequate "choice of comparators" (p. 25). In summary, EFSA (2013) concurs with the hypothesis that the risk assessment of GE organisms carrying a gene drive poses new challenges in comparison to current EU risk assessment.

Some Relevant Aspects of Spatio-Temporal Complexity
It can be concluded that the following questions must be answered in respect to the risk assessment of GE organisms inheriting a gene drive and other GE organisms that can persist and propagate in the environment, including in regard to spatio-temporal complexity (see Table 8.4): 1. Can genetic stability be controlled in following generations? 2. How can genetic diversity in the target population be taken into account? 3. Will there be any gene flow to other species? 4. How can the population dynamics and life cycle aspects of the target species be integrated? 5. Can the receiving environment be defined in regard to relevant interactions and confined in regard to potential spread? Table 8.4 indicates that in many cases significant uncertainties remain and some unknowns might prevail that make the risk assessment inconclusive: the multiplex interrelations with the closer and wider environment pose a real challenge for the risk assessor. An even bigger problem is caused by the necessity of thoroughly assessing all of the offspring generation: while genetic stability over several generations might be demonstrated in the laboratory, genome × environmental interactions and introgression into heterogeneous genetic backgrounds can still trigger unpredictable next generation effects. Whatever the case, the technical characterisation of gene drive organisms or experiments carried out in the laboratory cannot be regarded as sufficient to predict all relevant effects that can emerge in the next generations, and in interaction with the receiving environments.
This could prompt a strategy to address the remaining uncertainty within environmental monitoring. However, if monitoring reveals undesirable effects, it might not be the possible to remove the organism from the environment as it was assumed to be compatible with crop plants. In this case, EU Regulation 2001/18 foresees the possible withdrawal of authorisation. This is not likely to be effective for species which (1) Can genetic stability be controlled in following generations?
Self-replication and environmental as well as epigenetic effects can lead to emergence of next generation effects not observed in the first generation Several generations should be observed under a wide range of defined environmental conditions. The outcome has to be put in context to questions (2) and (3) (2) How can genetic diversity in the target population be taken into account? In most cases a high degree of genetic diversity exists in natural populations. These heterogeneous genetic backgrounds can trigger unexpected effects not observed in the lab populations In most cases the inserted genes cannot be tested in interaction with the real genetic diversity within natural populations. For example, in insects, the strains reared in the lab might only represent a small selection of the real genetic diversity within wild populations (3) Will there be any gene flow to other species?
If gene flow is possible and hybrid offspring are viable, the resulting organisms have to be seen as new events that have to be assessed separately from the original GE organisms It might be possible to perform crosses under controlled conditions. Results have to put in context with question (1) and (2) (4) How can population dynamics and life cycle aspects of the target species be integrated?  (1) and (2) (5) Can the receiving environment be defined in regard to relevant interactions and confined in regard to potential spread?
Adverse effects can emerge from interaction with closer (associated microbiomes) or wider environments (such as food webs, predators, beneficial organisms). Terrestrial and aquatic systems have to be taken into account, as well as complex interrelations (such as signalling pathways) and behavioural aspects. Further, interrelations may vary greatly throughout the life cycle (different developmental stages such as egg, larva, pupa, adult) These aspects have to be assessed case by case and step by step. In most cases, long-term, cumulative and combinatorial effects cannot be tested or investigated ex ante are self-reproducing and self-sustaining. Therefore, monitoring might be useful but cannot be regarded as sufficient to address the problems outlined above.

Problems Emerging from Spatio-Temporal Complexity for Risk Assessment
If the spatio-temporal dimension cannot be defined, risk assessment has to consider evolutionary dimensions. The problem: evolutionary dynamics combine large numbers of individuals on the population level and singularities on the molecular scale. Thus, evolutionary processes make it possible to turn events with a low probability of ever happening into events that may feasibly happen (Breckling 2013). Under these conditions, for example, the fitness of new genomic constituents cannot be calculated in absolute terms; it will depend on the environment and future changes. Such evolutionary processes can cause major problems in regard to GE organisms regulated under EU Directive 2001/18. It has to be concluded that a sufficient and robust risk assessment of GE organisms can only be conducted if it is based on a spatio-temporal dimension that is clearly confined. It is evident that, in the context of gene drives organisms, the spatio-temporal dimension is a much more pressing concern in comparison to GE plants only grown for one season. Consequently, the environmental risk assessment of GE organisms that can persist and propagate in the environment and especially of 'gene drive organisms' will result in an increasing level of uncertainty, depending on the relevant spatio-temporal dimension. At some stage, the level of uncertainties might increase to an extent that the delicate balance between knowledge and non-knowledge is distorted allowing tipping points to be reached in risk assessment, if inherent nonknowledge increases to an extent that robust risk assessment is disabled. This problem is illustrated in Fig. 8.1.

'Spatio-Temporal Controllability' as a Cut-Off Criterion
Therefore, coming back to the question posed by Böschen (2009) about "actual evidence on which decisions about the applicability of the PP are to be taken": Is it possible to categorise non-knowledge and uncertainties in a way that decision-making can be based on sufficiently clear criteria? How can non-knowledge, uncertainties, or as EFSA (2013) puts it, "incertitude, caused by limitations of scientific knowledge and knowledge production systems" be integrated into a regulatory system of decision-making? In other words, how can we create sufficient knowledge to facilitate decision-making when faced with substantial non-knowledge? C. Then   Fig. 8.1 The balance between spatio-temporal complexity and the reliability of risk assessment

Lessons Learned from Risk Assessment of Chemicals
The spatio-temporal dimension also plays a role in the risk assessment of chemical substances. For example, Recital 76 of EU Regulation 1907/2006 (REACH) addresses the issue: "Experience at international level shows that substances with characteristics rendering them persistent, likely to bioaccumulate and toxic, or very persistent and very likely to bioaccumulate, present a very high concern, while criteria have been developed allowing the identification of such substances." Consequently, criteria to identify persistent, bio-accumulative and toxic, as well as very persistent and very bio-accumulative chemical substances, are defined in ANNEX XIII of the regulation EU Regulation 1907/2006. Further, EU Regulation 1107/2009 which concerns pesticides, integrates the criteria of POP (persistent organic pollutant), PBT (persistent, bio-accumulative, toxic) and vPvB (very persistent, very bio-accumulative) into the regulatory decision making process. These criteria function as so-called cut-off criteria: in essence, the approval process should not proceed if the substance is "POP", "PBT" or "vPvB". In this context, it is important that the chemical substances are not only assessed in regard to their toxicity but also, more generally, in regard to their "fate and behaviour in the environment" (EU Regulation 1107/2009, Annex II, 3.7.), which gives decisive weight to the spatio-temporal dimension: if a substance is regarded as very persistent and very bio-accumulative, there might still be some uncertainty or nonknowledge in regard to its actual long-term adverse effects. Nevertheless, according to EU Regulation 1107/2009, it cannot be approved.

Cut-Off Criteria in the Risk Assessment of GE Organisms
The way in which cut-off criteria were established for chemicals (including pesticides) could also be useful as a model for the risk assessment of GE organisms, especially gene drives. Similar to EU regulation of chemicals, the fate and behaviour of the organisms in the environment would be a crucial aspect. Therefore, if it were known that GE organisms could escape 'spatio-temporal controllability' by reproducing within natural populations without any effective control of spread or persistence, then the authorisation process could not proceed and the release of the GE organisms could not be allowed.
Thus, in effect, GE organisms could neither be approved nor released, including when actual long-term effects could not be determined in detail. How then can criteria be developed for the risk assessment of GE organisms that are sufficiently well defined and applicable in the approval process, as well as take into account uncertainties and limits of current knowledge? As described above in the context of chemical substances, the cut-off criteria are defined so that known characteristics of the substances are used to integrate uncertainties around actual long-term impacts into decision making.
In close analogy, the criteria applied in the risk assessment of GE organisms should be as clear and well defined as possible. Well-established scientific criteria from three areas of knowledge should be taken into consideration: (1) the (natural) biology of the target organisms (2) their (naturally) occurring interactions with the environment (biotic and abiotic) and (3) the intended technical characteristics (traits) inserted through genetic engineering. These criteria should be combined to establish an extra step in the risk assessment of GE organisms aimed at assessing 'spatio-temporal controllability'. Table 8.5 provides an overview of some relevant details that can be used to evaluate 'spatio-temporal controllability' in these three categories. For the approval process, further detailed information can be added and combined in these criteria e.g. the number of organisms to be released, specific regional biodiversity, abundance of protected species, occurrence of plant and animal pests and other relevant data, if available.

Case Studies: How to Apply 'Spatio-Temporal Controllability' in Practice
The following sections describe how 'spatio-temporal controllability' might ideally be applied in ERA. This includes two case studies for GE crops (maize and oilseed rape) and a case study on insects (olive flies, Bactrocera oleae), with and without gene drives. The case studies should be seen as mostly hypothetical; it is assumed that under real conditions more detailed data would be available.

GE Maize for Commercial Cultivation in Sweden
In the first case study, maize plants are engineered to produce a higher biomass (e.g. MON87403, see EFSA 2018) and a hypothetical application for their cultivation in Sweden has to be assessed. The outcome of the 'spatio-temporal controllability' assessment in this case (Table 8.6) is that the approval process could proceed and a full and detailed risk assessment should be conducted before a decision is taken on the safety of the crops.

GE Oilseed Rape for Commercial Cultivation in the EU
In this case, oilseed rape plants are engineered to be resistant to glyphosate (such as MON88302) and intended for release in the EU (for further references see EFSA 2014; Testbiotech 2014). The result of the 'spatio-temporal controllability' assessment (Table 8.7) is that the approval process can only proceed for releases on very limited scale, within clearly defined areas and when gene flow is prevented. Under these conditions, a full and detailed risk assessment should be conducted before a decision is taken on the safety of the crops. However, applications for releases without such control mechanisms could not proceed and would be terminated after assessment of 'spatio-temporal controllability'.

Olive Flies with RIDL for Experimental Release in Spain
In this case study, the olive flies are genetically engineered with so-called RIDLtechnology ("release of insects carrying a dominant lethal genetic system") developed Oilseed rape can persist and propagate in the environment, cross with other oilseed rape plants. Seeds can remain viable in the soil over more than ten years (seed dormancy) Kernels are taken up by wildlife species, and seeds can be transported over larger areas Although not intended by the trait, the GE plant will produce more than one generation if it is allowed to flower and produce kernels The plants can spread beyond agricultural fields, especially in rural habitats. Pollen can be distributed over several kilometres The flowering plants are an important food source for bees, thereby pollen can get transported over larger areas The insertion of the gene that renders resistance to glyphosate is assumed to unintentionally enhance fitness also in glyphosate free environment (see 2.2) Gene flow to populations of wild relative species can occur There are indications that the fitness of the plants is especially enhanced under stressful conditions (see 2.2) Further references: EFSA (2014), Testbiotech (2014) by Oxitec (Ant et al. 2012). The effects are gender-specific: male transgenic flies will mate with the native female flies and thereby introduce their artificial genes into the native populations. While the male offspring will survive, the female offspring will die at the larval stage. As a result, the natural population of olive flies will supposedly decrease (for more background see Ant et al. 2012). The outcome of the 'spatio-temporal controllability' assessment (Table 8.8) is that the approval process can only proceed if the olive flies are kept in cages and gene flow to native populations is prevented. However, sufficiently robust risk assessment could not be carried out if gene flow to natural populations were to occur. Therefore, an approval process for any release without high-level safety caging could not proceed and would be terminated. The 'spatio-temporal controllability' assessment might possibly yield other results if the field trials were located in areas where no olive flies occur naturally.

Experiments with Gene Drive in Olive Flies
In this case, it is assumed that the olive flies would be genetically engineered with a gene drive that would cause female offspring to die and leave the male offspring to survive and spread. Possible approaches are described by Champer et al. (2017).
Most criteria presented in Table 8.8 are also relevant for gene drives. However, in regard to the intended trait, the spread of the genes within natural populations would  (Daane and Johnson 2010) There are complex interactions with other species such as birds, spiders, ants, chalcid wasps and symbiotic bacteria (Bigler et al. 1986;Daane and Johnson 2010;Gonçalves et al. 2012;Neuenschwander et al. 1983;Picchi et al. 2016) Once released, the GE flies will mate in natural populations and cause the emergence of next generations without human intervention. Next generation effects might occur without being noticed Under specific conditions, such as high population densities, maximum dispersal distances for olive flies reported in literature range from 4000 to 5000 m (Economopoulos et al. 1978;Remund et al. 1976) The interrelationships include grazing, predation and symbiosis. The interrelations vary greatly throughout the fly's life history and different developmental stages (egg, larva, pupa, adult) Population dynamics and life cycle go through several stages (egg, larva, pupa, adult) and are subjected to winter seasons, creating potential bottlenecks in regional populations (Augustinos et al. 2005;Ochando and Reyes 2000) There are specific and symbiotic microbes associated with the olive flies (Ben-Yosef et al. 2014;Capuzzo 2005) The trait is unlikely to enhance fitness Molecular analyses indicate a high level of gene flow among the Mediterranean populations (Augustinos et al. 2005;Ochando and Reyes 2000;Segura et al. 2008) It can be assumed that, depending on the amount and frequency of GE flies released, they might be eliminated by natural processes after a period of time (Preu et al. 2019). However, various factors can have an impact on these processes and their actual duration cannot be determined There are other known species that can mate with olive flies, however, it is unclear whether they can produce viable offspring and enable gene flow (Schutze et al. 2013) exceed the Mendelian pattern of inheritance. Consequently, the artificial genes would spread rapidly within the natural populations and the elimination of the GE flies via natural processes is less likely, or would at least take longer compared to the case study in Table 8.8. Therefore, the outcome of the 'spatio-temporal controllability' assessment is that the approval process should only proceed if the olive flies are kept in the laboratory in regions where no native populations of olive flies occur. However, approval processes for experiments in regions where these flies occur naturally (such as the Mediterranean area) could not proceed because of lack of 'spatio-temporal controllability'. The assessment of 'spatio-temporal controllability' might come to other conclusions where the flies are kept in a laboratory with very high safety standards (for further reasoning see: Testbiotech 2018b).

The Role of the Risk Manager
In the EU, the regulatory system for GE organisms is based on a system of risk analysis set out in Regulation 178/2002: risk analysis is based on risk assessment (carried out by the EFSA) and risk management (carried out by the EU Commission and the member states). Additional regulations concern specific aspects such as environmental releases (Dir. 2001/18) and food and feed safety (Regulation 1829(Regulation /2003. In all decision making in the EU, approval process lies with the risk manager and in risk assessment policy (JRC 2008). Therefore, the EU Commission and the EU member states have to make sure that their decisions are based on a sufficiently robust risk assessment. Reliable decision making as required in EU regulation is not possible without reliable risk assessment: products can only be released or allowed on the market if they are shown to be safe (EU Directive 2001/18; EU Regulation 1829/2003). Without reliable risk assessment, no reliable decision-making as requested by EU regulations is possible (see Fig. 8.2).
In this context, the risk manager, and especially the EU Commission, can make use of their power to set adequate standards in risk assessment by establishing a robust framework for the EFSA (JRC 2008). It is interesting to note that the EU Commission adopted Regulation 503/2013 which sets the standards for assessing food and feed safety. However, no similar regulation has as yet been set by the EU Commission in regard to environmental risk assessment.
In regard to spatio-temporal control, EU Directive 2001/18 could be used as a legal basis to set the relevant standards: according to Krämer (2013), spatio-temporal control is a necessary prerequisite to enable the PP. He comes to the conclusion that "Where there is, in a concrete case, a likelihood that genetically modified plants or animals cannot be retrieved, the legal obligation to ensure that any release must be 'safe' requires the refusal to authorize such releases." (Paragraph 250) However, Krämer also shows that there are significant uncertainties in the implementation of EU regulation that require further attention and which could be ruled out by additional implementing regulations. At a certain point within such a framework, the The balance between spatio-temporal complexity and the reliability of risk management decision making EU Commission could request EFSA to assess 'spatio-temporal controllability' to deal with substantial uncertainties and non-knowledge.
As addressed in the case of the olive fly, the risk assessment of an application for laboratory uses or experimental releases of GE organisms can also become a challenge for national regulatory authorities. Thus, national legislation should foresee adequate regulatory oversight and request 'spatio-temporal controllability' to be applied as a step within the approval process.
There are further reasons why the risk manager should use 'spatio-temporal controllability' to determine relevant cut-off criteria: • Spontaneous transboundary movements: if GE organisms can spontaneously cross borders, their release can be considered to be a violation of rights under the socalled Cartagena Protocol (CBD 2000). • No possibility of coexistence: if coexistence with relevant standards for food production, such as organic agriculture, is not possible, the release of GE organisms would infringe consumers' choice and the livelihoods of organic farmers (Reeves and Phillipson 2017).

Discussion
The development of gene drive organisms and other GE organisms that can persist and self-replicate in the environment and/or can cross with natural populations create new challenges in risk assessment. As described, GE organisms inheriting a gene drive differ substantially in their characteristics compared to other GE organisms assessed by EFSA and other regulatory authorities. Existing EFSA guidance shows (EFSA 2013) that the authority is already well aware that risk assessment of GE organisms able to persist, spread and propagate in the environment will bring about new challenges. As shown, these new challenges chiefly concern next generation effects: there are many reasons why the spontaneous offspring of GE organisms might differ in their biological characteristics from one generation to the next, especially if the gene constructs are introduced into wild populations. Closely related to this challenge are questions concerning the impact of the genetic background of the target populations and interactions with abiotic and biotic stressors, as well as aspects of population dynamics and life cycle.
The basic challenge for risk assessment in this context is how regulatory decisions can be made in the face of substantial non-knowledge. To solve this problem, it is proposed to apply cut-off criteria similar to those applied in the EU regulation of chemicals. To define these cut-off criteria within the regulatory decision making on GE organisms, a new step in the risk assessment of GE organisms should be applied, i.e. 'spatio-temporal controllability'. This step is composed of three criteria: (1) the biology of the target organism, (2) its known interaction with the environment and (3) the biological characteristics of the GE organisms.
This approach uses specific 'knowns' to decide upon 'known unknowns' (such as next generation effects and genomic × environmental interactions). It is assumed, the criterion of 'spatio-temporal controllability' can inform regulatory decision making even in the light of major uncertainties emerging from the spatio-temporal dimension. This can be seen as the equivalent of cut-off criteria such as "PBT" and "vPvB" that are anchored in the EU regulation of chemical substances.
It should, however, be recognised that the assessment of 'spatio-temporal controllability' is just a step within risk assessment and not a replacement for it. In this context, it is also important to acknowledge that there are further issues, such as horizontal gene transfer, which concern the spatio-temporal control of GE organisms. Therefore, environmental risk assessment cannot and should not be reduced to the step of 'spatio-temporal controllability'. Risk assessment might well be terminated after 'spatio-temporal controllability' assessment, but if it proceeds, all other steps and criteria still have to be applied.
This approach is exemplified in case studies for the cultivation of GE maize (in Sweden) and oilseed rape (in the EU), as well as experimental releases of olive flies, with and without gene drives (in Spain). Preliminary results show that the assessment of 'spatio-temporal controllability' produces results which are meaningful and allow the application of cut-off criteria within the process of risk assessment: the authorisation process should not proceed and the release of the GE organism should not be allowed if it is known that the GE organisms are able to escape 'spatio-temporal controllability' due to propagation in natural populations, with no effective control of spread or way of preventing persistence in the environment.
The schematic and partially hypothetical cases as presented lack some data and information that under real conditions would be included in the pending application dossier. Further, in practice the results would depend on a shifting baseline of information and might therefore differ from future results. The approach as proposed can be considered flexible enough to be improved by adding further criteria and new data. It can be applied very generally to applications for environmental releases, no matter if these concern experimental field trials or commercial cultivation.
As far as the role of the risk manager in the EU is concerned, it should be acknowledged that applications for releases of GE organisms that lack spatio-temporal control can already be rejected, even without an additional and specific step in risk assessment as suggested. However, especially in the light of the PP, the application of cut-off criteria within the process of risk assessment has some significant, important and convincing advantages: it can provide more clarity, transparency and also more reliability in final decision making. Further, it can save on resources, since this additional step in risk assessment would influence the approval process at an early stage.
Finally, these criteria can be used to inform upstream processes and thereby generate more clarity and certainty at an early stage of research and development. Many researchers currently developing gene drive applications are already aware of the problem of spatio-temporal complexity (see, for example, Noble et al. 2017). At present, several projects are looking to develop gene drives that can be refined to specific regions or defined periods of time (see for example Min et al. 2017). It is assumed, there is no general obstacle to the future application of 'spatio-temporal controllability' assessment for these developments, and therefore meaningful results could be expected.

Conclusions
New challenges arise with applications such as 'gene drive' that can be introduced into natural populations where they can propagate and spread further. Due to the complexity of the biology of these organisms and their interactions with the environment, increasing uncertainty and areas of non-knowledge have to be taken into account.
It has been shown that risk assessment of intended environmental releases of GE organisms linked to self-propagation of artificial genetic elements over several generations will suffer from major uncertainties and unknowns, emerging in most cases from next generation effects. It can be assumed that a tipping point can emerge at a certain point in the dissolution of spatio-temporal boundaries where it becomes necessary to apply cut-off criteria and stop the approval process. This means that risk assessors and risk managers face the problem of how to come to robust conclusions and reliable decisions within the approval process that also give substantial weight to the PP.
It is proposed to introduce cut-off criteria, based on a specific step of 'spatiotemporal controllability' within risk assessment. This new step combines three criteria: (1) the biology of the target organisms, (2) their naturally occurring interactions with the environment (biotic and abiotic), (3) the intended biological characteristics (trait) of the GE organisms.
The combination of these three criteria in one specific, additional step in risk assessment has the advantage of them already being used to some extent in current EFSA risk assessment; many of the details to assess these criteria are also very well known. If it is known that GE organisms can escape 'spatio-temporal controllability' because they can propagate within natural populations with no effective control of spread or persistence, then the authorisation process cannot proceed and the release of the GE organism cannot be allowed. This concept can be used to delineate some of the boundaries between known and unknowns considered to be crucial. Further, it can help to develop an adequate regime for risk assessment which overcomes problems with the so-called comparative approach (EFSA 2010). This will foster the robustness of risk assessment and can substantially benefit the reliability of decision making within approval processes.
Open Access This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.
The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.