Steps Towards a Precautionary Risk Governance of SPAGE Technologies Including Gene-Drives

In view of the rapid dynamics of genetic engineering development (in particular regarding the ‘new gene-technologies’ gene editing, self-propagating artificial genetic elements (SPAGE) and synthetic biology), the question is being intensively discussed whether the currently practiced risk governance is sufficient to guarantee the desired high level of health, consumer and environmental safety. Especially the extreme spatial and temporal environmental exposure due to released SPAGEs is identified as a problem because it leads to an enormous expansion of ignorance about possible situations and interactions. Therefore it requires measures based on the precautionary principle. Workable ways to integrate the precautionary principle into environmental risk assessment and risk regulation are being developed.

risk governance of new gene-technologies. 3 Few exceptions to this are activities in Switzerland and Austria (cf. Ammann et al. 2007;Eckerstorfer et al. 2010). Possibilities for a stronger integration of the precautionary principle into the governance of new technologies are the subject of this text.
The precautionary principle plays an important role in international, European and national regulation. 4 Its interpretation is controversial, little operationalized and the subject of intense discourse. 5 The precautionary principle legitimizes precautionary measures especially when it would be irresponsible to wait until a risk can really be proven. This waiting would be particularly irresponsible if serious and/or farreaching hazards with tendencies towards irreversible exposures and adverse effects were involved, after the occurrence of which corrective action cannot be taken.
At least in Europe, the discussion has revealed the following combination of prerequisites for precautionary measures with regard to technological innovations 6 : (a) Lack of knowledge (reaching from uncertainty to ignorance) (b) Comprehensible reasons for concern (indications pointing to particularly powerful, irreversible and far-reaching technological effects or to particularly serious consequences affecting irreplaceable values, particularly vulnerable population groups or ecosystems) (c) A rudimentary cost-benefit analysis (in which at least the expected costs of precautionary measures are compared with the expected costs of inaction, or in which, for example, medical applications for which few or no alternatives exist are given more weight than applications in the food chain in which numerous alternatives exist) (d) The availability of adequate and proportionate measures (besides risk communication and participation ranging from labelling, certifications, accreditations and codes of conduct through containment or moratorium up to substitution by less problematic alternatives).
Although the precautionary principle is well represented in the political and legal bases for action at international and European level, its anchoring and operationalization in the technology-related regulations must so far be described as rather rudimentary. Fisher et al. write: "In particular, the messy business of integrating the principle into existing institutions and relating it to well-established decision-making processes has not received the attention it should have" (2006, p. 1). The integration of the precautionary principle into the governance of self-propagating artificial genetic 3 In the publication of the National Academies of Sciences (2016), the current need for policy reform is at least addressed as an opportunity to broaden the view: "The novelty of this technology also provides an opportunity to reflect more generally on the principles governing scientific research and suggest areas for improvement" p. 137. 4 (2002), Fisher et al. (2006), Stirling (2016), European Commission (2017. 6 Commission of the European Communities (2000), Renn et al. (2003), von Schomberg (2006), Stirling et al. (2006), Amman et al. (2007), Stirling (2016), Persson (2017). In the recent past, scientific and public debates relating to the safety of genetic engineering processes and products have increasingly focused on applications in the food chain. As a result, the European Food Safety Authority (EFSA) and its 'Panel on Genetically Modified Organisms' became particularly important. 8 Genetic engineering governance thus increasingly focused on the relatively late innovation phase of product approval. However, since gene drives have so far mainly been in the research and development phase, a "governance of science and technology" is required for these new gene-technologies (National Academies of Sciences 2016, p. 138ff). This must go far beyond the existing regulations on laboratory or facility safety and deliberate release. 9 Efforts to anchor the precautionary principle more firmly in the governance of new genetic technologies focus on risk assessment procedures and the interface between risk assessment and risk management. The necessary reforms are dealing with the further development of existing methods of environmental risk assessment (ERA) towards a precautionary hazard and exposure assessment that also takes appropriate 7 The European Commission's communication on the applicability of the precautionary principle emphasizes that cost-benefit analysis must not be a matter of purely economic cost-benefit considerations, but also of "the efficiency of possible options and their public acceptance" (p. 5). 8 Between 2003 and 2019 there were 367 EFSA publications on genetically modified organisms https://www.efsa.europa.eu/de/publications/?f%5B0%5D=im_field_subject%3A61906, last accessed 20.02.2019. 9 Cf. German Genetic Engineering Act with its differentiation of genetic engineering facilities for research and commercial purposes, the introduction of safety levels and the involvement of the Central Commission for Biological Safety (ZKBS), as well as the approval of genetic engineering facilities (last amendment 2017) including the Genetic Engineering Procedure Ordinance (GenTVfV) and the Genetic Engineering Protection Ordinance (GenTSV) (last amendment 2015), then the EU Directive on the contained use of genetically modified microorganisms (Directive 90/219/EEC, now Directive 2009/41/EEC), the EU Directive on the protection of workers from risks related to exposure to biological agents at work (Directive 90/679/EEC, now Directive 2000/54/EEC) and the EU Directive on the deliberate release into the environment of genetically modified organisms (Directive 90/220/EEC, now Directive 2001/18/EEC). account of various forms of lacking knowledge, and with their more stringent linkage with precautionary risk management with the help of the construct of 'reasons for concern'. In addition, the design and establishment of systematic and orderly procedures for enforcement are necessary (administrative regulations). These should also include participation opportunities for the public and civil society actors (e.g. public hearings). Finally, those institutions that are responsible for these procedures, in which information is collected, evaluation criteria are sharpened and, to a certain extent, weighing processes are carried out, must prepare themselves for a corresponding reorientation towards a precautionary risk and exposure assessment and evaluation that starts early in the innovation process. 10 Central to the implementation of the precautionary principle within the framework of risk assessment and its interface with risk management is the identification of those 'reasons for high concern' which are capable of triggering precautionary measures. If not only the probabilities of occurrence are unknown, but also the contours of possible threat scenarios are unclear, then it is a matter of scientifically comprehensible information indicating that particularly severe and/or far-reaching consequences must be reckoned with. The EU Commission's Communication on the applicability of the precautionary principle speaks of "reasonable grounds for concern" in this respect (Commission of the European Communities 2000, p. 3; cf. also von Schomberg 2006, p. 19). In another part of the Communication 'sufficient' or 'reasonable' grounds for concern are mentioned (pp. 11 and 31). The precautionary principle is applicable in "those specific circumstances where scientific evidence is insufficient, inconclusive or uncertain and there are indications through preliminary objective scientific evaluation that there are reasonable grounds for concern that the potentially dangerous effects on the environment, human, animal or plant health may be inconsistent with the chosen level of protection" (p. 9f). The elaboration and establishment of methods and criteria for determining such reasons for concerns against the background of uncertainty and ignorance are therefore of central importance. Thus, despite incomplete knowledge, reasons for concern that could trigger precautionary measures must indicate the possibility, severity and range of adverse effects and thus their incompatibility with the desired level of environmental and health protection. With the help of defined methods and criteria, it must be possible to clarify which reasons for concern can be considered as triggers for precautionary measures, what significance should be assigned to these 'reasons for concern' in each case, and what consequences should be drawn from them in precautionary risk management.
For the identification of reasons of concern, two perspectives are essential: on the one hand, the focus on the entity that affects the systems concerned, the agent, the 10 In Germany, the Federal Institute for Risk Assessment (BfR) and the Central Commission for Biological Safety (ZKBS) should be mentioned, as should the Federal Office of Consumer Protection and Food Safety (BVL), the Federal Environment Agency (UBA) and the Federal Agency for Nature Conservation (BfN). At European level, the European Food Safety Authority (EFSA) and the European Environment Agency (EEA) deserve special mention. Internationally, the Conference of the Parties to the Convention on Biological Diversity with its Secretariat and its Subsidiary Body on Scientific, Technological and Technical Advice (SBSTTA) is of great importance. technology, the intervention. This is done with the method of technology characterization using the criteria intensity and depth of intervention, resulting in technological power and range, and reliability. Then the view turns to the systems that may be affected. Their vulnerability and social criticality are investigated using the method of vulnerability analysis, with the focus on identifying highly vulnerable entities, weak points as well as tipping points as sources of surprise.
SPAGE technologies are currently still at a very early stage of innovation. Thus, the exact application objectives and application contexts or possibly affected systems are still largely unknown. Additionally, to the technology (the agent) these objectives and contexts have to be investigated as further sources of hazards, exposures and risks. The precaution-oriented prospective technology assessment in the phase of science and technology development therefore first concentrates on what can be already known, on the SPAGE technologies currently under development. The search for reasons for concern is therefore initially carried out within the framework of a technical characterisation and as soon as application objectives, contexts and systems are known to some extent, in the form of a vulnerability analysis of the potentially affected systems.

Dealing with Non-knowledge: Precautionary, Prospective Technology Assessment Versus Environmental Risk Assessment
Currently, the debate on health and environmental risk assessment of planned releases of genetically modified organisms (GMOs) focuses on the authorization of food processes and products, with EFSA and its 'Panel on Genetically Modified Organisms' as key actors. They act in a comparatively late stage of innovation. A precautionary prospective technology assessment must start much earlier, already in the research process and during the development of technologies. 11 This has the advantage that necessary changes of direction and switching to lower-risk development paths can be carried out comparatively easily as long as path dependencies have not been established by far-reaching investments. However, this also has the already mentioned disadvantage that hazards and exposures resulting from application goals cannot yet be adequately considered.
The methods for Environmental Risk Assessment (ERA) 12 of genetically modified organisms (GMOs), which was first described in the EU Directive 2001/18/EC and 11 "Each phase of research activity -from developing a research plan to post-release surveillanceraises different levels of concern depending on the organism being modified and the type of gene drive being developed", determines the National Academies of Sciences (2016, p. 7). 12 The National Academies of Sciences (2016) define the Environmental Risk Assessment as follows: "the study and use of probabilistic decision-making tools to evaluate the likely benefits and potential harms of a proposed activity on the well-being of humans and the environment, often under conditions of uncertainty" p. 105. later in the EFSA documents (cf. in particular EU Directive 2001/18/EC, EFSA 2006, 2013a, unfortunately does not take into account the fundamentally different views of the agent on the one hand (technology characterisation) and the affected systems on the other (vulnerability analysis). Anyhow, in its methodological instructions ERA takes into account many of the important aspects from both points of view in a loose order. In their technology-related analytical steps, some of which are explicitly referred to as 'characterization', there are numerous overlaps with a precautionary, prospective technology characterization. However, there is a gap that cannot be easily bridged concerning the understanding of and dealing with uncertainties and lack of knowledge. Additionally, on the system-related side of the ERA, with regard to the analysis of affected systems and possible impacts in these, on the one hand points of reference can be identified and on the other hand further developments are necessary. Last but not least, a clear structure regarding the determination of vulnerabilities would approve the practiced ERA.
We distinguish two different types of vulnerability analysis an event-related impact oriented and a structure-related vulnerability analysis as explained in Chap. 4. The event-related vulnerability analysis (eVA) differentiates according to the (a) Disturbing event/agent (b) Exposure to the agent (c) Sensitivity of the system to the agent (d) Adaptive capacity of the system, its ability to process disturbance events (e.g. immune system). 13 The vulnerability of the affected (eco)system to the agent (the SPAGES) then results from the integrated consideration of all four aspects. In fact, also the ERA pays great attention to exposure and sensitivity. The 'Guidance to develop specific protection goals options for environmental risk assessment at EFSA, in relation to biodiversity and ecosystem services' additionally addresses possible tipping points, the complexity and resilience of ecosystems, which are in the focus of the structurerelated vulnerability analysis (EFSA 2006). However, this second approach, the structural vulnerability analysis (sVA), is neglected due to ERA's mainly eventrelated approach. 14 The sVA concentrates on weak points in the affected systems independently of possible disturbance events, which can be both punctual and creepingly continuous. It analyses which elements and relations of the system may draw back when it comes under pressure. 15 They all are about evidence for the presence of particularly sensitive or particularly critical system elements and relations or the presence of particularly unstable or pre-tensed system states. 16 This subheading also includes the discussion about possible contamination of (particularly valuable?) ecosystems by GMOs. Can or must the transfer of modified genes to wild forms or the mere presence of GMOs in certain ecosystems already be regarded as a reason for concern? 17 In case of interventions into particularly sensitive, pre-tensed 18 or predamaged systems there are good reasons to expect that this will have far-reaching consequences. In such situations the implementation of the precautionary principle is recommended, and the corresponding indications of 'particularly alarming system conditions' are among the comprehensible and valid triggers (reasons for concern) for measures according to the precautionary principle. The sVA and the eVA are thus complementary and their approach is significantly more precise than that of the ERA due to the differentiated consideration of disturbance events, exposure, sensitivity and capability to adapt (eVA) as well as the analysis of internal weak points in the affected system (sVA). The technical characterization concentrates on the agent, the sVA concentrates on the affected systems and the eVA focuses on the interactions between the two.
The sVA plays a special role within the framework of a precautionary, prospective technology assessment and especially as a starting point for designing resilient systems. It is a method that opens options to minimize weak points, to increase the resilience of the systems concerned, to prepare them for possible surprises, for events and mechanisms of action, which are not yet known. This opens up a second way to practically reduce the realm of possibilities and thus the extent of ignorance regarding possible far-reaching disruptive events. The first strategy applies to the agent and focuses on substitution, on less depth of intervention, less powerful and far-reaching technologies, which reduce the range of possibilities of disruptive events. This was already elaborated in Chap. 7 and will be picked up again later on. The second strategy tries to reduce the realm of possibilities by strengthening the adaptive capacity and resilience of the systems concerned. Resilient systems are able to successfully cope with a wide range of even unknown disruptive events. This presupposes, however, that one is-not least also technically-in a position to relate precautionary risk management not only to the intervening technologies, but also to the affected Analysis (FTA) (Böhnert 1992;Thums 2004) or regarding complex organisations e.g. bank stress tests (cf. e.g. Quagliariello 2009). 16 Criticality refers to areas that are particularly important for social life or survival, e.g. nutrition, health, medicine. Sensitivity refers to particularly sensitive areas, phases or subgroups, e.g. pregnancy, previous damage or tipping points. 17 The report of the National Academies of Sciences (2016) states: "The mere presence of the modified genetic element in other species could be considered an endpoint" p. 110 and further: "Because the goal of a gene-drive modified organism is to spread, and possibly persists, in the environment, the necessary ecological risk assessment is more similar to that used for invasive species, than for environmental assessment of genetically engineered organisms" p. 110, see also Landis (2004). 18 In his presentation of the 'Adaptive Cycle' Holling gives an example of a pre-tensed system. A forest may have accumulated a great deal of energy in the form of wood during its conservation phase and is then in danger to suddenly burn down in case of ignition (release), cf. Holling (1986). systems (their vulnerability, adaptive capacity or resilience). 19 Such a 'constructive precautionary approach', as Hansen et al. (2007, p. 400) call it, is totally out of reach of the strictly event-related ERA and the risk management based on it.
Conventional risk assessments focus on risks and not on precautions. According to the ERA-related understanding of risk, they aim at quantifiable statements in which risk is defined as the product of the amount of loss and the probability of occurrence. Risk analysis is then dependent on immense amounts of data. Though the deficiency of knowledge is omnipresent. But the latter is only understood as a deficiency that can be overcome by more research. There is no room for reflection on the fundamental limits of knowledge, e.g. the impossibility of predicting the effects of interventions in complex systems due to non-linearity, tipping points or emergence. There is certainly no room for reflections about ignorance, cluelessness and complete surprises, 20 about a non-knowledge of which we don't even know that we don't know (unknown unknowns Wynne 1992;Kerwin 2016).
In fact, the ERA also incorporates approaches that relate to minimizing nonknowledge. The ERA practitioners, especially at EFSA, are well aware that the possibilities for risk assessment based on predictions are clearly limited. In order to bridge the lack of knowledge, ERA uses a number of auxiliary constructions to get data after all (cf. e.g. EFSA 2010). Most important are the so-called 'concept of familiarity', the assessment of risks of GMOs by estimating the partially better known risks based on the donor organism, the recipient organism and the vector, and the 'comparative approach', the use of risk assessment with reference to already naturally occurring comparable organisms (non-GM-surrogates). The fact that ERA additionally makes use of a number of methodological steps of 'societal learning' shows that surprises are at least implicitly expected (cf. von Knies and Winter 2011, p. 8). To be mentioned here are the step-by-step procedure on the way towards release and the approach of post-market environmental monitoring (PMEM) as well as general surveillance (GS). All this, however, has the fatal disadvantage that learning may come too late and there may be no chance for corrective action. 19 Stirling et al. (2006) speak of a "Resilience-focused (risk absorbing) Management Style" in this respect: "Improving capability to cope with surprises: diversity of means to accomplish desired benefits, avoiding high vulnerability, allowing for flexible responses, preparedness for adaptation" (p. 302). Strategies of an 'adaptive management' of socio-ecological systems, as they were developed after Holling (1978), Shea et al. (2002), Stankey et al. (2005), can also be considered. Specific references to design principles for increasing the resilience of socio-technical systems can be found in von Gleich and Giese (2019). 20 Having in mind causes for surprises Stirling et al. (2006) mention "substantive novelty" or "unprecedented characteristics" (p. 294). Brooks (1986) developed a typology of surprises regarding the interaction between technology and society. He suggests three types of surprises: (1) unexpected discrete events, (2) discontinuities in long-term trends, and (3) emergence und sudden public awareness of new information. Filbee-Dexter et al. (2017) gave an interesting overview of 'ecological surprises' mostly combined with tipping points, phase transitions and feedback mechanisms (e.g. unanticipated behaviour or regime shift in ecological systems as collapse of coastal fisheries, state change during eutrophication of lakes, outbreaks of insects or species invasions). Close to the problem of total surprises are extremely improbable events, which Taleb (2007)  And all this deficiencies of the ERA approach are not adequately communicated at any point. This leaves the impression of an indefinitely optimistic quantitative 'risk assessment', which is aware only of the not-yet-known, which gets by completely without precautionary approaches and ignores essential references to possible surprises, to the remaining lack of knowledge, to unpredictability and ignorance (the unknown unknowns) (cf. Wynne 1992;Wehling 2009). If EFSAs guidance documents address uncertainties 21 and, to some extent, the limits of knowledge, then it is only in relation to the limits set on risk research by scarce resources (money and time). Statements about the limits of predictability in dealing with complex ecosystems-i.e. aspects of not being able to make predictions-are suppressed, as can be shown by two processes. 22 The 'Scientific Opinion on Guidance on the risk assessment of genetically modified microorganisms and their derived food and feed products' (n.d.) states: "Predicting impacts of GMMs and derived food or feed on complex ecosystems can be difficult due to continuous flux and spatial heterogeneities in ecosystems creating a myriad of potential microbial habitats in which interactions between GMMs and their products with the indigenous organisms and or abiotic components can take place. It is recognized that an environmental risk assessment cannot provide data of a GMM or their products which would cover all potential environmental habitats and conditions. Consideration of environmental impact (damage) should therefore focus on environments in which exposure is most likely or in which, when relevant, viable GMMs could potentially proliferate" (EFSA n. d., p. 38). These formulations are missing in the Guidance Document of 2006 on the same topic, which was finally published. Instead it reads: "Predicting impacts of GMMs and derived food or feed on complex ecosystems that are continually in flux is difficult and largely based on experiences with other introductions and an understanding of the robustness of ecosystems. It is recognized that an environmental risk assessment is limited by the nature, scale and location of experimental 21 The most comprehensive and clearest summary is the "Generic list of common types of uncertainty affecting scientific assessments" EFSA (2018a, p. 19). 22 The background of this attitude may be that lack of knowledge and ignorance are often put forward as reasons for the need to apply the precautionary principle. Whether the mere indication that complex systems are being interfered with and that not all possible reactions of these systems have yet been known and researched can suffice as a trigger for far-reaching precautionary measures is currently the subject of controversial debate. However, such an approach is already addressed in the Rio Declaration as a possibility: "In the case of measures relating to complex systems that have not yet been fully understood and for which the consequences of disruptions cannot yet be predicted, the precautionary approach could serve as a starting point" (Chapter 35, Paragraph 3 of Agenda 21). If one wants to justify the necessity of far-reaching precautionary measures, however, one should not stop to point out a lack of knowledge. Comprehensible knowledge about reasons for concern is necessary. Identifyable non-linearity, feedback mechanisms, bifurcations and tipping points, as well as pre-tensed and pre-damaged systems are reasons for concern recognizable through vulnerability analysis comprising the use of models, and they are also dependent on identifiable system architectures and system states. What is not comprehensible, however, is an argumentation in which the complexity paradigm is first relied upon to justify a lack of knowledge, and in which the argument suddenly jumps back into the reductionist paradigm and demands complete controllability. If one uses the complexity paradigm argumentatively, a demand for complete controllability does not make sense. releases, which environments have been studied and the length of time the studies were conducted" (EFSA 2006, p. 59). In the preliminary version, the limits were still mentioned which, in principle, are set for predicting the consequences of interventions in complex systems. In the version finally adopted, on the other hand, EFSA returns to the position previously published several times, according to which the project is limited above all by the research expenditure.
This course of action is also repeated in the two-stage process of the development of the EFSA 'Guidance on the environmental risk assessment of genetically modified animals'. The precedent 'Scientific Opinion' mentions uncertainties due to assumptions and extrapolations, conflicting scientific literature and perspectives, and specified uncertainties, the latter divided into linguistic uncertainty (lack of linguistic precision), variability (in the subject area), and uncertainty caused by limitations of scientific knowledge and knowledge production such as motivational and systematic bias, censoring, measurement error, missing data, lack of suitable comparators or surrogates, and other causes of incomplete awareness, understanding and descriptions of a mechanism, process or system (i.e. model and scenario uncertainty)" (EFSA 2013a, p. 41f). Subsequently, methods for the reproducible identification and handling of these uncertainties are discussed. The knowledge problems arising from long-term exposures are addressed and reference is made to the auxiliary construction of the familiarity concept (EFSA 2013b, pp. 38f, 41, 163f). The following conclusion is then drawn: "ERA is often constrained/restricted by the available knowledge and experience of the GM animal and it can be difficult to predict and consider all potential future applications, production systems and receiving environments of the GM animal. Thus large-scale and long-term use of a GM animal could result in some effects which were not predictable at the time of the ERA or consent. Therefore, according to Directive 2001/18/EC, applicants are required to conduct general surveillance (GS) to detect unanticipated adverse effects on the environment" (EFSA 2013a, p. 44). Interestingly here too, another text appears in the later adopted and published version: "Overall, the results of the ERA will be subject to varying levels of uncertainty associated with factors such as (1) the availability of data and use of non-GM surrogates to inform the ERA, (2) the range of receiving environments in the EU where the GM animals are likely to be intentionally or accidentally released and (3) the diversity of management practices across EU regions. As far as possible, the overall conclusions of the ERA should specify under which conditions (e.g. receiving environments, management practices of the placing on the market, release and production) the risks/uncertainties identified are most likely to occur and clearly identify the factors/processes which might affect the conclusions of the ERA in order to make explicit the robustness of the conclusions of the ERA (EFSA 2013b, p. 30)". Here, too, the concession of limits of predictability is withdrawn in favour of a diversity that cannot be coped with in terms of effort and resources. Unknown unknowns are not mentioned. At least, an extreme extension of the spatio-temporal range of GMOs is addressed as an insight-limiting aspect, but as a way out, reference is again only made to ex-post observation via Post Market Environmental Monitoring and General Surveillance (EFSA 2013b, p. 163), which only takes effect when 'the child has, so to speak, already fallen into the well'. The biggest deficits of the ERA therefore exist in dealing with special forms of non-knowledge (non-determinacy, unknown unknowns and possible surprises).
This did not really change when EFSA started a comprehensive process to address uncertainty from 2014 onwards. In the two documents representing the preliminary outcome of this process possible surprises or unknown unknowns are not addressed, not even the difficulties of predicting the consequences of interventions in complex systems (see EFSA 2018a, b). The term complexity appears only once, but related to the complexity of the assessment process. Non-determinacy, non-linearity and tipping points do not occur as terms. 23 Unknown unknowns are mentioned, but EFSA simply excludes them from the 'Uncertainty Analysis'. "It is important to note that overall uncertainty cannot and does not include any information about unknown unknowns, i.e. uncertainties not known to the assessors. Since these are unknown, they cannot be either quantified or described" (EFSA 2018a, p. 34). EFSA acts according to the motto 'you can't say anything about non-knowledge, because it's non-knowledge'. Unknown unknowns may exist, but we're not in charge. It is not the risk assessors who are responsible for dealing with unknown unknowns, but only the risk managers. 24 "Decision-makers should understand that all assessments are conditional on the current state of scientific knowledge, and do not take account of 'unknown unknowns', and take this into account in decision-making (e.g. they might treat novel issues differently from those with a long history of scientific research)" (EFSAa, p. 35). Such statements do not, however, prevent EFSA from repeatedly stressing that risk assessors should identify all relevant sources of uncertainty and that the outcome of the risk assessment naturally includes statements about how problematic or harmless (no concern) certain possible developments are (EFSA 2018b, p. 34). Which, of course, must be related to the level of protection targeted in the EU (von Schomberg 2006, p. 25). A risk assessment is not possible without values. In addition, risk managers are also dependent on scientific preparatory work for a cost-benefit assessment. 23 This contrasts with the 'Scoping paper' of the Group of Chief Scientific Advisors of the EU Commission. They distinguish between 'scientific uncertainty', 'indeterminacy' and 'ignorance' (2018, p. 3), which corresponds to our distinctions. 24 "Deciding how much certainty is required or, equivalently, what level of uncertainty would warrant precautionary action, is the responsibility of decision-makers, not assessors" (EFSA 2018a, p. 16). Contrary to this, Sterling et al.: "It is clear that the precautionary principle is of relevance not only to the management, but also to the assessment of risk" (2006, p. 289). The EFSA statement also reveals a serious misunderstanding about the functioning of the precautionary principle. It is assumed that the level of uncertainty could be a trigger for precautionary measures. Rather, it is true that in essence the scale of a potential threat triggers precautionary measures, despite remaining uncertainties about the probability of occurrence and the precise outcome of the threat.

Depth of Intervention
Scientific uncertainty, not-yet-knowledge (in the sense of lacking research results), unpredictability (due to non-determinacy in complex systems) and ignorance (regarding complete surprises) are the previously mentioned forms of non-knowledge. In the Environmental Risk Assessment especially unpredictability and ignorance are not considered adequately or not at all. Now another important form of non-knowledge has to be mentioned: The technically generated non-knowledge. 25 It is generated by spatio-temporal delimited entities characterized by an expanded half-life, persistence, mobility, chain reactions or the ability of self-proliferation. The magnitude of this form of cluelessness about possible consequences and surprises comes about through the practical extension of the effectiveness and scope of the results of technological interventions, of their technical power and range in space and time. This enormously expands the realm of possibilities for unexpected events. Seen precisely, the technically generated expansion of the realm of ignorance is owed to the enabling of a not overseeable and incalculable magnitude of possible interactions of spatiotemporal delimited entities in countless system contexts. 26 Well known examples are unmanageable chain reactions in the field of nuclear and chemical technologies on the side of power and on the side of exposure extreme half-lives of substances in the environment (persistence or radioactivity) or GMOs (modified genetic elements), which are able to multiply and spread on their own. If a genetically modified organism is able to reproduce itself and is also mobile, there is a threat that it will tend to spread globally, not retrievably and irreversibly. Due to its expanded spatial and temporal range, this organism can emerge in an infinite number of diverse systems and contexts and enter into completely unpredictable and surprising interactions. The best known example of such a surprise from the realm of chemicals is the ozone-depleting effect of CFCs in the stratosphere, i.e. at an altitude of more than 10,000 m. The magnitude of ignorance about the possible consequences of such technological interventions will thus increase in proportion to their technical power and range. From this point of view, the fact that gene drives are explicitly produced for the purpose of rapidly spreading in populations must be classified as particularly concerning. Since this expansion of ignorance was created technically, however, it can also be reduced technically. An extreme spread can be technically reduced by switching to self-limiting reactions or to less invasive, less persistent, less mobile and reproductive GMOs. The focus of such a 'constructive precautionary approach' is on substitution is on changing the character of the agent (see Chap. 7). This marks a clear difference from the widespread practice of reducing exposure to an unchanged agent by containment. 27 Anyhow, both of these strategies aim at reducing the range of possibilities.
The technical roots of the expansion of the realm of possibility and the associated magnitude of non-knowledge can be identified within the framework of technology characterization. This is done not least with the help of the technical evaluation criterion "depth of intervention". This criterion was introduced at the end of the 1980s (cf. von Gleich 1989), building on considerations by Günther Anders and Hans Jonas (cf. Anders 1958;Jonas 1979Jonas , 1985. It refers to technologies based on the mathematicalexperimental natural sciences, for which a distinction between the level of phenomena and the level of laws of nature behind the phenomena is constitutive. Within this paradigm the laws of nature produce the natural phenomena and control them to a large extent. The atoms (or elementary particles) largely determine the physical properties of the physical objects, the molecular structures control the chemical properties of substances and also the genes are decisively involved in the control of the biological properties of organisms. Technologies that technically address (manipulate) such control structures as atoms/elementary particles, molecular structures or genes generate a significantly higher power over phenomena and more far-reaching consequences in comparison to techniques that only address (manipulate) directly perceptible phenomena, as has been the case for millennia with e.g. artisanal agricultural technology (e.g. in the form of breeding by selection). Breeding in which a greater variability is produced by irradiation or chemicals must therefore also be regarded as a deep intervention. A greater depth of intervention by addressing such 'control structures' can thus be identified as the basis for a broader technical power and range in space and time. 28 The basis of this approach to technology characterization is thus the insight that the magnitude of ignorance about possible consequences of deep interventions is not 'simply already there', but rather is generated and enormously expanded by the character (the depth) of the technical intervention. 29 The use of particularly powerful and far-reaching technologies increases the scope of what can happen and thus also the ignorance regarding possible consequences. Conversely, this range of possibilities and the combined extent of cluelessness regarding possible consequences can also be reduced by the use of techniques with a lower depth of 27 This difference is also addressed as intrinsic biocontainment versus extrinsic and technical containment cf. European Science Foundation (ESF 2012). 28 The criterion depth of intervention was successfully used within the framework of prospective technology assessment in areas such as synthetic chemistry (Böschen et al. 2003), nanotechnology (Rip 2006) and synthetic biology (Grunwald 2016). Related conceptualizations of the criterion depth of intervention in the biological field can be found in Deutscher Ethikrat (2011) as well as in Engelhardt et al. (2016). 29 Wynne (2005) formulates this regarding the technical intervention at the gene level as follows: "The very idea of intervening in nature at the utterly novel genetic level, despite being championed both as a way of increasing knowledge, and as a new dawn of precision-biotechnology, introduces and releases onto society previously unencountered (and hitherto irrelevant) elements of the unknown, thus augmenting unpredictability and potential lack of control. Scientific research may not only diminish ignorance, but also thus amplify it too." p. 69.
intervention. If, as part of substitution efforts, we can select or develop agents that are degraded or die after a few hours or days, this will significantly reduce the extent of ignorance about possible consequences (see Chap. 7).

Identification of Substances of Very High Concern in REACH
Such considerations about technically generated non-knowledge are not new. They have already played an important role in the environmental policy debate on persistent industrial chemicals and in particular on stratospheric ozone depletion as a result of CFC release (cf. Scheringer 1996). And finally, based not least on the debate about persistent organic pollutants (POPs) in the oceans, they have also found their way into the European chemicals regulation under REACH, which classifies very persistent and bioaccumulative substances as 'substances of very high concern'. Until then, the debate on possible triggers for precautionary measures had focused exclusively on environmental and health risks. Particularly widespread exposure to agents has not been accepted as a cause of concern and for precautionary measures so far. This changed with the REACH chemicals legislation. Article 1 of REACH states in point 3: "This Regulation is based on the principle that manufacturers, importers and downstream users must ensure that they manufacture, place on the market and use substances which do not adversely affect human health or the environment. Its provisions are based on the precautionary principle" (REACH Regulation 2006, p. 28).
The design of REACH then really endeavors to operationalize the precautionary principle and explicitly deals with the question of how precautionary measures are to be triggered. Point 69 of the list of recital grounds for the adoption of the Regulation states: "In order to ensure a sufficiently high level of protection of human health, including affected populations and, where appropriate, of certain vulnerable subgroups, and of the environment, substances of very high concern should be treated with great care in accordance with the precautionary principle" (p. 14). The concept of 'substances of very high concern' must be emphasized in this formulation. According to REACH, 'substances of very high concern' are not only substances with a high hazard potential (in particular carcinogenic, mutagenic or toxic for reproduction), but also substances with an especially high exposure potential, namely particularly very persistent and bioaccumulative substances, even irrespective of any associated hazard hypothesis. This is the first important step that should be followed when implementing the precautionary principle in the governance and regulation of GMOs. The basis for this step towards a practically 'hazard-independent operationalization of the precautionary principle' is the acceptance that an enormous expansion of exposure expands the scope for unexpected interactions in the environment to the extreme and thus also the ignorance of possible surprises. Interestingly, this logical consideration can also be interpreted as a special form of the hazard hypothesis. 30 Within REACH the characterization of a substance as being of very high concern has a precise and direct impact on risk management. This is the second important step that should be taken in the operationalization of the precautionary principle in the area of the GMO. The characterization of a substance as being 'of very high concern' leads to a general ban on its use (Art. 56) and to an explicit obligation to obtain authorization. The substance will be included in Annex XIV of the REACH Regulation. An authorization is only possible if it can be shown that the risk posed by this substance can be controlled throughout its entire life cycle. For substances for which no threshold value can be given, 31 exemptions are possible on the basis of a balance between benefit and risk potentials. 32 In addition, even after an exceptional authorization, still exists the minimization requirement, an obligation to search for less problematic alternatives (substitution), and an obligation to monitor the fate of the substance in the environment (Article 60 (10)). For the governance of chemicals, therefore, a comparatively easy way has been found to integrate the precautionary principle into the technology-related regulation following these two steps.
The particular practicability of this procedure lies in the fact that it first focuses on the properties of the agents. This has the advantage that one does not depend yet on precise toxicological knowledge and on the particularly complex vulnerability analysis of certain target systems. The procedure concentrates on the 'inherent properties' of the agents in the sense of their technical character. It is comparatively straightforward to determine the physico-chemical or biochemical properties of substances as being very persistent [CFCs have half-lives of up to 400 years (cf. Koch 1995, p. 259)] and as being very bioaccumulative. The extension of their range is in many cases a consequence of the depth of intervention during their production, the synthesis of non-natural, persistent and mobile chemicals.
However, also with REACH two problems have not yet been solved with regard to the implementation of the precautionary principle. REACH only starts late in the innovation process with the approval of chemicals and products. The constructive approach of precautionary risk management, the precautionary design of environmentally friendly chemicals, 33 is largely ignored. The quest for substitutes comes too 30 Sterling therefore describes the exposure-oriented characteristics as "proxies for possible harm" Sterling (2016, p. 15). 31 For persistence and bioaccumulation, no effect threshold can be given because these properties relate only to exposure. All the more a PEC/PNEC comparison is not possible due to the lack of a quantifiable impact threshold and the impossibility of determining an expected environmental concentration. On the other hand, emphasis is placed on the spatial and temporal decoupling of emission and possible effect (cf. Merenyi et al. 2011). 32 For these substances "authorization can only be granted if it is demonstrated that the socioeconomic benefits outweigh the risks to human health or the environment arising from the use of the substance and if no suitable alternative substances or technologies are available" (Art. 60, para. 4). 33 Cf Hansen et al. (2007), e.g. with the goal 'benign by design', see Laber-Warren (2010), Leder et al. (2015). late. And another necessary step is still missing. If applications are already known, an event-related eVA and a structural vulnerability analysis sVA should be established with a focus on sensitivity, adaptive capacity, criticality, non-linearity and tipping points. In this way, the classification of an agent as being 'of very high concern' as result of the technology characterization should be supplemented by the classification of (elements of) target system as 'being of very high concern' or being in a 'state of very high concern' from vulnerability analysis, each with corresponding consequences in risk management.

Operationalization of the Precautionary Principle in the Governance of GMOs Along the Lines of REACH
If the political will is there to do so, it should be comparatively easy to transfer the steps taken in REACH to operationalize the precautionary principle into the governance and regulation of genetic engineering. The hazard and exposure assessment would initially focus on the characterization of GMOs and genetic engineering constructs. The intensity and depth of intervention and its two consequences, technological power and range, initially will play a central role. As a result of this step alone, GMOs or genetic engineering constructs can be characterized as being of high concern. However, it has to be taken into account that a technology characterization with regard to the release of living organisms is associated with additional challenges in comparison to chemical substances. It is true that even chemicals must be expected to change after release (ageing, oxidation, metabolisation). However, organisms have a significantly higher ontogenetic and phylogenetic plasticity. Nevertheless, the technical characterization of the GMO is a rather simple and low-cost procedure in comparison to the vulnerability analysis of the affected systems (and also in comparison to the steps required by the ERA). And the determination of the persistence and invasiveness of SPAGEs and GMO, their ability to reproduce themselves, to survive (evolutionary fitness) as well as their ability to spread over time and space is already a subject of the ERA.
The Regulation (EC) No. 1829/2003 of the European Parliament and of the Council of 22 September 2003 on genetically modified food and feed and the guidance documents of the EFSA describe in detail which risk-relevant analyses an applicant has to carry out and on which questions he should provide information (Regulation (EC) No. 1829No. /2003. The Environmental Risk Assessment related to the release of GMOs into the environment also begins with certain forms of technical characterization (characteristics of GMOs and releases). 34 The extension of the technology characterization by the criteria depth and intensity of intervention (regarding intensity of intervention see Chap. 1) and their consequences, technological power and range and additional liability should therefore be feasible by comparatively simple means. Aspects of intervention intensity are already represented by the 'scale of the intended release'. Almost all essential points of a (precaution-oriented) hazard and exposure analysis are already addressed, however with quite different intensity and, if Annex II of Regulation No. 1829/2003 is consulted, with a very different degree of detail. Very detailed instructions and scientific methods exist for technology characterization (partly referred to as molecular characterization) and for toxicological analyses (with standards also for allergy and nutrition physiology analyses). This also applies in part to comparative analyses with a "conventional counterpart". On the other hand, the area of environmental impacts is extremely under-represented, which is probably mainly due to the focus on food and feed. There are few indications of ecosystem effects and no indications of possible influences on biodiversity. However, these are discussed in detail in EFSA 2016. Comparatively great attention is paid to the reliability of genetic engineering methods and their undesirable side effects and consequences. What is striking, however, is that their spatial and temporal significance is not queried. There is a lack of specifications how to accurately identify and assess a "potential risk associated with horizontal gene transfer". The technical range and exposure is also given attention in the form of an exposure characterization, not only with regard to the quantitative or estimable aspects "predicted consumption, probable individual and age-specific intake", "recommendations for use, handling", but also with regard to aspects of technology characterization. "the spread of the GMO(s) in the environment (persistence and invasiveness, biological fitness, pathways of dispersal, reproductive, survival and dormant forms); interactions with target or non-target organisms; vertical or horizontal gene transfer; exposure to humans to animals; competition for natural resources like soil, area, water, light, displacement of natural populations of other organisms; delivery of toxic substances; different growth patterns)" cf. Regulation (EC) No. 1829/2003EFSA 2006EFSA , 2013 but again no mention is made of the spatial and temporal implications, possible global spread and irreversibility of releases. The environmental risk assessment shall examine "possible changes in the interactions between the GM plant and its biotic environment resulting from the genetic modification, persistence and invasiveness, selection advantage or disadvantage, gene transfer potential, interactions between the GM plant and target organisms", "interactions between the GM plant and nontarget organisms". However, even here is a lack of guidance on how this requested information can actually be obtained.
In order not to be misunderstood, these enumerations serve above all to show that many aspects are already taken into account which are also important for a classification of 'genetic engineering constructs' or 'GMOs ' as being of 'very high concern'. In contrast to widespread criticism of the ERA, the approach taken here to implement the precautionary principle does not essentially aim at 'more knowledge' about what is not yet known or adequately considered. Rather, it aims at a different weighting of the already existing and comparatively easily accessible findings and, above all, at clear precautionary consequences from the latter. 35 Even if one is unwilling to follow the criterion 'depth of intervention', the following central criteria remain for the identification of constructs and GMOs as being of high concern: -Enormous technological power (e.g. virulence) combined with insufficient technical maturity and reliability. 36 -Ability for self-propagation whereby a distinction must be made here between the propagation rate and the generation times of GDO as well as the ability to overcome Mendelian inheritance rules in GDO -Genetic Fitness of the population (cf. Barker 2009) -Invasiveness or threshold value of propagation, colonization -Persistence, capability to persist and spread over time -Mobility, capability of spatial propagation -Potential for vertical gene transfer/hybridization potential e.g. use of conserved sequences as target loci for integration of homing endonuclease based gene drives -Potential for horizontal gene transfer.

Conclusion
Important points of a precautionary technology characterization and vulnerability analysis are already addressed in the current guidelines and regulations. However, this should be better structured and operationalized. The main task of integrating the precautionary principle into the governance of new genetic engineering, however, is not to collect countless additional data. The identification and classification of GMO or genetic engineering constructs as being of high concern on the one hand and the improvement of the interface between hazard and exposure evaluation and risk management on the other hand by drawing clear precautionary consequences from this classification are of crucial importance. In the current situation, it is completely unclear what follows from the statements required by the applicant on 'risk characterization', 'exposure assessment' or 'environmental compatibility'.

Pursue a constructive precautionary approach
Precautionary risk assessment and precautionary risk management should not begin with product approval. They must begin already in the phase of research and development. Early in the innovation process, when path dependencies have not yet been consolidated by far-reaching investments, corrections, substitutions and the development of lower-risk development paths are much easier. However, due to the much greater degree of lack of knowledge about impacts at this stage, we should not talk about risks (whose assessment at this stage would require information that cannot be obtained), but about risk potentials and the underlying hazard and exposure potentials. Options for measures to influence research and development lie in precautionary risk research, in target-oriented funding programs for lower-risk alternatives, 37 in competitions and prizes, but also in the transparency of processes and opportunities for participation. The targeted promotion of a low-risk design of genetic and biotechnological constructs (benign by design) is particularly important. If it does not want to always come too late and intervene restrictively, the orientation to such a design is an indispensable approach of the precaution-oriented risk management.

Consider all kinds of non-knowledge
Precautionary measures are dependent on the generation of precautionary knowledge. Knowledge related to precaution should be able to understand the extent and possible consequences of a lack of knowledge in the form of comprehensible reasons for concern. All forms of non-knowledge must be taken into account, not only the uncertainties currently mentioned in the ERA and in approval procedures, but also unpredictability (the limits of prediction in dealing with complex systems due to non-determinacy), complete ignorance of possible surprises (unknown unknowns), and last but not least the technically produced extension of the realm of possibility and the thus extended ignorance.

Pursue technology characterization and vulnerability analysis
Technology characterization can start particularly early in the innovation process, even in the phase in which gene drives are currently being developed, in which hardly any applications are yet on the market. With the criteria of intensity and depth 37 Wynne (2005) points out that the UK Biotechnology and Biological Sciences Research Council recommends "genomically-informed but non-transgenic approaches to crop science research", as a kind of research that is more likely to meet society's expectations in its rejection of green genetic engineering (BBSRC 2004, p. 35). In its report on 'Genomics and Crop Plant Science in Europe', the European Academies Science Advisory Council also recommended "non-reductionist functional genomics informing marker-assisted selection for identifying non-GM, naturally occurring desired crop traits" (EASAC 2004, p. 7). Wynne stated "A previously invisible alternative scientific trajectory marginalized by the exclusive GM paradigm, came rapidly to the fore, as the necessity suddenly arose" (p. 79f). of intervention as well as liability, the focus is not first on effects, but on the character of the intervention, which produces these effects in the first place, especially technological power and range. Particularly high technological power (up to the triggering of chain reactions) and particularly high range of exposure (up to globality and irreversibility) through GMO and genetic engineering constructs can be characterized as being 'of high concern'. The criteria intensity and depth of intervention with the dimensions of technological power and range meet the requirements for the performance of precautionary criteria and reasons for concern, with regard to the degree of seriousness of hazard and exposure potentials, the magnitude of the possible consequences and the extension of the ignorance generated by the depth of intervention. In addition, it provides indications as to the direction in which lower-risk alternatives can be successfully sought.
In structural vulnerability analysis, it is important to identify particularly critical and sensitive systems or system elements, to identify tipping points and threatening phase transitions and bifurcations. If intervention is planned in systems which are critical for society (e.g. nutrition, health), which are pre-loaded or pre-tensed or include tipping points, then intervention in these systems can be characterized as being of high concern.
Further work is needed on the two methodological approaches of technology characterization and vulnerability analysis, as well as on the criteria depth and intensity of intervention, and on further indications of serious hazard and exposure potentials.

Exposure is just as important as hazard
Exposure after releases into the environment must be given as much attention as to the hazard dimension. This has now become established in the risk governance of chemicals. Very persistent and very bioaccumulative chemicals are classified under REACH as substances of very high concern. Thus an extreme exposure potential, even without an associated hazard hypothesis, is considered to be of very high concern. Regarding an independent spread of GMOs or their genes in ecosystems, the term 'so what?' is still all too often used (von Schomberg 2006, p. 24). The ability of GMOs and genetic engineering constructs to spread in the environment must therefore be anchored as a major concern in the regulation of genetic engineering. The underlying risk hypothesis refers to the fact that the temporal and spatial extension of the presence of such persistent and invasive constructs enormously increases the likelihood of their interaction with different elements and relations in different ecosystem contexts and can thus lead to major surprises, as had to be learned from the example of CFCs. Minimizing exposures is therefore a promising approach to dealing with unknown unknowns.

Improving the link between hazard and exposure evaluation and risk management
The interface between hazard and exposure assessment and evaluation and risk management needs to be improved. The characterization of a GMO or construct as being of 'very high concern' should lead to the same consequences as under REACH, i.e. a ban on use, an authorization requirement with exemptions and an active search for lower-risk alternatives.
Open Access This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made. The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.