Abstract
Background
In 2008/2009, Schmidt and colleagues published a study reporting lethal effects of the microbial Bt toxins Cry1Ab and Cry3Bb on the coccinellid biological control organisms Adalia bipunctata. Based on this study, in concert with over 30 other publications, Mon810 cultivation was banned in Germany in 2009. This triggered two commentaries and one experimental study all published in the journal 'Transgenic Research' that question the scientific basis of the German ban or claim to disprove the adverse effects of the Bt toxins on A. bipunctata reported by Schmidt and colleagues, respectively. This study was undertaken to investigate the underlying reasons for the different outcomes and rebuts the criticism voiced by the two other commentaries.
Results
It could be demonstrated that the failure to detect an adverse effect by Alvarez-Alfageme and colleagues is based on the use of a significantly different testing protocol. While Schmidt and colleagues exposed and fed larvae of A. bipunctata continuously, Alvarez-Alfageme and colleagues applied an exposure/recovery protocol. When this exposure/recovery protocol was applied to a highly sensitive target insect, Ostrinia nubilalis, the lethal effect was either significantly reduced or disappeared altogether. When repeating the feeding experiments with the Bt toxin Cry1Ab using a combined protocol of both previous studies, again, a lethal effect on A. bipunctata larvae was observed. ELISA tests with Bt-toxin fed larvae and pupae confirmed ingestion of the toxin.
Conclusions
The new data corroborates earlier findings that Cry1Ab toxin increases mortality in A. bipunctata larvae. It was also shown that the different applied testing protocols explained the contrasting results.
See related articlehttp://www.enveurope.com/content/24/1/9
Similar content being viewed by others
Explore related subjects
Find the latest articles, discoveries, and news in related topics.Background
In 2009, Schmidt et al. [1] published a study where they showed that larvae of the ladybird beetle, Adalia bipunctata, died at a significantly higher rate when raised on meal moth eggs (Ephestia kuehniella) coated with a solution containing the microbially produced, purified toxins from Bacillus thuringiensis [Bt] than in the Bt-free control (Figure 1). In the control, A. bipunctata larvae were fed with meal moth eggs that were either coated with buffer solution without the Bt toxin or with buffer solution containing the expression vector pBD10 (the 'empty construct' containing all materials expressed by the vector except the target trait, the Bt toxin) to exclude effects of the production method itself. The feeding trials were carried out using two different Bt toxins, each at three different concentrations. These experiments have been of interest to regulators and developers of genetically modified [GM] crops because the toxins tested were of the same class as those that are expressed in GM Bt plants, such as Mon810, the only Bt maize currently approved for cultivation in Europe. To date, such experiments using isolated, purified Bt toxins from microbes tested on about half a dozen standard testing organisms, including ladybird beetle species, form the ecotoxicological data basis for the environmental risk assessment [ERA] required for market approval of GM plants [2–4]. In spring 2009, the German Federal Office of Consumer Protection and Food Safety instituted a ban on the cultivation of Mon810 maize [5] expressing one of the tested Bt toxins used in the study of Schmidt et al. [1]. Although in support of the ban, more than 30 scientific articles published in peer-reviewed journals were cited [5]; the results of Schmidt et al. [1] were claimed to have had a pivotal role in reaching the political decision for a ban. While we cannot verify whether that was the case, it did trigger two response papers [6, 7] and the publication of one experimental study [8] claiming to disprove the results published by Schmidt et al. [1]. In the study by Alvarez-Alfageme et al. [8], larvae of the same ladybird species were subjected in feeding trials to the same toxins but the authors failed to detect a statistically significant increase in mortality rates. Hence, Alvarez-Alfageme et al. [8] concluded that the 'apparent harmful effects of Cry1Ab and CryBb1 reported by Schmidt et al. [1] were artifacts of poor study design and procedures' and 'false positives.' In Figure 1, we provide the comparative mortalities observed at the various Bt toxin concentrations in the two studies discussed here, and in Table 1, the relevant statistical values.
Since the three papers cited above [6–8] grounded their critique on the same points and cross-referenced each other, we have considered them jointly and explored the most critical issues raised by the authors experimentally. These critical issues include the following: (1) feeding habits of neonate A. bipunctata larvae and exposure routes; and (2) applied testing protocols in feeding studies.
In this paper, we present the outcomes of these additional experiments and, simultaneously, address other points of critique raised by Rauschen [6], Ricroch et al. [7], and Alvarez-Alfageme et al. [8]. We further identified a series of double standards that must be addressed and resolved within the biosafety sciences and for the regulatory approval process in order to provide clarity for decision making and biosafety testing in the future.
Materials
Firstly, we describe the materials used in our experiments, including the insects, Bt toxins, and the Bt plants. Secondly, the experimental designs and applied protocols of the feeding studies are described and which points of critique they address.
Insects
We used two species of insects, the herbivorous target pest species Ostrinia nubilalis (European corn borer) and the nontarget beneficial predator A. bipunctata (two-spotted ladybird beetle).
Ostrinia nubilalis
Eggs and larvae of O. nubilalis were taken from our in-house colony maintained since 2010 without any introductions of field-collected insects. This colony was started from a population received from INRA (Le Magneraud, France) in May 2010. The larvae were maintained on artificial diet, and the adults were kept on a honey-water solution. The colony was held in a climate chamber at 25°C, 70% relative humidity [RH], and a photoperiod of 16:8 h light to dark [L:D].
Adalia bipunctata
As in the studies of Schmidt et al. [1] and Alvarez-Alfageme et al. [8], eggs of A. bipunctata were purchased from Andermatt Biocontrol, Grossdietwil, Switzerland. Treatment of the eggs and larvae followed exactly the procedures described by Schmidt et al. [1].
Meal moth eggs
Sterilized eggs of the meal moth (E. kuehniella) were also purchased from Andermatt Biocontrol, Grossdietwil, Switzerland.
Bt toxins
In the experiments of Schmidt et al. [1], the Bt toxins applied were provided by the collaborating laboratory within the European Union [EU]-project Bt-BioNoTa, Plant Research International, Wageningen under the leadership of Dr. Ruud de Maagd, a recognized expert of Bt toxins and their mode of action [9, 10] who routinely produced Bt toxins for his own use and for collaborative research. Ricroch et al. [7] raised the critique that Schmidt et al. [1] neither provided details on 'the activity of the cleaved toxin and on the possible formation of unusual peptides as a consequence of the trypsic digestion nor on the composition of the bacterial extracts, which do not seem to consist of pure protoxins.' If this criticism would be valid, it would apply to all experiments carried out with these toxins by all other partners, in particular the wealth of research carried out by Dr. Ruud de Maagd, who also produced the empty constructs used in the experiments by Schmidt et al. [1]. However, we are unaware that this critique as to the quality of the Bt toxin material has ever been raised or evidence for it put forward regarding other experiments (e.g., [11]), including those by Dr. Ruud de Maagd (cross-referenced in the study of Crespo et al. [11]). Hence, we consider this criticism by Ricroch et al. [7] as baseless speculation.
Despite our confidence in the material used previously, we accounted for this issue in our new studies by using Bt toxins from another source. These toxins were produced by Dr. Marianne Pusztai-Carey, Case Western University, Cleveland, Ohio, one of the most experienced, widely respected and routined producer of Bt toxins used by a large community of scientists researching on particular issues of resistance evolution to these Bt toxins in target pests (e.g., [11–13]). For details on the production method of these toxins, see the study of Pusztai-Carey et al. [14], which is also cited and explained in the study of Crespo et al. [11].
Bt plants
The following two hybrids were used: PAN 6Q-321B (GM MON810) and PAN 6Q-121 (non-Bt near iso-hybrid for PAN 6Q-321B). The GM plants express the Cry1Ab toxin targeting in Europe the European corn borer, O. nubilalis. They were grown in climate chambers at a fluctuating temperature of 20°C (8 h/night) and 25°C (16 h/day) and a relative humidity (RH) of 65%. They were used for the experiments when they were in the six- to eight-leaf stage.
Feeding habit and exposure route of neonate Adalia bipunctata larvae
Alvarez-Alfageme et al. [8] postulated that young A. bipunctata larvae 'puncture the prey and suck out the contents' and thus do not ingest compounds that are deposited on the outside egg shell. Because the literature provided by Alvarez-Alfageme et al. [8] in support of their claim [15–17] does not document visually how A. bipunctata larvae actually do feed, we decided to observe and document their feeding behavior.
Feeding behavior of first instar Adalia bipunctata larvae
As their first food, newly hatched first instar larvae were offered both meal moth eggs and their own eggs in a small (petri dish) arena. With the aim of describing their feeding behavior, more than 20 larvae were observed under a stereomicroscope for many hours over the course of 2 weeks. Their mouthparts and their feeding behavior were photographically documented.
Exposure route
Alvarez-Alfageme et al. [8] argued that exposure and uptake must be verified using a positive control which in their case was potassium arsenate. While we agree that it is important to validate uptake, we argue that validation of uptake through a particular pathway using a positive control is much more critical in cases where no differences between the treatment and control can be observed, as for example in the direct feeding studies by Alvarez-Alfageme et al. [8]. This is to exclude the possibility that the lack of an effect was due to the lack of ingestion. We argue that in the case of Schmidt et al. [1], the researchers did indeed demonstrate uptake through the incurment of significantly higher mortalities in their treatments serving effectively as the 'positive control.' The only difference between the controls and the treatments in the studies by Schmidt et al. [1] was the applied Bt toxin. Hence, by all scientific standards, this is a reasonably strong indication that the Bt toxin elicited this response.
However, to corroborate uptake of the Bt toxin in our studies here, we tested exposed A. bipunctata larvae for presence or absence of the Bt toxin. We raised an additional number of small A. bipunctata larvae on Bt toxin-moistened cotton balls and Bt toxin-coated meal moth eggs and subjected them to a qualitative ELISA test. Three groups of 10 neonate larvae were treated as in the feeding experiment. They were offered Bt sugar solution-dipped cotton balls for the first 24 h followed by Bt toxin-treated meal moth eggs on cardboard for another 2 days before being frozen. We placed 1, 3, 6, or 12 Bt-fed A. bipunctata larvae into 2-ml screw cap tubes together with 500 μl of phosphate-buffered saline with Tween-20 [PBST] buffer to the 12 larvae sample or 150 μl of 1× PBST buffer to each of the other samples. Six steel beads (2-mm diameter) were added to each tube, and the larvae were then homogenized using a FastPrep®-24 instrument (MP Biomedicals, Inc., Santa Ana, CA, USA). After centrifugation at 12,000 rpm for 10 min, supernatants were transferred to 1.5-ml Eppendorf tubes and tested for presence of the Bt toxin using Agdia Bt-Cry1Ab/1Ac ImmunoStrips® according to the Agdia protocol (Agdia Inc., Elkhart, IN, USA). Additionally, we raised a total of 10 Bt toxin-fed A. bipunctata larvae and 10 A. bipunctata larvae fed with a Bt-free control diet as described above to the pupal stage (from two replicates with five individuals each). All pupae were washed in buffer solution to rinse off the remaining Bt toxin from the pupal skin. Four and six pupae were placed together into 1.5-ml Eppendorf tubes and subjected to the same treatment as the larvae described above.
Applied protocols in feeding studies
In order to explore the underlying reasons for the differences in mortality rates observed by Schmidt et al. [1] and by Alvarez-Alfageme et al. [8], two series of experiments were carried out: The first series explored the differences of the applied protocols in a 'proof-of-concept' experiment using a highly sensitive target herbivore species. The second series of experiments repeated the experiments with A. bipunctata using a combined protocol.
Firstly, the differences in the applied protocols between the two studies by Alvarez-Alfageme et al. [8] and Schmidt et al. [1] are listed and discussed (Table 1).
There are at least four striking differences with regard to diets, doses, exposure times, and controls, aside the large differences in statistical robustness (e.g., much larger sample size in Schmidt et al. [1]):
1. Diet: In an attempt to overcome the perceived shortcoming when offering Bt toxin-coated meal moth eggs and to allow for easier uptake of the toxin, Alvarez-Alfageme et al. [8] offered each larva droplets of sugar/water solution with and without Bt toxins. However, since carnivorous larvae of ladybird beetles cannot sustain themselves on sugar solution only, this kind of liquid food could only be offered for 24 h, after which the larvae were given more substantial food containing proteins. Hence, Alvarez-Alfageme et al. [8] transferred their A. bipunctata larvae to clean petri dishes and fed them 'ad libitum with meal moth eggs.' This procedure was apparently carried out four times in total, always at the beginning of a new larval instar (see below). Notably, the meal moth eggs were not coated with Bt toxins.
2. Exposure: While Schmidt et al. [1] raised and exposed their A. bipunctata larvae to Bt toxins for a total of 9 to 10 consecutive days, Alvarez-Alfageme et al. [8] did not. Instead, they offered A. bipuncata larvae droplets of water/sugar solution with Bt toxin only once within a period of 24 h at the beginning of each of the four larval instars. Following the 24-h exposure period to Bt toxins, untreated meal moth eggs were provided. We consider this a critical difference and possibly the main reason why their experiment yield different results from Schmidt et al. [1].
3. Doses: While Schmidt et al. [1] tested three different doses for each Bt toxin, Alvarez-Alfageme et al. [8] tested only one for each Bt toxin. Although the dose-response curves observed by Schmidt et al. [1] were strongly criticized by Alvarez-Alfageme et al. [8], no attempt was made to repeat it.
4. Controls: Schmidt et al. [1] treated the meal moth eggs with two types of control solutions: (a) buffer solution of the same concentration as in the treatments and (b) buffer solutions containing the potential expression products of the empty constructs. These stemmed from cloned Escherichia coli containing only the plasmid pBD10 without the target transgene cry1Ab or cry3Bb. This was in an attempt to exclude possible effects caused by the microbial production process or the transgenes other than the target transgenes. This control was lacking in the Alvarez-Alfageme et al. [8] studies as well as the addition of buffer solution to the control. Each Bt toxin treatment series in the study of Schmidt et al. [1] was accompanied by its own zero control including the pBD10 control. Further, three concentrations of pBD10 were added as additional controls, yielding a total of six controls altogether. Alvarez-Alfageme et al. [8] used only one control for both single-dose Bt toxins (Table 1).
Proof of concept: testing the power of the different protocols to detect toxic effects on the susceptible target pest, Ostrinia nubilalis
From the methodological differences described above, we concluded that, although claimed otherwise, Alvarez-Alfageme et al. [8] did not repeat the experiments of Schmidt et al. [1] but followed a different protocol with a significantly shorter exposure time followed by extended periods of recovery. In order to explore the significance of these differences in protocols between continuous exposure vs. exposure/recovery, we carried out a series of experiments with the most Bt toxin-sensitive target herbivore species, O. nubilalis. Cry1Ab toxins, either as toxin solutions sprayed on non-Bt, isogenic maize leaves or as Bt-maize plant material, were offered to 4-day old O. nubilalis larvae either for 24 h only or continuously. We chose to work with 4-day O. nubilalis larvae because they are slightly less sensitive and stay alive long enough in the continuous exposure treatment to serve as a model system.
The bioassays were carried out using insect rearing trays (BIO-RT-32©, C-D International Inc., Pitman, NJ, USA) consisting of 32 chambers/tray that were covered with perforated lids. The bottoms of the chambers were covered with filter paper, supplied with a few drops of water and the different diets of the five treatments:
-
1.
Control: 2 × 2-cm leaf pieces of the isogenic maize line (no Bt) were supplied throughout the entire bioassay;
-
2.
Bt 24: 2 × 2-cm leaf pieces of GM Bt maize were supplied for 24 h; for the rest of the experiment, 2 × 2-cm leaf pieces of the isogenic maize line were provided;
-
3.
Bt continuous: 2 × 2-cm leaf pieces of GM Bt maize were supplied throughout the entire bioassay;
-
4.
Toxin 24: 2 × 2-cm leaf pieces of the isogenic maize line were sprayed with Bt toxin solution (45 μg ml-1) and supplied for 24 h; for the rest of the experiment, unsprayed leaf pieces of the isogenic maize line were provided;
-
5.
Toxin continuous: 2 × 2-cm leaf pieces of the isogenic maize line were sprayed with Bt toxin solution (45 μg ml-1) and supplied during the entire bioassay.
Sixteen O. nubilalis larvae per treatment were placed individually in the chambers of the rearing trays. The insect rearing trays were kept in a climate chamber at 25°C, 70% RH, and a photoperiod of 16:8 h (L:D). Filter papers were moistened and diets replaced on a daily basis. We checked larvae for mortality daily for a period of 7 days. The experiments were repeated four times over time with new insects and plants.
Merging protocols for optimization of bitrophic feeding trials with Adalia bipunctata larvae
Newly hatched A. bipunctata larvae were allowed to feed on companion eggs for 24 h before they were placed individually in the chambers of insect rearing trays (BIO-RT-32©, C-D International Inc., Pitman, NJ, USA) consisting of 32 chambers/tray that were covered with perforated lids. Initially, we followed the protocol laid out by Alvarez-Alfageme et al. [8] and provided each larva in each cell with two droplets of water/sugar solution containing 45 μl/ml-1 of Cry1Ab toxin deionized water for 24 h. However, we quickly discovered that, after at most 12 h (overnight) and probably earlier, these drops had completely dried up. Hence, we adjusted the protocol and offered the sugar-Bt toxin solution and the sugar-water solution with buffer as control in the form of moistened cotton balls of ca. 0.5 cm in diameter. These cotton balls remained moist for at least 24 h and allowed continuous access to the test or control solutions throughout the entire 24 h. A. bipunctata larvae readily accepted this form of administration of the solution and frequently fed on them (Figure 2). After 24 h, instead of switching the diets from sugar-Bt toxin solution to untreated meal moth eggs only, we replaced the old cotton balls with new cotton balls moistened with or without Bt toxin and, further, added sterilized meal moth eggs on 2 × 2-cm cardboard squares that had been coated with solutions containing either the Cry1Ab toxin at a concentration of 45 μl/ml water or buffer solution only. Thus, we ensured continuous exposure and uptake of Bt toxin throughout their entire larval stage.
Data analysis
Mortality of O. nubilalis and A. bipunctata was tested for significant differences using a two-sided Cochran-Mantel-Haenszel chi-squared test for O. nubilalis bioassay data and a one-sided Cochran-Mantel-Haenszel chi-squared test for the A. bipunctata bioassay data. All analyses were carried out using the statistics software R [18].
Results
Feeding habit of neonate Adalia bipunctata larvae and exposure route
In the series of pictures (Figure 3a, b, c), we provide a selection of all photographs taken that best show how these very small larvae feed.
Larvae feed on eggs, either their own or those of meal moths, by sinking their mouthparts deeply into the eggs and biting their way through the egg shells (Figure 3b, c). In close and intimate contact with the eggshells, they crack them open. As a consequence, the egg content leaks out of the eggs and the larvae lick and suck up whatever they can get out of it (Figure 3c). During this process, there seems to be ample opportunity for the egg contents to get 'contaminated' with any compound coating the eggs and the surface on which the eggs are placed which in our case and that of Schmidt et al. [1] were Bt-sprayed 2 × 2-cm pieces of cardboard. These observations strongly suggest that the main difference between the methods employed by Alvarez-Alfageme et al. [8] and Schmidt et al. [1] is that in the former trials, the larvae ingested a water/sugar-solution mixed with Bt toxins and in the latter, the liquid egg content mixed with Bt toxins.
Verification of exposure route and uptake of Bt toxin by Adalia bipunctata larvae and pupae
The results clearly revealed the presence of Bt toxin in the larvae and pupae homogenates as shown in Figure 4 and, thus, confirm uptake of the Bt toxin through the Bt-coated meal moth eggs. Except for the single-larvae sample, all samples exhibited positive signals with increasing strength as the number of larvae in the sample increased (Figure 4). Increasing strength of the positive band signaled the presence of the Bt toxin and an increasing dose as more individuals were added to the sample.
Agdia Bt-Cry1Ab/1Ac ImmunoStrips® showing positive detection of Bt toxin in bodies of tested A. bipunctata. Two bands signal positive detection. (a) Larvae. 1 = 1 larvae in 150 μl of extraction buffer, 2 = 3 larvae in 150 μl of extraction buffer, 3 = 6 larvae in 150 μl of extraction buffer, and 4 = 12 larvae in 500 μl of extraction buffer. (b) Pupae. 1 = control 10 pupae in 350 μl of extraction buffer, 2 = 4 pupae in 150 μl of extraction buffer, and 3 = 6 pupae in 150 μl of extraction buffer.
Feeding trials
Here, the results of the proof-of-concept feeding experiments with O. nubilalis and the feeding trials with A. bipuncatata larvae using the combined protocols of Schmidt et al. [1] and Alvarez-Alfageme et al. [8] are reported.
Proof of concept: differences in protocol matter
When applying the protocol used by Alvarez-Alfageme et al. [8] to the target pest species, we observed substantial differences in the response to the Bt toxins (Figure 5).
Mortality of 4-day old O. nubilalis larvae [± SE] exposed to Bt toxins. The Bt toxins were either from microbial solutions (activated Cry1Ab toxin) sprayed on non-Bt, isogenic maize leaves or Bt-maize plant material. Exposure times were either continuous for 7 days (Bt continuous = Bt maize, Toxin continuous = Bt toxin-sprayed isogenic maize) or for 24 h only (Bt 24, Toxin 24) followed by Bt-free, isogenic maize for 6 days, respectively. Data represent the mean of 4 replications (n = 16). Different letters above the bars (a, b and c) indicate significant differences between treatments.
Compared to the continuous Bt-exposure treatment, mortality was highly significantly lower when O. nubilalis larvae were exposed to Bt maize (χ2MH = 60.174, df = 1, p < 0.001) or Bt-sprayed isogenic maize (χ2MH = 40.494, df = 1, p < 0.001) for 24 h only and then allowed to recover on optimal, untreated food (Figure 5). In the continuous Bt-exposure treatments, regardless whether of plant or microbial origin, all O. nubilalis larvae died. In the 24-h/recovery treatment using Bt-sprayed, isogenic maize, mean mortality was half of the mortality rates in the treatments where Bt-containing food was offered continuously but still significantly higher than those in the control (χ2MH = 3.924, df = 1, p < 0.0048). However, in the 24-h/recovery treatment using Bt maize, the lethal effect on O. nubilalis was lost entirely compared to the control.
Bitrophic feeding trials with Adalia bipunctata
Two observations motivated us to repeat the direct feeding trials with A. bipunctata using a combined protocol. For one, Alvarez-Alfageme et al. [8] did not use Bt toxin-coated meal moth eggs but used untreated meal moth eggs in their bitrophic experiments with A. bipunctata. Coating meal moth eggs is a simple and straightforward method widely used in ecotoxicity testing of side effects of pesticides, for example, with another predator species Chrysoperla carnea (the green lacewing) (see also commentary in this volume). Secondly, we noted in the Alvarez-Alfageme et al. [8] study that mortality rates in the Cry1Ab toxin treatment were also 9% higher than those in the control (15.0% vs. 24.3%) and close to those observed by Schmidt et al. [1] in their lowest Bt concentration treatment (27.5%) (Figure 1 and Table 1).
In our trials applying a combined protocol, A. bipunctata showed significantly higher mortality rates when raised continuously on Bt-containing diet compared to the control (χ2MH = 4.679, df = 1, p = 0.015) (Figure 6). Across the three runs, mean mortality after 6 days was 40% [SE ± 2.3] in the Bt-treatment group and 24.0% [SE ± 0.9] in the control. Mean mortality rates were already, on day 2, significantly higher in the Bt treatment than in the control (χ2MH = 5.1929, df = 1, p = 0.011). The greatest difference in mean mortality rates between the Bt treatment and the control was observed on day 4 (χ2MH = 6.5058, df = 1, p = 0.005). Afterwards, mortality rates leveled off. These data confirm that under continuous exposure to Bt-containing food, A. bipunctata is adversely affected and dies at a significantly higher rate than in the Bt-free controls.
Cumulative mortality rates. Cumulative mortality rates of A. bipunctata larvae [± SE] raised on Bt toxin-coated eggs plus Bt toxin-soaked cotton balls compared to the Bt-free control. Data represent the mean of 3 replications (n = 32). Significant higher mortalities (χ 2MH test) compared to the control (dotted line) are indicated by an asterisk (p < 0.05) and double asterisks (p < 0.01).
Discussion
With these additional experiments, we did not only corroborate the previous results reported by Schmidt et al. [1] but also alleviated the discussed perceived shortcomings in the protocols by using a combination of both, the protocol used by Schmidt et al. [1] and the one used by Alvarez-Alfageme et al. [8]. The new data strengthened the evidence for direct adverse effects of activated Cry1Ab toxin on the juvenile stages of the coccinellid species, A. bipunctata. We demonstrated uptake, documented bioactivity of the used toxins in a robust manner, and lastly but quite importantly, provided a convincing explanation for the lack of a statistically significant lethal effect in the bitrophic studies of Alvarez-Alfageme et al. [8]. We conclude that these authors did not observe the effects reported by Schmidt et al. [1] because they used a substantially different exposure/recovery protocol instead of the continuous exposure protocol applied by Schmidt et al. [1]. When using the exposure/recovery protocol even for the most sensitive target organism, O. nubilalis, the lethal effect observed in the continuous exposure treatment also largely disappeared.
Meanwhile, Dhillon and Sharma [19] reported that 'direct exposure of Cheilomenes sexmaculatus larvae to Bt toxins resulted in reduced larval survival and adult emergence as compared to the controls.' They did not observe such effects when raising the ladybird larvae on aphid prey that had fed on Bt toxin-containing artificial diet (indirect exposure)]. This study has also been dismissed by Alvarez-Alfageme et al. [8] as being 'inconclusive' because toxic effects were reported in some bioassays but not in others and because 'no convincing explanation was provided for the observed differences.' Dhillon and Sharma [19] clearly stated that one likely reason for these differences is that one experiment was a direct exposure trial and the other an indirect exposure trial adding a layer of complexity through potential interactions of the aphids with the toxin either regarding bioactivity or concentration. Moser et al. [20] observed significantly prolonged development times for coccinellid larvae (Coleomegilla maculata) that fed on Bt maize compared to those that fed on isogenic non-Bt maize. Although this could have been due to the Bt toxin, no conclusions about potentially lethal effects could be drawn because mortality was not studied. Furthermore, in 2009, Waltz [21] reported in an article for the journal Nature Biotechnology that researchers in the United States also observed strong lethal effects when feeding Bt plant material containing toxins of the coleopteran-active Cry3 class (likely pollen) to ladybird larvae. Because this data did not receive permission for publication from the corporate developer of that Bt-maize event [21], we cannot compare it to our data.
Additional criticisms raised in the concerted papers that we wish to address here include the following: (a) control mortalities, (b) lack of quantification of ingested amount of Bt toxin per insect, (c) lack of linear dose-response relationship and the mechanism behind the observed phenomenon, and (d) consideration of field studies.
Control mortalities
In standard ecotoxicological trials, 20% of control mortality rates are routinely accepted. When control mortality rates exceed 20%, the experiments are terminated but not discarded (see various dossiers listed on relevant websites of the US Environmental Protection Agency (e.g. [22, 23]) or European Food Safety Agency, or for a critical review, see the study of Charleston and Dicke [24]). Indeed, control mortality rates as high as 24% have sometimes been accepted as part of dossiers submitted for regulatory approval [24]. Hence, we consider the mortality rate reported here within acceptable limits. However, out of the six control mortality rates published by Schmidt et al. [1], three ranged well below the mortality rate (< 15%) recorded in the single control run carried out by Alvarez-Alfageme et al. [8], two rates were at 15%, and only one control mortality rate was > 15% (Table 1). Hence, the one control mortality rate exceeding the widely accepted standard by roughly only 2% is hardly 'flawed' and does not invalidate the other data. Quite on the contrary, it is unusually rigorous and unparalleled in any of the relevant cited studies that a total of six control treatments were carried out including the 'empty construct' at three different concentrations. Furthermore, the slightly higher control mortality rate reported by Schmidt et al. [1] for one of these six control treatments led to a more conservative outcome and nonsignificant differences to all except one of the three tested Bt toxin concentrations. In contrast, Alvarez-Alfageme et al. [8] used only one negative control treatment for both toxin treatments.
Lack of quantification of ingested amount of Bt toxin
The amount of Bt toxin ingested was not quantified in any feeding study including the one by Alvarez-Alfageme et al. [8]. As we noticed in our attempt to repeat their protocol, the droplets provided for insects dried before being entirely consumed, and hence, no quantification of the exact uptake could possibly be carried out. Likewise, no quantification was or is possible when administering the toxin via prey, and so, none was reported for the tritrophic trials of Alvarez-Alfageme et al. [8] (Table 1). Nevertheless, the lack of quantification of the ingested amount of toxin does not invalidate the observed phenomenon. Additionally, in our studies published here, we confirmed uptake of the offered Bt toxin by detecting it positively in the larvae and pupae which fed on the Bt toxin via sugar solution-dipped cotton balls and sprayed meal moth eggs, but not in the ladybird larvae and pupae raised on Bt toxin-free control food.
Lack of linear response relationship
The lack of a linear dose-response relationship as addressed explicitly by Schmidt et al. [1] also does not invalidate the observed phenomenon. It has long been known that some toxicological processes, for instance, protein binding, are capacity limited (e.g., [25]) and that response curves can therefore deviate from a linear line above a certain threshold. For example, Watanabe et al. [25] explained that such deviation of 'toxicity at high doses may be due to saturated detoxification or excretion mechanism which when fully operative at lower doses does not result in the same toxic response.' Although we do not know the mechanism causing the adverse effects we observe in A. bipunctata larvae, the nonlinear nature of the response may in fact provide first clues. In that context, Schmidt et al. [1] are similarly astonished as Rauschen [6] to note that the coleopteran-specific Cry3Bb protein had a smaller effect than the lepidopteran-specific Cry1Ab. Indeed, Schmidt et al. [1] state this in the first sentence of their discussion: '...it would not be expected that a predacious coleopteran nontarget species exhibits susceptibility to the lepidopteran-active protein Cry1Ab.' However, the lack of an explaining mechanism for a documented phenomenon certainly does not constitute a 'methodological flaw' (see below). We note that Alvarez-Alfageme et al. [8] chose not to repeat the criticized dose-response experiment, but to work with one dose only.
Consideration of field studies
It is often pointed out that several field studies have failed to confirm the adverse effects shown in laboratory studies like ours and those of others [7, 26]. These field studies are taken as proof of safety and refuting evidence of the adverse effects observed in laboratory studies. We maintain, however, that not only have Schmidt et al. [1] put their studies in the proper context, but also devoted at least three rather long paragraphs to discussing the problems of detecting these effects in the field, given the diversity of possible exposure routes in the field. This treatment of the topic is hardly 'superficial' by any standard of a scientific journal with strict page limitations. In fact, Schmidt et al. [1] clearly stated that 'exposure of ladybirds to Bt toxins in Cry1Ab maize fields is, therefore, rather unlikely as long as they feed solely on aphids.' They further explain that this is grounded on the findings of an earlier study where neither aphids feeding on Cry1Ab maize nor the phloem sap of that Bt-maize event were found to contain Cry1Ab toxins [27]. However, many of the published field studies (including the list provided by Ricroch et al. [7]) worked at the community level (i.e., differences in species diversity between Bt and non-Bt crops) or/and at the population level (abundance/densities of populations of species or larger taxonomic units). Many of these field studies worked with temporal sampling intervals spanning at least 1 week or more (e.g. [28–30]) and are confounded by a number of other factors such as use of pesticides and focus on the pesticide-intensive industrial crop cotton [31, 32]. Any coccinellid or other predator larva detected at one sampling date has long developed into a pupa or adult until the next sampling date. Hence, all of these above-listed parameters measured in field trials can at best detect intergenerational effects. Underlying this approach is the assumption that diversity indices or population densities are the proper parameters for validating ecotoxicological trials that studied intragenerational effects of individual insects. While it might seem trivial that an increase of mortality of a species should lead to the decline in abundance in the field, in nature, few processes if any work in such a linear manner. All species are part of complex food webs next to many other species and many environmental factors confounding such supposedly trivial relationships (e.g., immigration/emigration, movement behavior) [33]. For a more elaborate discussion whether single species measurements can be used to anticipate community consequences, we refer to the literature that exists on this topic, for example, Holmes and Kareiva [33]. Also, in the ecotoxicology science community of chemicals, there is an ongoing, lively discussion on these issues spanning now at least four decades and the field is constantly striving for improved protocols to gain more relevance for field situations. However, on principle, these discussions affect all ecotoxicological laboratory studies regardless of their outcome as they are all riddled by the same limitations.
The studies by Schmidt et al. [1] and the one we report about here are typical early tier tests carried out to determine a possible hazard that might require additional testing. These studies stand in concert with other studies using other species outside of the orders of the classic target insects (i.e., Lepidoptera, Coleoptera and Diptera) and include, e.g., C. carnea, Daphnia magna, Lepidostoma liba, or Helicopsyche borealis [24, 26, 34–38]. These reports provide indications for possible hazards that require further investigation (or possibly long-term field monitoring) to determine whether they pose a risk or translate into 'harm' in the field: no more and no less.
Recent research reports provide new knowledge regarding the mode of action of Bt toxins, and a paradigm shift in our understanding of Bt toxins has been suggested [39]. Pardo-Lopez et al. [40] reports that the activity of Cry toxins can be significantly increased in the presence of certain proteins, e.g., chitinases or protease inhibitors. Both proteins are not rare in organisms like insects. Efficacy was also enhanced with different modification at the toxin gene level, such as a deletion of small fragments from the amino-terminal region of the Bt toxin or when introducing cleavage sites in specific regions of the protein [40]. While this was considered as a solution option to overcome the rising levels and incidences of resistance among target pests again using Bt toxins in GM Bt plants, such efficacy increase can just as well affect nontarget organisms that have exhibited some susceptibility. Graf [39] postulates that the precise mode of action of Bt toxins against lepidopterans is poorly understood, referring to a recent study where scientists confirmed that the mode of action of Bt toxins is enhanced or may even require the presence of certain gut microbes, something that has been overlooked until Broderick et al. [41] published a paper that established the phenomenon that has now been followed up by more research shedding some light on the underlying mechanism [42]. There is no reason to believe that such microbe, matrix, and toxin interactions do not apply to insects of higher trophic levels, too. Kramarz et al. [43] found that nematode-infected snails grew significantly slower when raised on Bt maize compared to its non-Bt maize. Van Frankenhuyzen [44] recently reported that there are an increasing number of Bt toxin families with cross-order activity. Fifteen out of eighty-seven toxin families are active against more than one order. However, he also pointed out the still restricted knowledge on Bt toxin specificity regarding the range of affected species with 91% of Bt toxins tested against 10 or less insect species only, and those were almost always herbivorous species and presumed target pests. Very few studies are carried out with predatory or parasitoid species. Van Frankenhuyzen's [44] conclusion that this cross-border activity does not threaten the environmental safety of Bt-based pest control, because they tend to be much less toxic to taxa outside of the primary specificity range and due to their short persistence in the field, may hold true for sprayable Bt formulations. However, with GM plants expressing multiple Bt toxins in their activated form for several months every year in widely grown commodity crops (e.g., maize, cotton), a whole new spatio-temporal dimension is added to the exposure side. This may very well accelerate or completely change the dynamics of small adverse effects observed in bioassays. As a further note of caution, it should be noted that small sublethal effects, such as a slight shift in development time, can result in a much more drastic adverse ecological effect in the field if it leads to the disruption of a fine-tuned pest-antagonist relationship than a somewhat increased lethal effect. A given population may be able to compensate for a limited increase in mortality but may be doomed if it misses its prime food/host source in time.
However, we firmly reject the trajectory as laid out by Rauschen [6]. The author postulates that any political decision maker (presumably a regulator) can reach a decision on 'real potential risks' only if the full chain of events is 'scientifically sound' and 'thoroughly' demonstrated from the 'relative importance of different exposure pathways' to the testing of a measurable effect at 'realistic exposure regimes', to the wider implications for the exposed populations, and lastly, to the biological control function of that species. If the biological control function of a natural enemy is demonstrably adversely affected in a realistic regime (i.e., the field), by convention, this constitutes not a real potential risk but a 'proof of damage/harm.' In that case, risk assessment is irrelevant as the harm has realized. Then, the only option left for any regulator is damage control and mitigation, if possible. Risk assessment for regulatory purposes by convention is an exercise carried out prior to field releases and commercial approval with the goal to prevent any of the above from actually happening in the field. Thus, any EU regulator following the recommendations by Rauschen [6] would be in conflict with national (German), EU, and international regulations (Cartagena Protocol on Biosafety) as they all require ERA to be carried out in consideration of precaution.
References
Schmidt JEU, Braun CU, Whitehouse LP, Hilbeck A: Effects of activated Bt transgene products (Cry1Ab, Cry3Bb) on immature stages of the ladybird Adalia bipunctata in laboratory ecotoxicity testing. Arch Environ Contam Toxicol 2009, 56: 221–228. 10.1007/s00244-008-9191-9
Hilbeck A, Meier M, Römbke J, Jänsch S, Teichmann H, Tappeser B: Environmental risk assessment of genetically modified plants - concepts and controversies. Environ Sci Eur 2011, 23: 13. 10.1186/2190-4715-23-13
Dolezel M, Miklau M, Hilbeck A, Otto M, Eckerstorfer M, Heisenberger A, Tappeser B, Gaugitsch H: Scrutinizing the current practice of the environmental risk assessment of GM maize applications for cultivation in the EU. Environ Sci Eur 2011, in press.
Hilbeck A, Jänsch S, Meier M, Römbke J: Analysis and validation of present ecotoxicological test methods and strategies for the risk assessment of genetically modified plants. BfN-Skripten 2008, 236: 287.
Bundesamt für Verbraucherschutz BVL: Bekanntmachung eines Bescheides zur Beschränkung des Inverkehrbringens von gentechnisch veränderten Mais der Linie Mon810.2009. [http://www.bvl.bund.de/SharedDocs/Downloads/08_PresseInfothek/mon_810_bescheid.html]
Rauschen S: A case of "pseudo science"? A study claiming effects of the Cry1Ab protein on larvae of the two-spotted ladybird is reminiscent of the case of the green lacewing. Transgenic Res 2010, 19: 13–16. 10.1007/s11248-009-9301-0
Ricroch A, Bergé JB, Kuntz M: Is the German suspension of Mon810 maize cultivation scientifically justified? Transgenic Res 2010, 19: 1–12. 10.1007/s11248-009-9297-5
Alvarez-Alfageme F, Bigler F, Romeis J: Laboratory toxicity studies demonstrating no adverse effects of Cry1Ab and Cry3Bb1 to larvae of Adalia bipunctata (Coleoptera: Coccinellidae): the importance of study design. Transgenic Res 2010. doi: 10.1007/s11248–010–9430–5 doi: 10.1007/s11248-010-9430-5
de Maagd RA, Bravo A, Crickmore N: How Bacillus thuringiensis has evolved specific toxins to colonize the insect world. Trends Genet 2001, 17: 193–199. 10.1016/S0168-9525(01)02237-5
de Maagd RA, Bravo A, Berry C, Crickmore N, Schnepf HE: Structure, diversity, and evolution of protein toxins from spore-forming entopathogenic bacteria. Annu Rev Genet 2003, 37: 409–433. 10.1146/annurev.genet.37.110801.143042
Crespo ALB, Spencer TA, Nekl E, Pusztai-Carey M, Moar WJ, Siegfried BD: Comparison and validation of methods to quantify Cry1Ab toxin from Bacillus thuringiensis for standardization of insect bioassays. App Environ Microbiol 2008, 74: 130–135. 10.1128/AEM.01855-07
Anilkumar KJ, Pusztai-Carey M, Moar WJ: Fitness costs associated with Cry1Ac-resistant Helicoverpa zea (Lepidoptera: Noctuidae): a factor countering selection for resistance to Bt cotton? J Econ Entomol 2008, 101: 1421–1431. 10.1603/0022-0493(2008)101[1421:FCAWCH]2.0.CO;2
Anilkumar KJ, Rodrigo-Simón A, Ferré J, Pusztai-Carey M, Sivasupramaniam S, Moar WJ: Production and characterization of Bacillus thuringiensis Cry1Ac-resistant cotton bollworm Helicoverpa zea (Boddie). Appl Environ Microbiol 2008, 74: 462–469. 10.1128/AEM.01612-07
Pusztai-Carey M, Carey PR, Lessard T, Yaguchi M: Isolation, quantitation, and purification of insecticidal proteins from Bacillus thuringiensis . 1994. U.S. patent 5,356,788
Banks CJ: The behaviour of individual coccinellid larvae on plants. Br J Animal Behav 1957, 5: 12–24. 10.1016/S0950-5601(57)80039-2
Hagen KS: Biology and ecology of predaceous Coccinellidae. Annu Rev Entomol 1962, 7: 289–326. 10.1146/annurev.en.07.010162.001445
Hodek I, Honek A: Ecology of Coccinellidae. Dordrecht: Kluwer; 1996.
R Development Core Team: R: A Language and Environment for Statistical Computing. Vienna, Austria: R Foundation for Statistical Computing; 2010.
Dhillon MK, Sharma HC: Effects of Bacillus thuringiensis δ-endotoxins Cry1Ab and Cry1Ac on the coccinellid beetle, Cheilomenes sexmaculatus (Coleoptera, Coccinellidae) under direct and indirect exposure conditions. Biocontrol Sci Technol 2009, 19: 407–420. 10.1080/09583150902783801
Moser SE, Harwood JD, Obrycki JJ: Larval feeding on Bt hybrid and non-Bt corn seedlings by Harmonia axyridis (Coleoptera: Coccinellidae) and Coleomegilla maculata (Coleoptera: Coccinellidae). Environ Entomology 2008, 37: 525–533. 10.1603/0046-225X(2008)37[525:LFOBHA]2.0.CO;2
Waltz E: Under wraps. Are the crop industry's strong-arm tactics and close-fisted attitude to sharing seeds holding back independent research and undermining public acceptance of transgenic crops? Nat Biotechnol 2009, 27: 880–882. 10.1038/nbt1009-880
US Environmental Protection Agency Office of Pesticide Programs: Biopesticide registration action document: Bacillus thuringiensis Cry34Ab1 and Cry35Ab1 proteins and the genetic material necessary for their production in event DAS-59122–7 corn.[http://www.epa.gov/oppbppd1/biopesticides/ingredients/tech_docs/brad_006490.pdf]
US Environmental Protection Agency Office of Pesticide Programs: Biopesticide registration action document: Bacillus thuringiensis Cry1F corn.[http://www.epa.gov/oppbppd1/biopesticides/ingredients/tech_docs/brad_006481.pdf]
Charleston DS, Dicke M: Designing experimental protocols to investigate the impact of GM crops on non-target arthropods.[http://www.cogem.net/index.cfm/en/publications/publicatie/designing-experimental-protocols-to-investigate-the-impact-of-gm-crops-on-non-target-arthropods]
Watanabe PG, Young JD, Gehring PJ: The importance of non-linear (dose-dependent) pharmacokinetics in hazard assessment. J Environ Pathol Toxicol 1977, 1: 147–159.
Hilbeck A, Schmidt JEU: Another view on Bt-proteins - how specific are they and what else might they do? Biopesticides Int 2006, 2: 1–50.
Raps A, Kehr J, Gugerli P, Moar WJ, Bigler F, Hilbeck A: Immunological analysis of phloem sap of Bacillus thuringiensis corn and of the nontarget herbivore Rhopalosiphum padi (Homoptera: Aphididae) for the presence of Cry1Ab. Mol Ecol 2001, 10: 525–533. 10.1046/j.1365-294x.2001.01236.x
Pilcher CD, Obrycki JJ, Rice ME, Lewis LC: Preimaginal development, survival and field abundance of insect predators on transgenic Bacillus thuringiensis corn. Environ Entomol 1997, 26: 446–454.
Pilcher CD, Rice ME, Obrycki JJ: Impact of transgenic Bacillus thuringiensis corn and crop phenology on five nontarget arthropods. Environ Entomol 2005, 34: 1302–1316. 10.1603/0046-225X(2005)034[1302:IOTBTC]2.0.CO;2
Sisterson MS, Biggs RW, Olson C, Carrière Y, Dennehy TJ, Tabashnik BE: Arthropod abundance and diversity in Bt and non-Bt cotton fields. Environ Entomol 2004, 33: 921–929. 10.1603/0046-225X-33.4.921
Marvier M, McCreedy C, Regetz J, Kareiva P: A meta-analysis of effects of Bt cotton and maize on non-target invertebrates. Science 2007, 316: 1475–1477. 10.1126/science.1139208
Naranjo SE: Impacts of Bt crops on non-target invertebrates and insecticide use patterns. CAB Reviews: Perspectives in Agriculture, Veterinary Science, Nutrition and Natural Resources 2009, 4: 1–11.
Holmes EE, Kareiva PM: Using single-species measurements to anticipate community level effects of environmental contaminants. In Environmental Contaminants and Terrestrial Vertebrates: Effects on Populations, Communities, and Ecosystems. Edited by: Albers PH, Heinz GH, Ohlendorf HM. Florida: Society of Environmental Toxicology and Chemistry (SETAC); 2000:315.
Hilbeck A, Moar W, Pusztai-Carey M, Filipini A, Bigler F: Prey-mediated effects of Cry1Ab toxin and protoxin and Cry2A protoxin on the predator Chrysoperla carnea (Neuroptera: Chrysopidae). Entomol Exp Appl 1999, 91: 305–316. 10.1046/j.1570-7458.1999.00497.x
Hilbeck A, M Baumgartner M, Fried PM, Bigler : Effects of transgenic Bt corn-fed prey on immature development of Chrysoperla carnea (Neuroptera: Chrysopidae). Environ Entomol 1998, 27: 480–487.
Hilbeck A, Moar W, Pusztai-Carey M, Filipini A, Bigler F: Toxicity of Bacillus thuringiensis Cry1Ab toxin to the predator Chrysoperla carnea (Neuroptera: Chrysopidae). Environ Entomol 1998, 27: 1255–1263.
Bohn T, Primicerio R, Hessen DO, Traavik T: Reduced fitness of Daphnia magna fed a Bt-transgenic maize variety. Arch Environ Contam Toxicol 2008, 55: 584–592. 10.1007/s00244-008-9150-5
Rosi-Marshall EJ, Tank JL, Royer RV, Whiles MR, Evans-White M, Chambers C, Griffiths NA, Pokelsek J, Stephens ML: Toxins in transgenic crop byproducts may affect headwater stream ecosystems. Proc Natl Acad Sci USA 2007, 104: 16204–16208. 10.1073/pnas.0707177104
Pardo-Lopez L, Munoz-Garay C, Porta H, Rodriguez-Almazan C, Soberon M, Bravo A: Strategies to improve the insecticidal activity of Cry toxins from Bacillus thuringiensis . Peptides 2009, 30: 589–595. 10.1016/j.peptides.2008.07.027
Graf J: Shifting paradigm on Bacillus thuringiensis toxin and a natural model for Enterococcus faecalis septicaemia. mBio 2011, 2: e00161–11.
Broderick NA, Raffa KF, Handelsman J: Midgut bacteria required for Bacillus thuringiensis insecticidal activity. Proc Natl Acad Sci USA 2006, 103: 15196–15199. 10.1073/pnas.0604865103
Mason KL, Stepien TA, Blum JE, Holt JF, Labbe NH, Rush JS, Raffa KF, Handelsman J: From commensal to pathogen: translocatoin of Enterococcus faecalis from the midgut to the hemocoel of Manduca sexta . mBio 2011, 2: e00065–11.
Kramarz PE, de Vaufleury A, Zygmunt PMS, Verdun C: Increased response to cadmium and Bacillus thuringiensis maize toxicity in the snail Helix aspersa infected by the nematode Phasmarhabditis hermaphrodita . Environ Toxicol Chem 2007, 26: 73–79. 10.1897/06-095R.1
Van Frankenhuyzen K: Insecticidal activity of Bacillus thuringiensis crystal proteins. J Invertebr Pathol 2009, 101: 1–16. 10.1016/j.jip.2009.02.009
Acknowledgements
This project was supported by institutional funding of the Plant Ecology Group of the Institute of Integrative Biology. Further, we are grateful for the constructive input of three anonymous reviewers and the stimulating discussions with numerous colleagues. We are very grateful for the statistical support provided by Dr. Sabine Güsewell, Plant Ecology Group of the Institute of Integrative Biology.
Author information
Authors and Affiliations
Corresponding author
Additional information
Competing interests
The authors declare that they have no competing interests.
Authors' contributions
AH provided the ideas and supervision and wrote the manuscript. MM was primarily responsible for the statistical analyses and graphic representation. JSM and JMM joined their creativity and resourcefulness and managed to take nice photographic images of these really very small and agile insects 'in action', i.e., feeding. JSM, AH and JMM with the support of MT and the occasional help by MM carried out the experiments. All contributed jointly and equally to the discussion of the scientific and nonscientific issues that came along with this research and its outcomes. All authors read and approved the final manuscript.
Authors’ original submitted files for images
Below are the links to the authors’ original submitted files for images.
Rights and permissions
Open Access This article is distributed under the terms of the Creative Commons Attribution 2.0 International License (https://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.
About this article
Cite this article
Hilbeck, A., McMillan, J.M., Meier, M. et al. A controversy re-visited: Is the coccinellid Adalia bipunctata adversely affected by Bt toxins?. Environ Sci Eur 24, 10 (2012). https://doi.org/10.1186/2190-4715-24-10
Received:
Accepted:
Published:
DOI: https://doi.org/10.1186/2190-4715-24-10