Background

The technology for genetic modification of plants provides a tool for crop breeding and has been applied for the development of varieties with novel or improved traits. Insect resistance has been achieved via introduction of genes from the bacterium Bacillus thuringiensis (Bt), which is the second most distributed GM trait worldwide after the herbicide tolerance. The only GM plant cultivated commercially at large scale in EU is maize designed to produce a Bt toxin - Cry1Ab that provides protection against corn borers (Lepidoptera). GM crops producing different types of Bt toxins against other pest insects or in combination with other GM traits are also cultivated outside Europe and being considered for EU cultivation [1, 2].

Because Bt crops contain insecticidal proteins, potential interactions with non-target organisms are of major concern for the risk assessment [3]. In recent years, many field and laboratory studies have been conducted to evaluate the potential effects of the Bt crops on above-ground and soil dwelling non-target organisms.

Soil invertebrates are classified according to their size as microfauna (protozoa, small nematodes), mesofauna (nematodes, Collembola, mites, enchytraeids) and macrofauna (earthworms). They play significant roles in the nutrient cycling and energy flow and are actively involved in physical, chemical and biological processes. It is very likely that any changes in soil properties will affect invertebrate communities, thus their composition and abundance can be used as an indicator of soil quality [4, 5].

There are two possible ways that the Bt crop can influence soil biota: First, directly through root feeding, root exudates and litter that contain Cry toxin and/or directly through unintended changes in the plant, caused by the genetic modification; and secondly, indirectly through changes in agricultural management practices related to the genetic modification, e.g. changed insecticide regimes. The major concern is that the Bt crop cultivation may have an effect on soil invertebrate populations and their communities [6], which could attain a magnitude that would cause undesirable changes in the soil ecosystem functioning according to thresholds set by legislation or environmental authorities.

Systematic review and meta-analysis have been applied routinely for synthesis of data from medical studies assessing the risk or benefits of treatments or drugs. However, the potential to use this methodology for quantitative synthesis of data from impact assessment of GM plants has also been recognized [7]. Meta-analysis of field and laboratory studies assessing the effects of Bt crops to non-target organisms have been performed already [8–12]. Although comprehensive literature reviews exists [13–15] a quantitative synthesis of evidence with a quality control as required for a systematic review about the effects of Bt crops to the populations of soil invertebrates is still missing. It will contribute firm evidence-based conclusions about the possible impacts on soil biota communities and to the soil ecology in general.

Population abundance and biomass are the major end-points for monitoring of soil invertebrates in ecological studies, therefore this systematic review will study data from field experiments of these two measurement endpoints in relation to soil-dwelling and surface-dwelling species including protozoa, nematodes, Collembola, mites, enchytraeids, and earthworms.

Objective of the review

This systematic review aims for a synthesis of the field evidence about the effects of Bt crops to six groups of soil-dwelling and surface-dwelling invertebrate species, protozoa, nematodes, Collembola, mites, enchytraeids, and earthworms.

The review question (RQ) asked by the present systematic review protocol is:

Are population abundances and biomasses of soil invertebrates changed by Bt crops compared with conventional crops?

The question has the following components:

Population: Soil-dwelling and surface-dwelling invertebrate species: protozoa, nematodes, Collembola, mites, enchytraeids, and earthworms.

Exposure: Genetically modified Bt crops and the conventional comparator in their concomitant farming practice through the soil environment and in the rhizosphere.

Comparator: Conventional non-GM crops and their concomitant farming practice.

Outcome: Net changes in the population abundances or biomasses contrasted with the comparator.

Methods

The systematic review methodology describes the approach which will be used to find and analyse original articles containing data from field experiments assessing the effects of the Bt crop cultivation on soil invertebrates including: search strategy and terms; study inclusion criteria and quality assessment; data extraction and methods for evidence synthesis.

Search strategy

The aim of the search is to find all available studies containing data from field experiments assessing the effect of Bt crop cultivation on soil invertebrates. The main approach will be to conduct comprehensive electronic searches in web databases, search engines for scholarly literature and specialized databases. An additional source will be the personal data collected by experts and stakeholders of the GRACE project network. In addition, the reference lists of related review papers and datasets from previously conducted reviews will be checked manually for relevant studies.

Search terms

Search terms defining the population - types of soil invertebrates, the exposure – types of Bt proteins, the assessed outcome - population abundances or biomasses, and the method - field studies, will be used for retrieving of relevant studies. The search terms will be organised in strings, which will be modified according to the requirements of each bibliographic database. The used search strings and the results will be documented and presented as an additional file of the systematic review.

The following terms will be used:

Population terms - invertebrate* OR mesofauna OR macrofauna OR arthropod* OR “ground-dwelling arthropod*” OR “surface-dwelling” OR microarthropod* OR microfauna* OR nematod* OR springtail* OR collembola* OR protoz* OR protist* OR earthworm* OR lumbricid* OR enchytraeid* OR oligochaeta OR acar* OR mite*

Exposure terms - cry1* OR cry2* OR cry3* OR “bacillus thuringiensis” OR delta-endotoxin OR Bt OR “cry toxin”

Outcome terms - population* OR abundanc* OR communit* OR diversit* OR biodiversity OR number* OR biomass* OR effect* OR impact*

Method terms (optional) - field

Scoping exercise

Scoping exercise was performed for a preliminary assessment of the availability of relevant research literature and for optimisation of the search strings to be employed in the systematic review. The search for research publications which contained the necessary elements for inclusion in the systematic review was done by simplified and focused search strings in Scopus platform, one string for each of three major groups of soil invertebrates - nematodes, Collembola and earthworms, (population AND exposure AND [method]; e.g. earthworm* AND Bt AND field). In this way, 17 studies (see Additional file 1) were found which subsequently were used as references to evaluate the relevance of the search strings in a pilot search exercise. The aim of pilot search was to determine the exact content of the search strings to be used in the systematic review and was conducted in three literature web databases: Web of knowledge, Scopus and AGRIS.

The content of the search strings was modified by including or excluding terms until all 17 articles (if present in the database) were found among the records and in the same time have not resulted in an excessive amount of irrelevant studies. The search strings which produced manageable numbers of records with all the reference studies among them are shown in Table 1. The search strings thus defined will be used for the systematic review search.

Table 1 Pilot search results (search conducted on 17.04.2014)
Full size table

Web databases

Literature databases Databases containing scientific literature including theses, books, abstracts and articles will be searched using the defined strings. The following search service providers and bibliographic databases will be used:

  •  Web of knowledge (webofknowledge.com/) - search service including the following citation databases and platforms:

Web of Science® - platform which consists of nine databases containing scholarly journals, books, book series, reports, conferences, and other articles.

BIOSIS Citation IndexSM - comprehensive reference database for life science research which includes cited references to primary journal literature.

MEDLINE® - database of the U.S. National Library of Medicine (NLM) contains over 12 million records of journal articles in all areas of the life sciences.

Specialized databases The Center for Environmental Risk Assessment (CERA) Bibliography Database (http://cera-gmc.org/index.php?action=bibliography_database)

Bibliosafety (http://bibliosafety.icgeb.org/) - The Biosafety Bibliographic Database

Database of the Safety and Benefits of Biotechnology (http://biotechbenefits.croplife.org/)

Biosafety Information Resource Centre (BIRC) (http://bch.cbd.int/database/resources/)

Web search engines

The following search engines will be used:

Google scholar (http://scholar.google.com/)

JSTOR (http://www.jstor.org/)

The first 200 hits will be checked for relevance. The links will be followed once from the original hit.

Personal datasets

Experts and stakeholders from the consultation network created within the GRACE project will be asked to provide data relevant to the topic. The complete list of articles received by personal communication and the source will be recorded and will be included in the additional file of the systematic review.

Manual search

Literature datasets or databases from other reviews related to effects of Bt crops on non-target soil invertebrates, as well as the reference lists of articles found within the electronic search will be searched manually.

All searches will be performed in English; however, no restriction for language or year of publication will be made. In that way, all studies which have published an abstract in English will be within the scope of our search. Original full texts in either English or German will be included for further analyses. The search results from each of the used sources will be saved and citations will be imported in ENDNOTE® citation manager software. All duplicates will be removed and a list containing the accumulated results will be created and uploaded to the open-access database CADIMA (Central Access Database for Impact Assessment of Crop Genetic Improvement Technologies).

Study inclusion criteria

In order to be included a study needs to fulfil each of the following criteria:

Relevant population(s): Field collected soil invertebrates at species level or at higher taxonomic levels among protozoa, nematodes, Collembola, mites, enchytraeids, and earthworms.

Relevant exposure(s): Field soil and rhizosphere exposure to genetically modified Bt crops and comparator non GM crops and their associated farming practices.

Relevant comparator(s): Non-Bt near-isogenic crop or another non-Bt variety of the same crop species in an experimental design allowing for any of the comparisons:

  •  Bt with non-Bt plots, neither of which received any additional insecticide treatments.

  •  Bt plots not treated with insecticide with non-Bt plots that received insecticides.

  •  Bt with non-Bt fields when both are subject to insecticide treatments.

Relevant outcomes: Population abundance or biomass.

Appling study inclusion criteria

Two reviewers will independently perform the selection of studies, which fulfil the inclusion criteria at three stages. At first, articles will be selected by their title to remove highly irrelevant studies from the overall search results, followed by the second stage in which the inclusion criteria will be applied against the abstracts of the articles. If there is doubt or lack of enough information from the title and abstract alone to judge whether the article meets the inclusion criteria the full text of the study will be obtained to enable the assessment.

At the beginning of the second stage, after the first 100 publications are processed a test for consistency between two reviewers will be made using Cohen’s Kappa (see Additional file 2). The calculation of Cohen’s kappa coefficient will represent the agreement between the two reviewers. If the Cohen’s kappa coefficient is less than 0.6, the inconsistencies will be discussed and the criteria for inclusion will be adjusted taking into account the main reasons for disagreement.

In the third stage, the articles will be reviewed in full text for the presence of each of the elements needed for the inclusion. The reasons for including or excluding each study at this stage will be recorded and reported.

Study quality assessment

All the studies, which fulfil the inclusion criteria, will be assessed for bias. The aim of this phase is to ensure that the studies are providing evidence, which represent true statistical similarity and level of difference. The internal (design, conduct, and analyses) and external (population, interventions, and outcomes) validity will be assessed by applying four quality domains - selection bias, performance bias, measurement bias and attrition bias and defined as low, high or uncertain and the results will be reported separately for each domain.

One reviewer will perform the assessment using check lists. A random subsample (20%) of the studies will be assessed additionally by second reviewer and the outcomes from both will be compared with Cohen’s Kappa. The inconsistencies (Cohen’s kappa coefficient is less than 0.6) will be discussed and a third reviewer will be involved in case no agreement could be reached. Records with the evaluation results and the reasons for judgment will be made for each article included at this stage.

Selection bias

Pre-treatment differences between the studied groups and in the baseline characteristics of the study will be assessed in the selection bias domain. The following elements will be assessed:

  • plot location - low risk if the experimental plots of both the intervention and the control treatment are located in one field with known history. High risk if the plots are located at different fields and there is no information about the history of the field.

  • comparator - low risk if the comparison is Bt crop vs corresponding isogenic line. High risk if another variety is used as the comparator.

  • randomization - the randomisation is the best way to avoid selection bias; thus, studies which are designed by any block or plot randomization method will be considered as low risk. High risk studies will occur if there is no or poor randomization.

  • replications - low risk if there are 4 or more replicates per treatment. Moderate risk is if there are 1 to 3 replicates. High risk if there is no replication.

Performance bias

Performance bias arises if the studied groups are influenced by factors different from the intervention, which may have an effect on the measured outcomes. In the field studies, such influences might come from:

  • plot size - low risk if the plot size is properly defined, depending on the movement behaviour of the examined taxa and high risk if plot size is excessively small.

  • field management - low risk if both control and intervention received the same agro-technical management including tillage, fertilizers, fungicides, irrigation, cultivation etc. High risk if there are differences in the management between the treatments.

Measurement bias

The way the measurement of the outcomes is done can influence the true effect estimation if the selected method is not accurate or can be influenced by human subjectivity.

  • sampling of soil invertebrates - sample collection and extraction procedures are crucial when assessing the abundance and biomass of the soil invertebrates. A common source of errors is the variation in depth and number of soil cores. The techniques differ between taxa, and so a general recommendation is not appropriate. When sampling and extraction are performed using standardized techniques or other recognized methods, the risk of bias will be considered low. If the technique is not suitable for the examined taxa, or is prone to human influence, the risk of bias will be considered high.

Attrition bias

Imbalance in the final set of selected studies will be assessed. In theory, an imbalance may occur if studies with the following properties are excluded:

  • sample size - low risk if the sample size is equal between the treatments. High risk if it differs due to loss of samples.

  • missing data - low risk if the amount of data for the measured outcomes is equal for all the treatment. High risk if there is imbalance in the presented outcome data between the treatments.

Data extraction strategy

The aim of the data extraction stage is to retrieve information relevant to the design, performance and measured outcomes, which will be used for the quantitative synthesis and the analysis of the variability between studies. Details about the experimental sites and design will be extracted from the text in the sections describing the materials and methods of the study, as well as the description of sampling and extraction techniques and statistical analysis. Numeric data for the measured outcomes and the corresponding variance will be extracted from tables and figures in the result section of the study. All data will be imported into a standardised Excel table. Each measurement of the population abundance or biomass of the different treatment (e.g. Bt crop and comparator) will be included in the table as separate records containing all the defining variables. One review team member will perform the data extraction and checks for errors will be made by another review team member of a random subset (20%) of the data. The discrepancies will be solved by the involvement of a third reviewer. Data to be extracted from each included study are presented in Table 2.

Table 2 List of variables to be extracted from the papers for the systematic review
Full size table

Data analysis

The extracted data will be used to synthesize the evidence provided by the individual studies about the effects of Bt crop cultivation on soil invertebrates, as well as to investigate the heterogeneity among the studies.

Only quantitative population data will be used for the meta-analysis of Hedges’ g. There will be some aggregation of population abundance or biomass data into taxonomic groups to enable comparison between studies not using the same level of taxonomic resolution.

Assessment of statistical power of included studies

A post-hoc analysis of the power to detect effects with different magnitude small (effect size of 0.2), medium (effect size of 0.5) large (effect size of 0.8) at significance criterion 0.05 of each study will be made (Cohen, 1988).

Measures of treatment effect

Depending on the studied taxon and/or the sampling technique, the population abundance of soil invertebrates can be measured as number of individuals per volume of soil or per surface area, and the biomass as the weight of the population per volume of soil or per surface area. The reported values are usually the mean for the treatment and the calculation of the associated variance. Finally, irrespective of measurement units treatment effect sizes reported in each study will be expressed as Hedges’ g.

Dealing with missing data

If measurements of the population abundance or biomass of either the treated plots or the control plots are missing then the study is not included in the meta-analysis. In cases when, for the purpose of reporting the variance, other values than standard deviation or standard error are used, e.g. t, F, p or z-values, an appropriate mathematical method will be used to calculate the pooled standard deviation, if appropriate. If this is not feasible, authors will be contacted to provide the missing data.

Synthesis

Quantitative synthesis Quantitative synthesis will be performed to combine the magnitude of the effects from the individual studies. The meta-analysis will include calculation of the pooled effect size (Hedges’g) for each study accompanied with the corresponding confidence intervals. The results from the meta-analysis will be presented graphically in ‘forest plots’, where the estimated effect size with the confidence interval of each individual study will be plotted horizontally as the combined effect size and confidence interval will be plotted below them.

The complete dataset will be stored in database for open access after the finalisation of the review. For mixed effects modelling SAS PROC MIXED, PROC GLIMMIX or R ver. 3 will be performed [16, 17].

Heterogeneity of the variability across studies is already implicit in the estimate of the standard deviation for Hedges’ g.

A range of effect modifiers will be extracted from the selected papers and stored in the database. These effect modifiers include comparator properties, pesticide treatment, experimental design, cropping system, crop rotation, tillage date, date of GM experimental cropping system establishment and soil type. When the final dataset allows for assessing the effect of the effect modifiers on the outcome of the meta-analysis and any biases will be reported.

Assessment of heterogeneity The heterogeneity across the studies that may influence the outcome will be assessed. In field studies estimating the effects of Bt crop cultivation on soil invertebrates, several sources of heterogeneity may be expected.

Heterogeneity in studied populations The examined taxonomic groups will vary between the studies. The populations under investigation will include species or higher taxonomic groups among the soil invertebrates: protozoa, nematodes, Collembola, mites, enchytraeids, and earthworms.

Heterogeneity in type of exposure A source of heterogeneity related to the type of exposure will be the variability of the GM plant species and the type of Bt toxin, i.e. the CRY event, which they produce. Among the most widely studied Bt crops are maize producing Cry1Ab (against lepidopteran pests) and Cry3Bb1 (against coleopteran pests), followed by Cry1Ac producing cotton.

Preliminary assessment of the heterogeneity in the studied population and type of exposure among the studies used as references for the pilot search are shown in Table 3.

Table 3 Preliminary heterogeneity assessment of crops, taxonomic groups and types of Bt toxins
Full size table

Heterogeneity in methodology The main source of variability between the studies related with the methodology could be expected from differences in the experiment duration and time of sampling. Some authors report one or two years study with one sampling at the end of the season, while others report multi-year studies with several sampling occasions in the beginning, in the middle and in the end of the season. Considering the possible cumulative effect of the Bt- toxin in the soils the measurements carried out in several subsequent years will be compared separately if corresponding data is found.

Other sources of heterogeneity which may have an effect on the outcome of the field studies could be related to differences in plot size, sample size, sampling method and field management, or to be caused by the different factors of the environment such as the soil type and availability of water.

Heterogeneity in comparisons The experimental design of the studies can include one or a combination of the following comparisons: Bt with non-Bt plots, neither of which received any insecticide treatments; Bt- plots not treated with insecticide with non-Bt plots that received insecticides and Bt with non-Bt fields when both are subject to insecticide treatments.

To deal with the above mentioned heterogeneity sub-group meta-analysis will be performed if data suitable to calculate the effect size is found in a minimum of three distinct publications, which contain the same Bt crop producing and type of Cry toxin) and the same group of studied invertebrates (protozoa, nematodes, Collembola, mites, enchytraeids, and earthworms). Meta–analysis on the finest possible taxa will be made, however if necessary an aggregation of data from lower taxonomic groups will be performed. If heterogeneity in the sampling time is extreme for effect size calculation will be used the peak abundance or biomass of the season.

Statistical analysis of heterogeneity Heterogeneity will be addressed in conventional analyses of correlation, regression and mixed modelling ANOVA to reveal if the heterogeneities have any significant impact on the Bt crop effect estimates. E.g. if the studies can be grouped into soil type categories holding sufficient number of replicates, hypotheses about the effect of soil type on the outcome of effects can be elucidated. This will be one type of sensitivity analysis. In principle all the effect modifiers can be included in sensitivity analyses, i.e. do their inclusion or exclusion affect the assessment of effect levels in terms of Hedges’ g. Effects reported by a study will be critically assessed by excluding the risk of confounding between the effect modifiers and the effect measure of Hedges’ g.

Sensitivity analysis The validity of the systematic review findings will be verified by sensitivity analysis. Meta-analysis calculations will be undertaken twice using different assumptions related to the quality of experimental performance and reporting of the results, as well as differences of the methods used. Studies will be included or excluded from the calculations according to the following factors:

  •  weight of the individual study;

  •  study quality;

  •  near isogenic vs. non-isogenic comparator;

  •  methodological differences;

  •  missing data;

  •  outlier studies (results differ significantly from a range of other studies).

Other factors also considered relevant will be applied, described and recorded.

Assessment of publication bias In many research areas it is known that “positive” results are more likely to be published [18]. Publication bias will be assessed using “funnel plot” analysis, where the effect size of an individual study will be plotted on the horizontal axis and the standard error or sample size on the vertical axis. The asymmetry in the funnel plot may indicate publication bias [19].

Review teams

An extraction team at ABI and a review team at Dep. of Bioscience have been established to ensure internal quality assurance of the review process.