Hydrogeology Journal

, Volume 25, Issue 4, pp 1063–1076 | Cite as

The effects of ionic strength and organic matter on virus inactivation at low temperatures: general likelihood uncertainty estimation (GLUE) as an alternative to least-squares parameter optimization for the fitting of virus inactivation models

  • Jean-Marc Mayotte
  • Thomas Grabs
  • Stacy Sutliff-Johansson
  • Kevin Bishop
Open Access
Paper

Abstract

This study examined how the inactivation of bacteriophage MS2 in water was affected by ionic strength (IS) and dissolved organic carbon (DOC) using static batch inactivation experiments at 4 °C conducted over a period of 2 months. Experimental conditions were characteristic of an operational managed aquifer recharge (MAR) scheme in Uppsala, Sweden. Experimental data were fit with constant and time-dependent inactivation models using two methods: (1) traditional linear and nonlinear least-squares techniques; and (2) a Monte-Carlo based parameter estimation technique called generalized likelihood uncertainty estimation (GLUE). The least-squares and GLUE methodologies gave very similar estimates of the model parameters and their uncertainty. This demonstrates that GLUE can be used as a viable alternative to traditional least-squares parameter estimation techniques for fitting of virus inactivation models. Results showed a slight increase in constant inactivation rates following an increase in the DOC concentrations, suggesting that the presence of organic carbon enhanced the inactivation of MS2. The experiment with a high IS and a low DOC was the only experiment which showed that MS2 inactivation may have been time-dependent. However, results from the GLUE methodology indicated that models of constant inactivation were able to describe all of the experiments. This suggested that inactivation time-series longer than 2 months were needed in order to provide concrete conclusions regarding the time-dependency of MS2 inactivation at 4 °C under these experimental conditions.

Keywords

Virus inactivation Uncertainty Groundwater management Bacteriophage MS2 Health 

Les effets de la force ionique et de la matière organique sur l’inactivation du virus à basses températures: estimation de l′incertitude de la vraisemblance généralisée (GLUE) comme alternative à l′optimisation des moindres carrés pour l′ajustement des modèles d′inactivation virale

Résumé

Cette étude a examiné comment l’inactivation du bactériophage MS2 dans l’eau a été affectée par la force ionique (IS) et le carbone organique dissous (COD) en utilisant des expériences statiques d’inactivation en batch à 4 °C réalisées sur une période de deux mois. Les conditions expérimentales sont caractéristiques d’un système opérationnel de gestion d’un aquifère par recharge (MAR) à Uppsala, Suède. Les données expérimentales ont été ajustées avec des modèles d’inactivation constants et dépendants du temps en utilisant deux méthodes: (1) techniques traditionnelles des moindres carrés linéaires et non linéaires ; et (2) un technique d’estimation Monte-Carlo basée sur les paramètres appelée estimation des incertitudes de la vraisemblance généralisée (GLUE). Les méthodes des moindres carrés et de la méthode GLUE ont donné des estimations très similaires des paramètres du modèle et de leur incertitude. Ceci démontre que la méthode GLUE peut être utilisée comme une alternative viable aux techniques traditionnelles d’estimation des paramètres par moindres carrés pour l’ajustement des modèles d’inactivation virale. Les résultats ont montré une légère augmentation des taux d’inactivation constants suite à une augmentation des concentrations de COD, ce qui suggère que la présence du carbone organique a favorisé l’inactivation de MS2. L’expérience avec un IS élevé et un faible COD était la seule expérience qui a montré que l’inactivation de MS2 pouvait être dépendante du temps. Cependant, les résultats de la méthodologie GLUE ont indiqué que les modèles d’inactivation constante étaient capables de décrire toutes les expériences. Cela suggère que des séries temporelles d’inactivation de plus de deux mois étaient nécessaires pour fournir des conclusions concrètes concernant la dépendance temporelle de l’inactivation MS2 à 4 °C dans ces conditions expérimentales.

Los efectos de la fuerza iónica y la materia orgánica sobre la inactivación del virus a bajas temperaturas: estimación de la probabilidad generalizada de incertidumbre (GLUE) como alternativa a la optimización de los parámetros de mínimos cuadrados para la adaptación de modelos de inactivación de virus

Resumen

Este estudio examinó cómo la inactivación del bacteriófago MS2 en el agua se vio afectada por la fuerza iónica (IS) y el carbono orgánico disuelto (DOC) mediante experimentos de inactivación por lotes estáticos a 4 °C durante un período de dos meses. Las condiciones experimentales fueron características de un esquema de gestión operacional de recarga de acuíferos (MAR) en Uppsala, Suecia. Los datos experimentales se ajustaron con modelos de inactivación constantes y dependientes del tiempo utilizando dos métodos: (1) técnicas lineales y no lineales tradicionales de mínimos cuadrados; y (2) una técnica de estimación de parámetros basada en Monte Carlo llamada estimación de la probabilidad generalizada de incertidumbre (GLUE). Las metodologías de mínimos cuadrados y GLUE dieron estimaciones muy similares de los parámetros del modelo y su incertidumbre. Esto demuestra que se puede usar GLUE como una alternativa viable a las técnicas de estimación de parámetros de mínimos cuadrados tradicionales para la adaptación de modelos de inactivación de virus. Los resultados mostraron un ligero aumento en las tasas de inactivación constante después de un aumento en las concentraciones de DOC, lo que sugiere que la presencia de carbono orgánico aumentó la inactivación de MS2. El experimento con un alto IS y un bajo DOC fue el único experimento que mostró que la inactivación de MS2 puede haber sido dependiente del tiempo. Sin embargo, los resultados de la metodología GLUE indicaron que los modelos de inactivación constante fueron capaces de describir todos los experimentos. Esto sugirió que se necesitaban series de tiempo de inactivación de más de dos meses para proporcionar conclusiones concretas con respecto a la dependencia temporal de la inactivación de MS2 a 4 °C bajo estas condiciones experimentales.

低温下离子强度和有机物对病毒灭活的影响:一般可能性的不确定性估计法作为最小二乘参数最优化的替代来匹配病毒灭活模型

摘要

本研究利用时间为两个月、温度为4°C进行的静态批量试验,检查了水中的MS2噬菌体灭活是怎样受到离子强度和溶解有机碳影响的。试验条件具有瑞典Uppsala地区运行的管理的含水层补给计划的特征。采用两种方法使试验数据与恒定的和依赖于时间的灭活模型匹配:(1)传统的线性和非线性最小二乘技术;(2)被称为一般可能性的不确定性估计法的基于Monte-Carlo参数估算技术。最小二乘法和一般可能性的不确定性估计法给出了模型参数及其不确定性非常类似的估算值。这说明,一般可能性的不确定性估计法可在传统的最小二乘参数估算技术中作为可行的替代法,来匹配病毒灭活模型。结果显示,溶解有机碳含量增加之后,恒定的灭活率稍微提高,表明,有机碳的存在强化了MS2的灭活效果。高离子强度和低溶解有机碳的试验是唯一一个显示MS2灭活可能是依赖于时间的试验。然而,一般可能性的不确定性估计法得到的结果表明,恒定灭活模型能够描述所有的试验。这表明,为了提供在这些试验条件下温度为4°C时MS2灭活时间依赖关系的具体结论,需要长于两个月的灭活时序。

Os efeitos da força iônica e da matéria orgânica na inativação de vírus em baixas temperaturas: estimativa da incerteza de probabilidade generalizada (GLUE) como uma alternativa para a otimização dos parâmetros de quadrados mínimos para ajuste de modelos de inativação de vírus

Resumo

Esse estudo examinou como a inativação em água do bacteriófago MS2 foi afetada pela força iônica (FI) e carbono orgânico dissolvido (COD) utilizando experimentos de inativações de carga estática a 4 °C conduzido por um período de dois meses. As condições experimentais foram características de um esquema operacional de gerenciamento de recarga de aquífero (GRA) em Uppsala, Suécia. Dados experimentais foram ajustados com modelos de inativação constantes e dependentes no tempo utilizando dois métodos: (1) técnicas de quadrados mínimos lineares e não lineares tradicionais; e (2) uma técnica de estimativa de parâmetro baseada em Monte-Carlo chamada de estimativa da incerteza de probabilidade generalizada (GLUE). As metodologias dos quadrados mínimos e de GLUE forneceram estimativas similares dos parâmetros do modelo e suas incertezas. Isso demonstra que o GLUE pode ser usado como uma alternativa viável às técnicas de estimativa de parâmetros tradicionais de quadrados mínimos para ajustes de modelos de inativação de vírus. Os resultados mostraram um ligeiro aumento nas taxas de inativação constantes seguindo um aumento nas concentrações de COD, sugerindo que a presença de carbono orgânico melhora a inativação do MS2. O experimento com alta FI e baixo COD foi o único experimento onde foi mostrado que a inativação de MS2 pode ter dependido do tempo. Contudo, os resultados da metodologia GLUE indicaram que modelos de inativação constante foram capazes de descrever todos os experimentos. Este fato sugere que as séries temporais de inativação maiores que dois meses foram necessárias para fornecer conclusões concretas quanto a dependência temporal da inativação de MS2 a 4 °C sob essas condições experimentais.

Introduction

Waterborne outbreaks of gastroenteritis from groundwater sources continue to be a relatively common occurrence in developed countries in spite of the diligence paid to treating drinking-water (Brunkard et al. 2011; Craun et al. 2010; Riera-Montes et al. 2011; Zacheus and Miettinen 2011) and can result in large economic costs to local (Corso et al. 2003; Halonen et al. 2012; Larsson et al. 2013) and national economies (Hoffmann et al. 2012). Viruses are of particular concern to water mangers exploiting natural and managed groundwater resources as they can remain active for months in soil-water systems (DeBorde et al. 1998; Hurst et al. 1980), are capable of traveling large distances in groundwater (Keswick and Gerba 1980; Pang et al. 2005), and can be resistant to chlorine disinfection (Keswick et al. 1985).

Viruses are removed during soil passage through a combination of adsorption and inactivation mechanisms (Keswick and Gerba 1980). While adsorption is the dominant mechanism of virus removal in soil-water systems, inactivation of viruses becomes important when timescales longer than a few days are considered as is the case in field-scale studies (Schijven and Hassanizadeh 2000). The most important environmental factor affecting virus inactivation is temperature with lower inactivation rates at cold temperatures (Hurst et al. 1980; Yates and Yates 1987; Yates et al. 1985). The presence of natural organic matter has also shown to affect virus survival (Bixby and O’Brien 1979; Chattopadhyay et al. 2002; Foppen et al. 2006; Moore et al. 1982). For water managers exploiting natural or managed groundwater resources it is essential that virus inactivation is understood at low temperatures and in the presence of natural organic matter as prolonged virus survival is conducive under these conditions. This is especially true for water managers in boreal regions as high concentrations of natural organic matter in natural waters (Pastor et al. 2003) and cold temperatures are characteristic of these regions.

Mathematical models which describe virus inactivation in water are an important part of comprehensive models of the fate-and-transport of viruses in flowing soil-water systems. For these models virus inactivation rate parameters can be estimated by using batch inactivation studies. Virus inactivation is generally assumed to follow a first-order process where the inactivation rate of viruses is thought to be constant over time (Schijven and Hassanizadeh 2000). This assumption has been successfully applied to several studies modeling both virus inactivation in batch inactivation experiments (Bae and Schwab 2008; Yates et al. 1985) and more comprehensive studies of virus removal in flowing soil-water systems (Foppen and Schijven 2006; Kvitsand et al. 2015; Schijven et al. 2016). However, Hurst et al. (1992) argue that virus inactivation is a time-dependent process and is not adequately represented by a constant inactivation rate. Sim and Chrysikopoulos (1996) propose a pseudo-first-order representation of inactivation, wherein inactivation is a time-dependent process. Anders and Chrysikopoulos (2006) and Chrysikopoulos and Aravantinou (2012) examined the performance of both model formulations (constant and time-dependent inactivation) as a means to discern which model best described data from batch virus inactivation experiments. In both studies, the model which produced the lowest squared error was chosen to best represent the data; models with a time-dependent inactivation rate were best suited for the majority of the data.

Models in which constant virus inactivation is represented as a first-order process are typically fit to experimental data using linear regression or linear least-squares optimization techniques whereas more sophisticated nonlinear regression and least-squares optimization algorithms are required for fitting models in which virus inactivation is modeled as time-dependent and treated as a pseudo-first-order process. Microbial data, such as viral concentrations, derived from microbial plate counts of colony forming units (PFUs) are considered as a loose approximation of the actual number of active microbial units in a sample as these measurements are variable by nature (Sutton 2011). Applications of the method within food microbiology suggest that, at best, the magnitude of the uncertainty surrounding viral concentrations derived from PFU data are ±0.5 log10 PFU/ml assuming all steps were carried out in an accurate and consistent manner (Corry et al. 2007). Studies examining the removal of viruses during soil passage can be used by water managers to aid in prediction of water quality levels in effluent groundwater and/or for designing managed aquifer recharge (MAR) schemes (Pang 2008); however, the uncertainty inherent to microbial data is rarely considered when fitting models to experimental inactivation data.

A common way to examine uncertainty for virus inactivation experiments is by examining the experimental error by making a replicate measurement from a parallel and ideally identical experiment. In this way, the experimental uncertainty (or a range of likely virus concentrations) of a particular observation is usually estimated by the standard deviation assuming that replicate measurements should be distributed randomly about a most-likely value (the mean of the replicates; Box et al. 1978). This method, however, makes large prior assumptions about the likelihood distribution of microbial data as many replicate measurements would be needed to perform any meaningful statistical analysis. Only a few studies exist which attempt to characterize the probability distribution of errors associated with microbial measurements. Suggested error distributions include a Poisson distribution (Tomasiewicz et al. 1980; Corry et al. 2007), a log-normal distribution (Corry et al. 2007) or, in some instances, a negative binomial distribution (Jarvis 1989; Niemelä 1996). Unfortunately, the time and resources needed for measuring virus concentrations using biological methods restrict many experimenters from examining data errors in a statistically robust way. This seems to be the most likely reason that experimentalists do not directly consider data uncertainty when fitting mathematical models to microbial data and instead use replicate measurements as a means to estimate an average value for the microbial concentration; however, this is not to say that experimenters neglect to consider uncertainty in model predictions altogether. A recent study by Schijven et al. (2016) examined the long-term inactivation of bacteriophage PRD1 and was diligent in reporting the prediction intervals around model fits assuming that the model errors were normally distributed.

It is important to consider how uncertainties in virus concentration data can affect the fitting of virus inactivation models. This is especially true if experimental data is used to test the adequacy of different mathematical model structures describing virus inactivation, examine the effects of environmental factors influencing virus inactivation, or to guide water resource management. General likelihood uncertainty estimation (GLUE) is an uncertainty analysis framework that makes it possible to consider data uncertainty in model parameter fitting in-lieu of any formal measures of data likelihood. GLUE is a Monte-Carlo simulation based approach that allows the experimenter to choose their own representation of the data likelihood and criteria for accepting a model as adequate/behavioral in describing the data (Beven 2002a, b). GLUE also allows the experimenter to consider several behavioral parameter sets and model structures simultaneously allowing for ensemble-based predictions, thus aiding in the assessment of predictive uncertainty. GLUE is a well-established approach which has been successfully applied to several different environmental models of varying complexity (Beven and Binley 1992; Beven 1993; Binley and Beven 2003; Christiaens and Feyen 2001; Freer et al. 1996; Zhang et al. 2006); however, there have yet to be any applications of GLUE to studies of virus inactivation.

Although traditional statistical methods like linear regression or nonlinear least-squares algorithms also account for data uncertainty these methods often rely on several key assumptions. Some of these assumptions are that (1) the fitted model is “true”, (2) model errors follow a known distribution, (3) model errors are uncorrelated, and (4) there exists a global optimum fit for the experimental data. If all of these assumptions are not justified this can result in a false confidence in the choice of a “correct” model structure and in the estimates of model parameter and predictive uncertainties. The GLUE methodology attempts to overcome these shortfalls by allowing the experimenter to make very few assumptions regarding data error distributions and foregoes the requirement to consider the optimum model fit as being “true”. For these reasons, GLUE can be seen as an alternative method for fitting models of virus inactivation to experimental data because, at this point in time, a formal statistical representation of the uncertainty of microbial data is lacking.

The primary aim of this study is to examine how different assumptions regarding measurement errors affect parameter estimates in virus inactivation models by comparing linear and non-linear least-squares model fitting approaches to the GLUE methodology. Secondly, a full-factorial design experiment was carried out to assess how virus inactivation at cold temperatures (4 °C) is influenced by ionic strength (IS) and dissolved organic carbon (DOC). To the authors’ knowledge, this is the first study where uncertainty in microbial data has been explicitly accounted for in the fitting of virus inactivation models using the GLUE methodology. Results of this study will be useful for water managers who rely on virus inactivation mechanisms in groundwater as a means for treating waters contaminated with virus. This study is also intended to provide experimenters with an introduction to GLUE as an alternative method for estimating parameter and prediction uncertainties for models describing virus inactivation in water.

Materials and methods

A series of static batch inactivation experiments was conducted to test the effects of ionic strength IS and DOC on the inactivation of bacteriophage MS2 at 4 °C. Water and sand was gathered from a facility producing drinking water through MAR in Uppsala, Sweden as a means to examine how changes in water chemistry may affect virus inactivation during winter months. Experimental data was fit with two mathematical models of virus inactivation: one which describes the rate of virus inactivation as constant and another which describes the rate of virus inactivation as time-dependent.

Study site

Water samples used for the batch virus inactivation experiments were characteristic of water used at the Tunåsen MAR scheme in Uppsala, Sweden. Uppsala’s drinking water is taken from a confined esker aquifer filled with glacial sand and gravel deposits. The aquifer itself is a part of the Uppsala Esker formation, a 200-km-long glaciofluvial sedimentary deposit with an average width of 1.3 km (Morosini 1989). Groundwater levels are maintained through the artificial infiltration of surface water into the aquifer at a number of locations using basin infiltration methods; for a detailed description of MAR through basin infiltration consult Bouwer (2002). Surface water for infiltration at Tunåsen is taken from the nearby Fyris River and pumped through a fast-sand filter before being infiltrated. Groundwater is extracted from the esker ∼2 km away from the Tunåsen infiltration basins after being in the ground for about 90–110 days (Bergström 1986). Water for this study was sampled during the winter of 2014/2015. All of the water for the experiments was taken from the Tunåsen MAR scheme after passage through the fast-sand filter but before being pumped into the infiltration basins. The average winter (Nov.–Feb.) temperature of the water used for infiltration is 1.5 °C (Sveriges lantbruksuniversitet 2015).

Water

Water for the experiment was collected from the Tunåsen flow-division chamber prior to infiltration into the sand basins. The water was filtered twice; first through 1.6 μm to remove large particulates then through 0.45 μm to remove particulate organic matter and stored in a dark room at 4 °C. The filtered water was analyzed for its chemistry. Chemical characteristics for the water were F: 0.4 mg/L; Cl: 1

4 mg/L; NO3: 5 mg/L; Na+: 10 mg/L; Ca+: 73 mg/L; Mg2+: 6 mg/L; Fe2+: 0.2 mg/L; DOC: 17 mg/L; HCO3: 170 mg/L; color: 100 mg/L Pt; conductivity: 41 mS/m @25 °C; pH: 8.0. Ionic-strength of the water was estimated to be 7 mM using the concentrations of the aforementioned measured ions with the exception of Fe2+. The pH of the water equilibrated with the CO2 in the atmosphere and the HCO3− in the water at pH 8.0 as a result of the handling of the water; the pH of the water was not adjusted back to its original near neutral pH in order to reduce the effects of adsorption during the batch experiments.

Four distinct aqueous solutions were required for the experiment representing every combination of a high and low level of IS and DOC. Each solution was prepared in individual 1-L glass bottles. The IS of the water was adjusted through the addition of NaCl. For the two solutions with a high IS 100 mg of NaCl were added to 1 L of water and vigorously shaken for 30 s. The DOC concentration of the water was adjusted through the addition of Nordic Reservoir natural organic matter (NOM; IHSS Cat. No. 1R108N (IHSS 2011). For the two solutions with a high DOC water preparation was done in five, 200-ml Erlenmeyer flasks. 10 mg of Nordic Reservoir NOM was added to each flask, totaling 50 mg in 1 L, and flasks were agitated at 140 RPM for 20 h at room temperature. Solutions were then filtered through a 0.45-μm filter to remove particulate organic matter and then poured into 1-L glass jars. For the high IS, high DOC solution the water was prepared with the Nordic Reservoir NOM prior to the addition of NaCl. The original water was used as the low IS, low DOC solution. The DOC of the water was measured with Sievers 900 Portable TOC Analyzer (General Electric). The IS of the water was estimated using the concentrations of the measured ions for the raw water plus the additional NaCl. The characteristics of each solution along with the average winter values for the infiltrating water are presented in Table 1. Low levels of IS and DOC are consistent with those during average winter conditions of the water at the Tunåsen infiltration scheme and high levels are just under the maximum observed values. The addition of the Nordic Reservoir NOM resulted in a 2–3 μS/cm increase in the conductivity of the water, most likely as a result of the ash content of the NOM; however, this was not taken into account when estimating the ionic-strength of the solutions containing Nordic Reservoir NOM as no information was given in regards to the mineral composition of the ash. Nevertheless, this was deemed insignificant in regards to the estimation of the ionic strength of these solutions as the increase in conductivity was only 3–7% in response to the addition of the NOM.
Table 1

Chemical characteristics of the water used for MAR at Tunåsen and water used in the batch experiments

Water description

pH

Conductivity (mS/m 25 °C)

IS (mM)

DOC (mg/L)

Maximum observeda

8.0

66

-

36b

Winter average (Nov.– Feb.)a

7.5

42

-

16b

low IS, low DOC (e-d-)

8.0

41

7.0

17

high IS, low DOC (e+d-)

7.9

62

8.6

17

low IS, high DOC (e-d+)

7.9

44

7.0 c

31

high IS, high DOC (e+d+)

7.9

64

8.7 c

31

aData gathered just downstream of the pump intake to the Tunåsen infiltration basins (station Fyrisån Klastorp) for the years 1993–2011 (Sveriges lantbruksuniversitet 2015)

bReported as total organic carbon (TOC)

cDoes not account for the small change in IS due to the addition of the Nordic Reservoir natural organic matter (NOM)

Viruses and viral assay techniques

Bacteriophage MS2 was used for the batch inactivation studies as it is considered an adequate model of enteric viruses (Gerba 2006; IAWPRC 1991) and because it tends to provide water managers with a more conservative estimate of virus removal than other bacteriophages (Pang 2008). MS2 has a diameter of 25 nm and an isoelectric point (pI) of 3.9 (Overby et al. 1966). Phage was obtained for this study from the American Type Culture Collection (ATCC 15597B1) and grown on bacterial lawns of Salmonella typhimurium WG49 (ATCC 700730). Phage stock solutions were prepared by first inoculating 50 ml of beef extract nutrient broth (SVA Art. No. B311040) with a single “male” S. typhimurium colony and incubating the solution at 37 °C until bacterial growth was at a high rate of multiplicity. The solution was then infected with 500 μl of high-concentration MS2 solution and incubated for 24 h at 37 °C. Phage purification was achieved by then centrifuging the solution at 2,000 g for 20 min and filtering the supernatant through 0.45-μm filters. Virus stock solutions were stored in 50-ml polypropylene tubes (Sarstedt Art. No. 62.547). Viral assays were completed using the double agar overlay method (Adams 1959) using agar plates filled with blood-agar-base No. 2 (SVA Art. No. B331020). Raw data for the colony counts and the equations used to estimate MS2 concentrations based on the plate counts can be found in the electronic supplementary material (ESM).

Experimental setup

A two-level, two-treatment full-factorial design was used to test the effects of high and low levels of IS and DOC on MS2 inactivation. A total of four unique experiments (each with an identical replicate experiment) were run in parallel in order to test all possible combinations of high and low levels of the treatments considered. Experiments were given names according to their respective combinations of treatments and levels. Experiments with high and low levels of IS were denoted with “e+” and “e-̄” respectively. Experiments with high and low levels of DOC were denoted with “d+” and “d-” respectively—for example, an experiment with a low level of IS and a high level of DOC was given the name “e-d+”.

Static batch inactivation experiments were conducted in Pyrex 11.5-ml glass tubes with Teflon screw-caps as the batch reactors. Glass tubes were washed with detergent, rinsed, autoclave sterilized, and then heat-sterilized at 180 °C for 2.5 h. Teflon screw-caps were washed with detergent, rinsed, and then autoclaved sterilized. Prior to the filling of the batch reactors, the aqueous background solutions received 1 ml of virus stock solution and were shaken vigorously for 10 s in order to prepare the virus suspensions. Virus suspensions were sampled for measurement of the initial concentration C0 immediately before the filling of the batch reactors.

Tubes were filled such that the aqueous solution formed a convex surface due to surface tension at the tube opening prior to screwing on the Teflon screw cap in order to eliminate the presence of air bubbles in the batch reactor, which resulted in each tube receiving 13.4 ml of virus suspension. The capacity of each 11.5-ml tube was estimated well below the level to which it was filled, thus resulting in the discrepancy between the experimental volume and the manufacturer’s specifications. For each experiment, 20 batch reactors were deployed in a dark climate-controlled chamber at 4 °C in order to examine inactivation during winter months. Two reactors from each experiment (measurement and replicate) were randomly chosen at each of the ten measurement time-steps (4.4, 8.6, 11.5, 15.6, 18.5, 21.5, 25.4, 30.5, 33.5, and 63.4 days) filtered through a 0.45-μm PES membrane syringe filter (Sarstedt Art. No. 83.1826), and then measured for MS2 concentration. Glass tubes were discarded after each measurement.

Virus inactivation models

The governing equation used in this study to describe virus inactivation in batch systems is shown in Eq. (1):
$$ \frac{dC(t)}{dt}=-\lambda (t) C(t) $$
(1)
where t (day) is the time step under consideration, C(t) (PFU/ml) is the concentration of virus at time t, and λ(t) (per day) is the inactivation rate at time t. Under the assumption that viral inactivation is constant then λ(t) = λ then the solution to Eq. (1) can be written as shown in Eq. (2):
$$ \ln \left(\frac{C(t)}{C_0}\right)=-\lambda t $$
(2)
where C0 (PFU/ml) is the initial virus concentration. Time-dependent virus inactivation assumes that different sub-populations of the same virus will be inactivated at different rates. One of the reasons for this may be due to virus aggregation (Grant 1995). Virus aggregates may inactivate at a different rate than individual viruses in suspension due to viruses on the surface layer of the aggregate acting as a protective barrier for the viruses towards the center of the aggregate (Sharp 1965). Modeling of the time-dependent inactivation is done using a pseudo-first-order rate expression proposed by Sim and Chrysikopoulos (1996) described in Eq. (3):
$$ \lambda (t)={\lambda}_0 \exp \left(-\alpha t\right) $$
(3)
where λ0 (per day) is the initial inactivation rate and α (per day) is the resistivity coefficient. Under this assumption, the solution to Eq. (1) can be written as shown in Eq. (4):
$$ \ln \left(\frac{C(t)}{C_0}\right)=\frac{\lambda_0}{\alpha}\left[ \exp \left(-\alpha t\right)-1\right] $$
(4)

The resistivity coefficient α (α > 0) governs the time-dependency of inactivation and can be thought of as a measure of how sensitive different sub-populations of virus are to inactivation (Sim and Chrysikopoulos 1996); a larger value of α indicates the presence a more resistant subpopulation of virus resulting in higher concentrations for longer periods of time. The rate at which the most sensitive subpopulation of virus inactivates is represented by the initial inactivation rate λ0.

Linear and nonlinear least-squares parameter estimation

Both models were initially fit to each experimental dataset using linear least-squares algorithms for fitting of Eq. (2) and a non-linear least-squares approach for the fitting of Eq. (4). Model fits were completed using MATLAB (The MathWorks Inc. 2014). The functions “lsqlin” and “lsqnonlin” were used for the linear and non-linear least-squares optimization respectively. The root-mean-square error (RMSE) was used as a goodness of fit measure for the model fitting and is shown in Eq. (5):
$$ \mathrm{RMSE}=\sqrt{n^{-1}{\displaystyle \sum_{i=1}^n}{\left(\ {\overline{Y}}_i-{F}_i\right)}^2} $$
(5)
where n is the total number of i time-steps, \( {\overline{Y}}_i \) (PFU/ml) is the average concentration of the replicate measurements at time-step i, and Fi (PFU/ml) is the model prediction at time-step i. Estimates of λ, λ0, and α are made for each experiment individually and the model producing the lowest RMSE is considered to best describe the data.

Models are fit using data up to 33.5 days and the examined in regards to their ability to predict the concentration at 63 days. This provides an opportunity to comment on the extent to which models fit to approximately 1 month of inactivation data are capable of prediction at time-scales larger than that used for model fitting.

GLUE model fitting

Uncertainty of MS2 concentration data

The uncertainty bounds at each time step were estimated by assessing the uncertainty of each individual measurement and the difference between the replicate measurements. The measurement uncertainty was estimated according to the “law of propagation uncertainty” which states that the uncertainty of a quantity can be expressed as a function of all of the components used to derive that final quantity (Taylor and Kuyatt 1994). Niemelä (2003) suggests that four components contribute significantly to the overall uncertainty of the final measurement of virus when using the double-agar-layer assay method: the liquid volume used for the plate inoculation, the random scatter of particles in the inoculation volume, the counting of colonies on the inoculated plate, and the extent to which a sample was diluted (dilution factor) before it is plated. Uncertainty estimations for the viral concentration measurements in this study were performed following the guidelines put forward in Niemelä (2003). All components of uncertainty, except the uncertainty of counting colonies on the plates, were accounted for when calculating the total uncertainty. The procedure followed for counting PFUs required that each counted colony was marked so an assessment of uncertainty in the counting was not possible; however, the uncertainty of the colony counts may be accounted for in other uncertainty components related to the colony count (Niemelä 2003). For this study it was assumed that the uncertainty pertaining to the random scatter of the particles in the inoculation volume accounted for the uncertainty of the colony counts as both components are based on the number of colonies counted on the inoculated plate.

The lower (LL) and upper (UL) limits of the total uncertainty for the concentration estimate at each time-step i were estimated by considering the measurement uncertainty for both replicates. By accounting for uncertainty in this manner both the experimental and measurement uncertainty were accounted for at each data-point. Values for UL and LL were found following the procedure outlined in Niemelä (2003). A detailed explanation of the equations used to calculate the uncertainty bounds is presented in the ESM.

Parameter sampling for GLUE

Parameter fitting and model evaluation was done using a Monte-Carlo approach called GLUE, wherein forward simulations of the inactivation models are produced using randomly generated parameter combinations termed “parameter sets”. The performance of each model was examined based on the squared-errors between the predicted concentration and the uncertainty bounds for the measured data. The “behavioral” models were chosen according to a pre-determined level of acceptance based on the squared-errors of the simulations using a modified form of the RMSE explained in the following. Forward simulations of the constant inactivation model were done by solving Eq. (2) for C(t) and sampling random combinations of the initial concentration C0 and inactivation rate λ. Forward simulations of the time-dependent inactivation model were done by solving Eq. (4) for C(t) and sampling random combinations of the initial concentration C0, the initial inactivation rate λ0, and the resistivity coefficient α. All parameter sets were generated by randomly sampling individual parameters between a predetermined upper and lower limit. The sampling limits for C0 were set equal to 1.0 × 104 and 4.0 × 1013. The sampling ranges for λ, λ0, and α were set to values assumed to overestimate the realistic range of each parameter in order to ensure an adequate sampling of the model space for the forward simulations. This was done by assuming that the models to be considered as behavioral would not be able to: (1) predict MS2 concentrations that were less than one percent of C0 after the first time-step; and (2) predict concentrations at the final time step that were greater than 99% of C0. The sampling range for α was set to 3.0 × 10–4–1.03 /day and the sampling range for both λ and λ0 was set to 3.0 × 10−4–14.5 /day. A total of 105 forward simulations were carried out for the constant inactivation rate model using 105 unique parameter sets of C0 and λ. The same was done for the time-dependent inactivation rate model using unique parameter sets of C0, λ0, and α. The assumptions and equations used to calculate the parameter ranges are further discussed in the ESM.

Performance of forward simulations

The RMSE for the forward simulations within the GLUE framework (RMSE′) accounts for the uncertainty of the measured data by calculating the error of the model based on the range of likely values rather than a point estimate using a modified form of Eq. (5) presented in Harmel and Smith (2007). The RMSE′ was calculated using Eq. (6):
$$ \mathrm{RMSE}^{\prime }=\sqrt{n^{-1}{\displaystyle \sum_{i=1}^n}{\left({E}_{\mathrm{i}}\right)}^2} $$
(6)
where Ei (PFU/ml) is the error between the function evaluation and the nearest uncertainty bound as defined by Eq. (7):
$$ {E}_i=\left\{\begin{array}{ccc}\hfill 0\hfill & \hfill \mathrm{if}\hfill & \hfill \mathrm{L}{\mathrm{L}}_i\le {F}_i\le \mathrm{U}{\mathrm{L}}_i\hfill \\ {}\hfill \mathrm{L}{\mathrm{L}}_i-{F}_i\hfill & \hfill \mathrm{if}\hfill & \hfill {F}_i<\mathrm{L}{\mathrm{L}}_i\hfill \\ {}\hfill {F}_i-\mathrm{U}{\mathrm{L}}_i\hfill & \hfill \mathrm{if}\hfill & \hfill {F}_i>\mathrm{U}{\mathrm{L}}_i\hfill \end{array}\right. $$
(7)
where Fi (PFU/ml) was the function evaluation at time-step i. Under this assumption the true-value of the measured concentration at time-step i was equally likely to exist anywhere between the estimated upper and lower uncertainty limits. A graphical representation of the calculation of Ei is shown in Fig. S1 in the ESM.

The RMSE′ for each forward simulation was calculated in the same model space as that used for linear and nonlinear least-squares optimization by dividing the results by the point-estimate of C0 for that particular experiment. The criteria for model acceptance were unique to each experiment and were set equal to the lowest RMSE achieved between the linear and nonlinear least-squares optimization techniques. All parameter sets producing an RMSE′ value lower than the threshold RMSE were considered behavioral thus giving a range of the parameters λ, λ0, and α capable of describing the data to a degree as good or better than those obtained through linear and nonlinear least-squares optimization. By choosing the acceptance criteria in this way, it was easier to compare parameter estimates from the GLUE methodology to those from the linear and nonlinear least-squares optimization model fitting techniques. The parameters producing the lowest RMSE′ were termed λ′, λ0′, and α′.

GLUE prediction uncertainty

In this study, the prediction uncertainty for GLUE is estimated using the maximum and minimum MS2 concentrations predicted by the ensemble of behavioral models at each time-step. The GLUE methodology, as applied in this study, lacks a formal statistical definition of the model errors and the MS2 concentration data is considered to be equally likely between the upper UL and lower LL uncertainty bounds unique to each data-point. Therefore, model predictions between the upper and lower prediction ‘quantiles’ (as they are commonly called in the GLUE methodology) are considered to all be equally likely to some degree.

Results

Batch inactivation data

The lowest concentration of MS2 after 34 and 63 days was observed in experiments e+d+ and e-d+ respectively (Table 2; Fig. 1). Experiment e+d- had the highest concentration of MS2 after 34 and 63 days.
Table 2

Batch inactivation time-series data for all experiments and the average data uncertainty for each experiment PFU

 

e+d+

e-d+

e+d-

e-d-

Time (days)

\( \overline{Y} \) (PFU/ml)

LL / UL

(PFU/ml)

\( \overline{Y} \)(PFU/ml)

LL / UL

(PFU/ml)

\( \overline{Y} \)

(PFU/ml)

LL / UL

(PFU/ml)

\( \overline{Y} \)

(PFU/ml)

LL / UL

(PFU/ml)

0

7.2 × 108

4.9 × 108

9.5 × 108

1.2 × 109

8.6 × 108

1.4 × 109

1.9 × 109

1.6 × 109

2.3 × 109

2.2 × 108

9.1 × 107

3.4 × 108

4.5

1.1 × 108

7.8 × 107

1.5 × 108

4.8 × 107

4.4 × 107

5.3 × 107

9.1 × 107

6.8 × 107

1.1 × 108

4.8 × 107

4.4 × 107

5.3 × 107

8.6

7.2 × 107

3.1 × 107

1.2 × 108

4.9 × 107

4.2 × 107

5.5 × 107

4.4 × 107

3.8 × 107

5.0 × 107

5.0 × 107

3.3 × 107

6.8 × 107

11.5

1.6 × 108

1.5 × 108

1.8 × 108

9.2 × 107

7.6 × 107

1.1 × 108

1.0 × 108

8.7 × 107

1.1 × 108

3.5 × 107

2.7 × 107

4.3 × 107

15.6

8.6 × 106

7.4 × 106

9.9 × 106

1.5 × 107

9.8 × 106

2.0 × 107

1.2 × 107

1.0 × 107

1.5 × 107

1.8 × 107

1.6 × 107

1.9 × 107

18.5

3.9 × 107

3.4 × 107

4.8 × 107

1.1 × 107

8.0 × 106

1.4 × 107

1.2 × 107

1.1 × 107

1.4 × 107

2.8 × 107

1.6 × 107

4.0 × 107

21.5

1.2 × 107

7.2 × 106

1.6 × 107

1.4 × 107

1.2 × 107

1.6 × 107

1.4 × 107

1.0 × 107

1.8 × 107

1.3 × 107

1.1 × 107

1.4 × 107

25.4

2.5 × 106

1.8 × 106

3.2 × 106

2.1 × 106

1.4 × 106

2.8 × 106

1.1 × 106

7.2 × 105

1.4 × 106

1.2 × 106

7.2 × 105

1.6 × 106

30.5

1.1 × 106

9.2 × 105

1.3 × 106

7.4 × 105

6.3 × 105

8.6 × 105

2.3 × 106

1.3 × 106

3.4 × 106

1.1 × 106

8.7 × 105

1.4 × 106

33.5

2.0 × 105a

1.4 × 105b

2.5 × 105b

4.2 × 105a

3.4 × 105b

4.9 × 105b

5.7 × 105

1.8 × 105

9.9 × 105

4.0 × 105 d

3.3 × 105b

4.8 × 105b

63.4

2.0 × 103

1.3 × 103

2.8 × 103

5.0 × 102

3.2 × 102

7.0 × 102

1.8 × 104

7.0 × 103

3.1 × 104

2.5 × 103

2.1 × 103

2.9 × 103

Ū ′c

-

0.25

-

0.19

-

0.26

-

0.24

a\( \overline{Y} \) based on one measurement (replicate measurement failed)

bUL and LL calculated based on one measurement using Eq. (S10) in ESM

c\( \overline{U}^{\prime }=\frac{1}{n}{\displaystyle \sum_{\mathrm{i}=1}^n}\left[{ \log}_{10}\left({\mathrm{UL}}_{\mathrm{i}}\right)-{ \log}_{10}\left({\mathrm{LL}}_{\mathrm{i}}\right)\right] \); average order-of-magnitude difference between UL and LL

Fig. 1

Batch experiment time-series data (white circles), data uncertainty bounds (red bars), and optimum models from linear and nonlinear least-squares model fitting (dashed line) using data up to time-step t = 33.5 days (vertical dotted line); uncertainty bounds that do not appear are smaller than the plotted point

Linear and nonlinear least-squares parameter estimation

For all experiments except e+d- the model of constant inactivation performed better than the model of time-dependent inactivation, according the RMSE, when using the first 33.5 days of data for fitting models using linear least-squares parameter estimation (Table 3). Models which produced the lowest RMSE were considered to best represent the data and are plotted in Fig. 1. For the experiments e+d+, e-d+, and e-d-, models of constant inactivation best explained the data. For these experiments the first-order inactivation rate λ was estimated to be higher in solutions with a high DOC. The magnitudes of the 95% confidence intervals for estimates of λ were similar for all experiments. For experiment e+d-, the model of time-dependent inactivation performed best (Table 3); however, the lower bound of the 95% confidence interval surrounding the resistivity coefficient α was negative, which is not considered to be possible under the assumptions used in the formulation of Eq. (4). For all experiments, the optimum models performed well in predicting the MS2 concentration at 63 days (Fig. 1).
Table 3

Optimum parameters found using linear and nonlinear least-squares, their corresponding 95% confidence intervals (CI), and the RMSE of the optimum model; RMSE threshold for GLUE parameter estimation is highlighted in italic

 

Constant inactivation; Eq. (2)

Time-dependent inactivation; Eq. (4)

Exp. name

RMSE

λ

95% CI

C0a

95% CI

RMSE

λ0

95% CI

α

95% CI

C0a

95% CI

(per day)

(PFU/ml)

(per day)

(per day)

(PFU/ml)

e+d+

0.80

0.20

0.17

0.24

3.8 × 108

4.9 × 107

2.9 × 109

0.84

0.22

0.11

0.33

0.003

−0.01

0.02

5.5 × 108

2.4 × 107

1.3 × 1010

e-d+

0.71

0.22

0.19

0.24

2.0 × 108

3.3 × 107

1.2 × 109

0.73

0.20

0.11

0.28

−0.003

−0.02

0.01

1.3 × 108

8.9 × 106

1.8 × 109

e+d-

0.97

0.17

0.13

0.21

3.3 × 107

2.8 × 106

3.8 × 108

0.82

0.28

0.14

0.42

0.02

−0.002

0.04

2.8 × 108

1.1 × 107

7.1 × 109

e-d-

0.64

0.18

0.15

0.21

2.9 × 108

5.9 × 107

1.5 × 109

0.65

0.16

0.08

0.23

−0.004

−0.02

0.01

1.8 × 108

1.7 × 107

1.9 × 109

aValues of C0 corresponding to the estimation of the intercept of the optimum model

Data uncertainty

Results of the data uncertainty are presented in Table 2 and plotted in Fig. 1 and Fig. 3. The average order-of-magnitude difference between UL and LL for all experiments was 0.24 log10 PFU/ml (Table 2). The uncertainty ranges for MS2 concentrations were similar for all experiments except for experiment e-d+ which had a noticeably lower average uncertainty (Table 2). The lowest absolute data uncertainty for all experiments occurred for the measurement at the last measurement (63.4 days). The uncertainty bounds at each time step for each experiment accounts for both the uncertainty in the measurement of the replicate data and the difference between the replicate measurements. Values for the individual contributions to the total uncertainty for each measurement can be found in the ESM.

GLUE parameter estimation

Results of the GLUE parameter estimation pertaining to the best performing model structure (constant verses time-dependent inactivation) for each set of experimental data agree with those found using the linear and non-linear least-squares methods (Table 4); however, behavioral parameter sets for both the constant and time-dependent inactivation models were found for all experiments (Table 4; Fig. 2). For all experiments, the RMSE′ value found using GLUE was lower than the RMSE corresponding to the optimum models found using the linear and non-linear least-squares methods. This was a result of calculating squared errors using the upper UL and lower limit LL of the data at each time step in the RMSE′ value rather than using a point-estimate as was done for the RMSE. For experiments e+d+, e-d+, and e-d- the estimates of the constant inactivation rate λ from the linear least-squares approach were essentially equal to the values of λ′ attained from GLUE (<5% difference). A similar result was seen for experiment e+d- when comparing estimates of the initial inactivation rate λ0 and the resistivity coefficient α from the non-linear least squares approach to the values of λ0′ and α′ attained from GLUE. The 95% confidence intervals for λ, λ0, and α estimated from the linear and non-linear least-squares methodologies are also similar to the behavioral ranges of each respective parameter attained from GLUE (Tables 34). For the resistivity coefficient α, all behavioral values were well below the upper limit of the sampling range but behavioral values were found at the lower limit of the sampling range for all experiments (Table 4; Fig. 2). This suggests that behavioral values of α are likely to exist below this value; however, the lower limit of the sampling range for α was set to 3.0 × 10-4 which, in this study, is considered to be essentially equal to zero.
Table 4

GLUE estimation of parameters producing the lowest RMSE′ and the behavioral ranges of the parameters

 

Constant inactivation; Eq. (2)

Time-dependent inactivation; Eq. (4)

Exp. name

Best RMSE′

λ

Range

C0a

Range

Best RMSE′

λ0

Range

α

Range

C0a

Range

(per day)

(PFU/ml)

(per day)

(per day)

(PFU/ml)

e+d+

0.57

0.21

0.16

0.24

6.1 × 108

2.0 × 108

1.5 × 109

0.57

0.24

0.18

0.37

0.003

∼0b

0.02

9.2 × 108

2.2 × 108

2.8 × 109

e-d+

0.53

0.21

0.18

0.25

4.8 × 108

2.1 × 108

1.1 × 109

0.55

0.22

0.19

0.28

0.002

∼0b

0.01

5.9 × 108

2.8 × 108

1.2 × 109

e+d-

0.63

0.18

0.14

0.21

4.2 × 108

1.8 × 108

9.1 × 108

0.49

0.30

0.15

0.53

0.02

∼0b

0.05

1.0 × 109

1.7 × 108

4.9 × 109

e-d-

0.40

0.18

0.14

0.21

2.3 × 108

8.4 × 107

6.5 × 108

0.41

0.18

0.15

0.24

0.001

∼0b

0.01

2.3 × 108

1.1 × 108

7.5 × 108

aBehavioral values of C0 corresponding to the intercept of the behavioral model

bBehavioral values of α were found at the lower sampling limit

Fig. 2

Scatter-plots of sampled parameters versus the RMSE′ of the Monte-Carlo simulations for all experiments showing the non-behavioral runs (grey dots), behavioral runs (black dots), the threshold RMSE used to determine behavioral models (horizontal dashed line), and the optimal parameter estimates from the linear and nonlinear least-squares optimization step (black ×)

GLUE prediction uncertainty

Prediction quantiles were calculated using all of the behavioral parameter sets from each individual model structure (Fig. 3). The prediction quantiles for both model structures (constant and time-dependent inactivation) were capable of capturing the MS2 concentrations after both 34 and 63 days for all experiments except for experiment e+d+ (Fig. 3). Prediction quantiles for the time-dependent model were capable of capturing every data-point for experiment e+d-; however, this was the only instance where the prediction quantiles were capable of capturing all of the data for either model structure.
Fig. 3

Average experimental data (white circles), data uncertainty bounds (red bars), and prediction quantiles for models of constant (dashed lines) and time-dependent inactivation (dot-dashed lines); models were fit using data up to time-step t = 33.5 days (vertical dotted line); uncertainty bounds that do not appear are smaller than the plotted point

Discussion

Effects of treatment on virus inactivation

The optimal and best performing values of the constant inactivation rates λ and λ′, estimated using linear least squares parameter estimation and GLUE respectively, suggest that MS2 inactivation was slightly enhanced in the presence of high levels of DOC (Tables 34). Chattopadhyay et al. (2002) also found that the constant inactivation rates of bacteriophages T2 and φX174 increased after the addition of natural humic materials; however, an examination of the uncertainty surrounding the estimates of λ and λ′ in the present study (95% CI and the range columns in Tables 34 respectively) reveals that the differences between the values of λ and λ′ across solutions with a low and a high level of DOC are less significant. A review by Hurst (1988) found that the total organic carbon content of a solution had a statistically significant effect on virus survival at concentrations between 1 and 17 mg/L. In the present study, DOC concentrations changed between 17 and 31 mg/L (Table 2), meaning that DOC concentrations may have already been too high to produce any significant effects on MS2 inactivation between the high and low levels of DOC used in the experiments. This suggests that the natural variation in the DOC for the infiltration water used at the Tunåsen MAR scheme will not significantly affect virus inactivation at 4 °C.

Optimum values of the constant inactivation rate λ, estimated using linear least-squares parameter estimation, showed a slight decrease as the solution IS increased from 7.0 to 8.6 mM (Table 3). However, an examination of the uncertainties surrounding the estimates of λ (95% CI column in Table 3) suggests that the change is likely insignificant. This is also seen when examining the best performing values of the constant inactivation rate λ′ estimated using GLUE as the change in IS between the low and high levels results in no change in the constant inactivation rate.

Estimates of the constant inactivation rate attained using both the linear least-squares and GLUE methods of parameter estimation (λ and λ′) ranged from 0.13 to 0.24 /day (Tables 34). A similar study by Chrysikopoulos and Aravantinou (2012) which investigated the inactivation of bacteriophage MS2 in solution using static batch experiments at 4 °C estimated a constant inactivation rate ranging from 0.013 to 0.39 /day over 15 individual experiments with an average constant inactivation rate of 0.062 /day. Results from the present study fell well within this range; however, the experiments done by Chrysikopoulos and Aravantinou (2012) used phosphate buffered saline solution and examined inactivation of MS2 using initial virus concentrations which ranged from ∼103 to ∼108 PFU/ml; therefore, their estimates of the constant inactivation rate are not directly comparable to those found in the present study. Schijven et al. (2000) examined the removal of MS2 by deep well injection. Their estimates of the constant inactivation rate of MS2 in the injection water ranged from 0.039 to 0.081 /day at 12 °C. Estimates of the constant inactivation rate in the present study were two to six times higher than those found by Schijven et al. (2000). The IS and total organic carbon (TOC) content of the infiltration water was not directly reported by Schijven et al. (2000); however, results from chemical analyses of the groundwater completed during the declogging of the injection wells suggest that TOC concentrations of the water did not exceed 4.3 mg/L. In the present study, DOC ranged from 17 to 31 mg/L. The difference in constant inactivation rates for MS2 in the present study and those reported by Schijven et al. (2000) may have been due to the large differences in the organic carbon content of the solutions used in the inactivation experiments.

Data from the experiment in the high IS, low DOC solution (e+d-) were the only set of data best represented by the model of time-dependent inactivation Eq. (4) when models were fit to the first month of data (Tables 34; Fig. 1). All other experiments were best represented by models of constant inactivation Eq. (2). This suggests that a model describing constant inactivation may be appropriate in most cases for describing MS2 inactivation at 4 °C at the Tunåsen MAR scheme. Schijven et al. (2016) investigated the inactivation of bacteriophage PRD1 as affected by pH and the addition of NaCl and CaCl2. Their results found that PRD1 inactivation tended to become more non-linear as IS increased due to the addition of NaCl at pH 8 and that the data were best fit with models that assumed there were two subpopulations of PRD1 which inactivated at different rates. In the present study, the non-linearity of the MS2 inactivation was captured by assuming that inactivation was time-dependent proportional to the resistivity coefficient α. The coefficient α is meant to represent the presence of a subpopulation of virus more resistant to inactivation. These results suggest that the addition of NaCl may result in the formation of a subpopulation of virus that is more resistant to inactivation; however, as noted by Schijven et al. (2016), there is not theory available at this point in time which can explain this effect.

Comparison of least-squares and GLUE parameter estimation

The optimum values and 95% confidence intervals of λ, λ0 and α estimated using least-squares parameter estimation were nearly identical to the best performing values and behavioral ranges of λ′, λ0′, and α′ estimated using GLUE (Tables 34). GLUE, however, was able to estimate the uncertainty surrounding the parameters without assuming that the model errors were normally distributed. Within GLUE, the criteria for model acceptance were unique to each experiment and were set equal to the lowest RMSE achieved between the linear and nonlinear least-squares optimization techniques (Table 3). Model performance was then assessed using the RMSE′ value which calculated the model error using the distance between the upper UL and lower bounds LL for each data point rather than using a point estimate of the data. All parameter sets producing an RMSE′ value lower than the threshold RMSE were considered behavioral. This resulted in a range of the parameters λ, λ0, and α capable of describing the data to a degree as good or better than those obtained through linear and nonlinear least-squares optimization. By doing this, the uncertainty of the concentration data was treated such that the true-value of the MS2 concentration at any given time step was assumed to be equally likely between UL and LL. The uncertainty of the parameter estimates is estimated simply by examining the range of behavioral parameters. The traditional approach to least-squares parameter optimization estimates parameter uncertainty (95% confidence intervals) by assuming that the distribution of the model errors is normally distributed and makes similar assumptions about the errors surrounding the data. This assumption may not be justifiable as, at this point in time, no studies have been able to produce any reliable conclusions regarding the distribution of errors surrounding measurements of virus concentrations achieved using the double agar overlay method. In the current study, GLUE was capable of estimating the uncertainty surrounding model parameters without having to make large assumptions about the model and data errors.

Comparison of constant and time-dependent inactivation model structure

For the least-squares optimization approach, it was clear that a model of constant inactivation was best suited for all but experiment e+d- when using the RMSE to judge model performance. However, the discussion regarding which model structure is appropriate for describing the inactivation data from experiment e+d- became more nuanced when examining the results from GLUE. Results from GLUE indicated that several behavioral parameter sets existed for both model structures across all experiments, which implied that a model of constant inactivation was adequate for all of the experiments in this study, including e+d-, once data uncertainty was considered in the model parameter estimation. A decision regarding which model structure is most appropriate to capture a given set of data needs to take additional steps as, within GLUE, all behavioral parameter sets are considered likely to some degree. Within GLUE, this can be done by (1) adjusting the acceptance criteria of what should be deemed a behavioral model such that only one model structure produces behavioral parameter sets; and/or (2) examining the capabilities of the ensemble of behavioral parameter sets to predict future events.

Model prediction

Models of constant and time-dependent inactivation were fit to the first 33.5 days of MS2 inactivation data. An additional measurement was taken after 2 months in order to assess the predictive capacity of the models. The optimal models fit using the linear and non-linear least-squares methodologies were all capable of predicting the concentration of MS2 after 2 months (Fig. 1). Experiment e+d- was the only experiment for which a model of time-dependent inactivation was chosen to best represent the data based on the RMSE. Within GLUE, behavioral parameter sets existed for both the constant and time-dependent inactivation model structures for all experiments. The upper and lower limits for which the ensembles of behavioral models were capable of predicting the concentration of MS2 at each time step are shown in Fig. 3. These limits are commonly called prediction quantiles within the GLUE methodology. The prediction quantiles showed that the behavioral ensembles of both constant and time-dependent models were capable of capturing the MS2 concentration after 2 months (Fig. 3). This suggests that, once uncertainty in the MS2 data is considered, models of constant inactivation estimated using 1 month of inactivation data may be sufficient for predicting future virus concentrations in the Tunåsen infiltration water at 4 °C. The ensemble of constant inactivation models is not capable of capturing all of the MS2 concentration data for experiment e+d- while the ensemble time-dependent models is capable of capturing all data points (Fig. 3). This further suggests that inactivation may be a time-dependent process in this instance; however, the uncertainty surrounding the initial inactivation rate λ0 and resistivity coefficient α for experiment e+d- is relatively large (Fig. 2). Many of the behavioral values of α exist at near the lower sampling limit of 3.0 × 10–4 /day which, in this case, is assumed to be essentially zero (Table 4; Fig. 2). Inactivation is essentially independent of time according to Eq. (4) when α is this small. One month of MS2 inactivation data was not sufficient for trying to determine if inactivation was or was not a time-dependent process for experiment e+d- and a longer time series should be used which would be able to further demonstrate any non-linearity in the data.

Interdependency of λ0 and α

The resistivity coefficient α governs the time-dependency of inactivation and can be thought of as a measure of how sensitive different sub-populations of virus are to inactivation (Sim and Chrysikopoulos 1996). The rate at which the most sensitive subpopulation of virus inactivates is represented by the initial inactivation rate λ0. The sampling range for λ0 (3.0 × 10−4 –14.5 /day) is adequate in describing the entirety of behavioral models for the model of time-dependent inactivation; however, this is a direct result of the interdependency of the two parameters λ0 and α in Eq. (4). The exponential decay portion in Eq. (4) approaches –1 as both α and t increases and solutions to the equation can be approximated by –λ0/α. An analysis of the behavioral parameter sets of λ0 and α for behavioral models fit to experimental data from experiment e+d- reveals a strong linear dependence between the two parameters (Fig. 4). This demonstrates that the range of behavioral values of λ0 may become entirely dependent on the range of behavioral values of α as inactivation curves become more non-linear. For experimental data that appear to reach a relative steady-state in viral concentration, Eq. (4) will approach –λ0/α at earlier time-steps and the range of behavioral values of λ0 and α will become quite large as model fits will be increasingly dependent on the value of their ratio only. For experiment e+d-, the beginnings of this behavior is clearly reflected in the ranges of behavioral λ0 and α values as they are much larger than those for the other experiments (Fig. 2). In order to overcome this problem, virus inactivation experiments which demonstrate a highly non-linear behavior early on should be carried out for a period of time which is long enough to demonstrate that virus concentrations are continually decreasing and do not, in fact, reach a steady-state concentration. This relationship between λ0 and α also highlights the need to conduct more dense experimental measurements at earlier time-steps in order to provide better estimates of the initial inactivation rate λ0 for inactivation data which exhibits highly non-linear behavior.
Fig. 4

Scatter-plot of behavioral pairs of λ0 and α for experiment e+d-

Conclusions

This study examined how the inactivation of bacteriophage MS2 in water was affected by ionic strength (IS) and dissolved organic carbon (DOC) using static batch inactivation experiments at 4 °C conducted over a period of 2 months. Experiments were conducted using a two-treatment, two-level full-factorial design where the high and low levels of the treatments (DOC and IS) were characteristic of the variation observed at the Tunåsen managed aquifer recharge (MAR) scheme in Uppsala, Sweden. Experimental data was fit with constant and time-dependent inactivation models using traditional linear and nonlinear least-squares techniques and generalized likelihood uncertainty estimation (GLUE). Modeling results from both the linear least-squares and GLUE methodologies indicated a slight increase in the constant inactivation rate λ when DOC concentrations were increased from 17 to 31 mg/L; however the increase in the constant inactivation rate λ was less significant when considering the uncertainty of the parameter estimates. Results from the linear least-squares methodology indicated a slight decrease in λ when IS increased from 7.0 to 8.6 mM; however, GLUE showed that IS had no effect on λ. Results from the least-squares model fitting indicated that the experiment with a high level of IS and a low level of DOC (experiment e+d-) was the only experiment that was best represented by a model of time-dependent inactivation when models were fit to 33.5 days of inactivation data. The traditional linear and non-linear least-squares methodologies and the GLUE methodology performed similarly in regards to their estimations of the model parameters and the uncertainty surrounding the parameter estimates. However, GLUE was able to arrive at these conclusions without making large assumptions about the distribution of the model and data errors, which indicates that GLUE could be used as a viable alternative to the traditional least-squares methodologies for parameter estimation in virus inactivation models. Inactivation models fit to 33.5 days of data using the least-squares methodology were all capable of predicting MS2 concentrations after 2 months. The least-squares methodology suggested that experiment e+d- was best described using a model of time-dependent inactivation. Models of constant inactivation were best suited for the remaining three experiments; however, results from GLUE indicated that a model of constant inactivation was sufficient for all experiments once data uncertainty was considered directly in model fitting. Time-series longer than 2 months would be needed in order to provide any concrete conclusions regarding the time-dependence of MS2 inactivation at 4 °C under the conditions present at the Tunåsen infiltration scheme.

Notes

Acknowledgements

This study was carried out within the Centre for Natural Disaster Science (CNDS) at Uppsala University

Supplementary material

10040_2017_1559_MOESM1_ESM.pdf (304 kb)
ESM 1(PDF 303 kb)

References

  1. Adams MH (1959) Bacteriophages. Interscience, New YorkGoogle Scholar
  2. Anders R, Chrysikopoulos CV (2006) Evaluation of the factors controlling the time-dependent inactivation rate coefficients of bacteriophage MS2 and PRD1. Environ Sci Technol 40:3237–3242. doi:10.1021/es051604b CrossRefGoogle Scholar
  3. Bae J, Schwab KJ (2008) Evaluation of murine norovirus, feline calicivirus, poliovirus, and MS2 as surrogates for human norovirus in a model of viral persistence in surface water and groundwater. Appl Environ Microbiol 74:477–484. doi:10.1128/AEM.02095-06 CrossRefGoogle Scholar
  4. Bergström RB (1986) Uppsalaåsen och Vattholmaåsen: en hydrogeologisk undersökning av delen Galgbacken-Svista/Fullerö [Uppsala Ridge and Vattholma Ridge: a hydrogeological survey of the Galgbacken-Svista/Fullerö part]. Societas Upsaliensis pro Geologia Quaternaria, Jättendal, SwedenGoogle Scholar
  5. Beven KJ (1993) Prophecy, reality and uncertainty in distributed hydrological modelling. Adv Water Resour 16:41–51. doi:10.1016/0309-1708(93)90028-E CrossRefGoogle Scholar
  6. Beven KJ (2002a) Towards a coherent philosophy for modelling the environment. Proc R Soc A Math Phys Eng Sci 458:2465–2484. doi:10.1098/rspa.2002.0986 CrossRefGoogle Scholar
  7. Beven KJ (2002b) Towards an alternative blueprint for a physically-based digitally simulated hydrologic response modelling system. Hydrol Process 16:189–206. doi:10.1002/hyp.343 CrossRefGoogle Scholar
  8. Beven KJ, Binley A (1992) The future of distributed models: model calibration and uncertainty prediction. Hydrol Process 6:279–298. doi:10.1002/hyp.3360060305 CrossRefGoogle Scholar
  9. Binley A, Beven KJ (2003) Vadose zone flow model uncertainty as conditioned on geophysical data. Ground Water 41:119–127. doi:10.1111/j.1745-6584.2003.tb02576.x CrossRefGoogle Scholar
  10. Bixby RL, O’Brien DJ (1979) Influence of fulvic acid on bacteriophage adsorption and complexation in soil. Appl Environ Microbiol 38:840–845Google Scholar
  11. Bouwer H (2002) Artificial recharge of groundwater: hydrogeology and engineering. Hydrogeol J 10:121–142. doi:10.1007/s10040-001-0182-4 CrossRefGoogle Scholar
  12. Box GEP, Hunter WG, Hunter JS (1978) Statistics for experimenters: An introduction to design, data analysis, and model building, 1st ed. Wiley, New YorkGoogle Scholar
  13. Brunkard JM, Ailes E, Roberts VA, Hill V, Hilborn ED, Craun GF, Rajasingham A, Kahler A, Garrison L, Hicks L, Carpenter J, Wade TJ, Beach MJ, Yoder Msw JS (2011) Surveillance for waterborne disease outbreaks associated with drinking water: United States, 2007–2008. MMWR Surveill Summ 60:1–75Google Scholar
  14. Chattopadhyay D, Chattopadhyay S, Lyon WG, Wilson JT (2002) Effect of surfactants on the survival and sorption of viruses. Environ Sci Technol 36:4017–24. doi:10.1021/es0114097 CrossRefGoogle Scholar
  15. Christiaens K, Feyen J (2001) Analysis of uncertainties associated with different methods to determine soil hydraulic properties and their propagation in the distributed hydrological MIKE SHE model. J Hydrol 246:63–81. doi:10.1016/S0022-1694(01)00345-6 CrossRefGoogle Scholar
  16. Chrysikopoulos CV, Aravantinou AF (2012) Virus inactivation in the presence of quartz sand under static and dynamic batch conditions at different temperatures. J Hazard Mater 233–234:148–57. doi:10.1016/j.jhazmat.2012.07.002 CrossRefGoogle Scholar
  17. Corry JEL, Jarvis B, Passmore S, Hedges A (2007) A critical review of measurement uncertainty in the enumeration of food micro-organisms. Food Microbiol 24:230–253. doi:10.1016/j.fm.2006.05.003
  18. Corso PS, Kramer MH, Blair KA, Addiss DG, Davis JP, Haddix AC (2003) Cost of illness in the 1993 waterborne Cryptosporidium outbreak, Milwaukee, Wisconsin. Emerg Infect Dis 9:426–431. doi:10.3201/eid0904.020417 CrossRefGoogle Scholar
  19. Craun GF, Brunkard JM, Yoder JS, Roberts VA, Carpenter J, Wade T, Calderon RL, Roberts JM, Beach MJ, Roy SL (2010) Causes of outbreaks associated with drinking water in the United States from 1971 to 2006. Clin Microbiol Rev 23:507–528. doi:10.1128/CMR.00077-09 CrossRefGoogle Scholar
  20. DeBorde DC, Woessner WW, Lauerman B, Ball PN (1998) Virus occurrence and transport in a school septic system and unconfined aquifer. Groundwater 36:825–834. doi:10.1111/j.1745-6584.1998.tb02201.x CrossRefGoogle Scholar
  21. Foppen JWA, Schijven JF (2006) Evaluation of data from the literature on the transport and survival of Escherichia coli and thermotolerant coliforms in aquifers under saturated conditions. Water Res 40:401–26. doi:10.1016/j.watres.2005.11.018 CrossRefGoogle Scholar
  22. Foppen JWA, Okletey S, Schijven JF (2006) Effect of goethite coating and humic acid on the transport of bacteriophage PRD1 in columns of saturated sand. J Contam Hydrol 85:287–301. doi:10.1016/j.jconhyd.2006.02.004 CrossRefGoogle Scholar
  23. Freer J, Beven KJ, Ambroise B (1996) Bayesian estimation of uncertainty in runoff production and the value of data: an application of the GLUE approach. Water Resour Res 32:2161–2173. doi:10.1029/95WR03723 CrossRefGoogle Scholar
  24. Gerba CP (2006) Bacteriophage as pollution indicators. In: Calendar R (ed) Bacteriophages. Oxford University Press, Oxford, UK, pp 695–701Google Scholar
  25. Grant SB (1995) Inactivation kinetics of viral aggregates. J Environ Eng 121:311–319. doi:10.1061/(ASCE)0733-9372(1995)121:4(311) CrossRefGoogle Scholar
  26. Halonen JI, Kivimäki M, Oksanen T, Virtanen P, Virtanen MJ, Pentti J, Vahtera J (2012) Waterborne outbreak of gastroenteritis: effects on sick leaves and cost of lost workdays. PLoS One 7, e33307. doi:10.1371/journal.pone.0033307 CrossRefGoogle Scholar
  27. Harmel DR, Smith PK (2007) Consideration of measurement uncertainty in the evaluation of goodness-of-fit in hydrologic and water quality modeling. J Hydrol 337:326–336. doi:10.1016/j.jhydrol.2007.01.043 CrossRefGoogle Scholar
  28. Hoffmann S, Batz MB, Morris JG Jr (2012) Annual cost of illness and quality-adjusted life year losses in the United States due to 14 foodborne pathogens. J Food Prot 75:1292–1302. doi:10.4315/0362-028X.JFP-11-417 CrossRefGoogle Scholar
  29. Hurst CJ (1988) Effect of environmental variables on enteric virus survival in surface freshwaters. Water Sci Technol 20:473–476Google Scholar
  30. Hurst CJ, Gerba CP, Cech I (1980) Effects of environmental variables and soil characteristics on virus survival in soil. Appl Environ Microbiol 40:1067–1079Google Scholar
  31. Hurst CJ, Wild DK, Clark RM (1992) Comparing the accuracy of equation formats for modeling microbial population decay rates. In: Hurst CJ (ed) Modeling the metabolic and physiologic activities of microorganisms. Wiley, New York, pp 149–175Google Scholar
  32. IAWPRC (1991) Bacteriophages as model viruses in water-quality control. Water Res 25:529–545. doi:10.1016/0043-1354(91)90126-B CrossRefGoogle Scholar
  33. IHSS (2011) Elemental compositions and stable isotopic ratios of IHSS samples (WWW Document). URL http://www.humicsubstances.org/elements.html. Accessed 1 Feb 2016
  34. Jarvis B (1989) Statistical aspects of the microbiological analysis of foods, 1st ed. Prog Ind Microbiol. Elsevier Science, AmsterdamGoogle Scholar
  35. Keswick BH, Gerba CP (1980) Viruses in groundwater. Environ Sci Technol 14:1290–1297. doi:10.1021/es60171a602 CrossRefGoogle Scholar
  36. Keswick BH, Satterwhite TK, Johnson PC, DuPont HL, Secor SL, Bitsura JA, Gary GW, Hoff JC (1985) Inactivation of Norwalk virus in drinking water by chlorine. Appl Environ Microbiol 50:261–264Google Scholar
  37. Kvitsand HML, Ilyas A, Østerhus SW (2015) Rapid bacteriophage MS2 transport in an oxic sandy aquifer in cold climate: field experiments and modeling. Water Resour Res 51:9127–9140. doi:10.1002/2014WR016259 CrossRefGoogle Scholar
  38. Larsson C, Andersson Y, Allestam G, Lindqvist A, Nenonen N, Bergstedt O (2013) Epidemiology and estimated costs of a large waterborne outbreak of norovirus infection in Sweden. Epidemiol Infect 142:592–600. doi:10.1017/S0950268813001209 CrossRefGoogle Scholar
  39. Moore RS, Taylor DH, Reddy MM, Sturman LS (1982) Adsorption of reovirus by minerals and soils. Appl Environ Microbiol 44:852–859Google Scholar
  40. Morosini M (1989) The artificial recharge of Tunåsen, Uppsala: a hydrochemical consideration. Uppsala University, Uppsala, SwedenGoogle Scholar
  41. Niemelä SI (1996) A semi-empirical precision control criterion for duplicate microbial colony counts. Lett Appl Microbiol 22:315–319Google Scholar
  42. Niemelä SI (2003) Uncertainty of quantitative determinations derived by cultivation of microorganisms. Centre for Metrology and Accreditation. Helsinki, FinlandGoogle Scholar
  43. Overby LR, Barlow GH, Doi RH, Jacob M, Spiegelman S (1966) Comparison of two serologically distinct ribonucleic acid bacteriophages: II. properties of the nucleic acids and coat proteins. J Bacteriol 92:739–745Google Scholar
  44. Pang L (2008) Microbial removal rates in subsurface media estimated from published studies of field experiments and large intact soil cores. J Environ Qual 38:1531–1559. doi:10.2134/jeq2008.0379 CrossRefGoogle Scholar
  45. Pang L, Close M, Goltz M, Noonan M, Sinton L (2005) Filtration and transport of Bacillus subtilis spores and the F-RNA phage MS2 in a coarse alluvial gravel aquifer: implications in the estimation of setback distances. J Contam Hydrol 77:165–94. doi:10.1016/j.jconhyd.2004.12.006 CrossRefGoogle Scholar
  46. Pastor J, Solin J, Bridgham SD, Updegraff K, Harth C, Weishampel P, Dewey B (2003) Global warming and the export of dissolved organic carbon from boreal peatlands. Oikos 100:380–386. doi:10.1034/j.1600-0706.2003.11774.x CrossRefGoogle Scholar
  47. Riera-Montes M, Brus Sjölander K, Allestam G, Hallin E, Hedlund K-O, Löfdahl M (2011) Waterborne norovirus outbreak in a municipal drinking-water supply in Sweden. Epidemiol Infect 139:1928–1935. doi:10.1017/S0950268810003146 CrossRefGoogle Scholar
  48. Schijven JF, Hassanizadeh SM (2000) Removal of viruses by soil passage: overview of modeling, processes, and parameters. Crit Rev Environ Sci Technol 30:49–127. doi:10.1080/10643380091184174 CrossRefGoogle Scholar
  49. Schijven JF, Medema G, Vogelaar AJ, Hassanizadeh SM (2000) Removal of microorganisms by deep well injection. J Contam Hydrol 44:301–327. doi:10.1016/S0169-7722(00)00098-X CrossRefGoogle Scholar
  50. Schijven JF, Sadeghi G, Hassanizadeh SM (2016) Long-term inactivation of bacteriophage PRD1 as a function of temperature, pH, sodium and calcium concentration. Water Res 103:66–73. doi:10.1016/j.watres.2016.07.010 CrossRefGoogle Scholar
  51. Sharp DG (1965) Electron microscopy and viral particle function. In: Berg G (ed) Transmission of viruses by water route. Wiley, New York, pp 193–217Google Scholar
  52. Sim Y, Chrysikopoulos CV (1996) One-dimensional virus transport in porous media with time-dependent inactivation rate coefficients. Water Resour Res 32:2607–2611. doi:10.1029/96WR01496 CrossRefGoogle Scholar
  53. Sutton S (2011) Accuracy of plate counts. J Valid Technol 17:42–46Google Scholar
  54. Sveriges lantbruksuniversitet (2015) Miljödata MVM [WWW Document]. URL http://www.slu.se/sv/webbtjanster-miljoanalys/miljodata-mvm/introduktion/. Accessed 20 March 2015
  55. Taylor BN, Kuyatt CE (1994) Guidelines for evaluating and expressing the uncertainty of NIST measurement results. NIST Technical Note 1297, NIST, Gaithersburg, MDGoogle Scholar
  56. The MathWorks Inc. (2014) MATLAB R2014b. MathWorks, Natick, MAGoogle Scholar
  57. Tomasiewicz DM, Hotchkiss DK, Reinbold GW, Read RB, Hartman PA (1980) The most suitable number of colonies on plates for counting. J Food Prot 43:282–286. doi:10.4315/0362-028X-43.4.282
  58. Yates MV, Yates SR (1987) A comparison of geostatistical methods for estimating virus inactivation rates in ground water. Water Res 21:1119–1125. doi:10.1016/0043-1354(87)90033-9 CrossRefGoogle Scholar
  59. Yates MV, Gerba CP, Kelley LM (1985) Virus persistence in groundwater. Appl Environ Microbiol 49:778–81Google Scholar
  60. Zacheus O, Miettinen IT (2011) Increased information on waterborne outbreaks through efficient notification system enforces actions towards safe drinking water. J Water Health. doi:10.2166/wh.2011.021 Google Scholar
  61. Zhang D, Beven KJ, Mermoud A (2006) A comparison of non-linear least square and GLUE for model calibration and uncertainty estimation for pesticide transport in soils. Adv Water Resour 29:1924–1933. doi:10.1016/j.advwatres.2006.02.004 CrossRefGoogle Scholar

Copyright information

© The Author(s) 2017

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.

Authors and Affiliations

  • Jean-Marc Mayotte
    • 1
    • 2
  • Thomas Grabs
    • 1
  • Stacy Sutliff-Johansson
    • 1
  • Kevin Bishop
    • 1
    • 3
  1. 1.Uppsala UniversityUppsalaSweden
  2. 2.Centre for Natural Disaster Science (CNDS) at Uppsala UniversityUppsalaSweden
  3. 3.Swedish University of Agricultural SciencesUppsalaSweden

Personalised recommendations