SIS model
Mean field theory of SIS model
We first consider a general SIS model that allows for immigration. Denoting the proportions of susceptible and infectious populations by \(x(t)\) and \(y(t)\), respectively, the SIS model in a population of size N is given by
$$ \begin{array}{rll} \dot{x} &=& -\beta x y - \eta x +\gamma y \\ \dot{y} &=& \beta x y + \eta x -\gamma y, \end{array} $$
(1)
where \(\beta \) is the the transmission rate, \(\gamma \) is the rate of transfer from the infectious class to the susceptible class, and a dot denotes the time derivative. The model assumes that infection may also occur through contact with a trickle of infectious imports at a rate \(\eta \), either by susceptibles briefly leaving the population and making contact with infectious individuals located elsewhere or through infectious visitors briefly entering the population and making contact with susceptibles, so that the total force of infection is \(\Lambda = \beta y + \eta \). Since the population size N is constant, i.e., \(x+y=1\), the SIS model may be described by a single equation,
$$ \dot{y} = \beta (1-y) y + \eta (1 -y) -\gamma y. $$
(2)
We note that Eq. (2) can encompass a variety of infectious disease systems, including closed population SI models. Thus, the system can be used to model diseases that confer no long-lasting immunity, e.g., sexually transmitted diseases or acute infections such as influenza or diseases that are fatal. If \(\eta = 0\), Eq. (2) has two equilibria: the disease-free equilibrium \(y_{0}=0\) and the endemic equilibrium \(y^{\ast } = (1-1/R_{0})\), where \(R_{0} = \beta / \gamma \) denotes the basic reproduction number. When \(R_{0}>1\), the infection can persist, but if \(R_{0}<1\), then the infection dies out. The equilibria \(y_{0}\) and \(y^{\ast }\) meet and exchange stability when the basic reproduction number \(R_{0}=1\), i.e., a transcritical bifurcation occurs. We will refer to the SIS model with no immigration as the limiting case. However, it is usually more realistic to assume that a low level of immigration occurs, i.e., \(\eta \) is positive. If \(\eta >0\), then Eq. (2) has a single positive equilibrium that is always locally stable, provided \(\beta , \gamma >0\), since the slope at the equilibrium \(\lambda = \beta - 2 \beta y^{\ast } - \gamma -\eta \) is negative. Moreover, whether \(\eta = 0\) or \(\eta >0\), the return time \(-1/\lambda \) decreases in the limit \(\beta \rightarrow \gamma \). At the critical point in the limiting case SIS model, \(\lambda =0\). Therefore, the transcritical bifurcation occurs in a suitably constructed limiting case (\(\eta =0\)), which we suggest gives an indication for the behavior of the fluctuations for the stochastic version of the model with immigration, provided \(\eta \) is small.
Slow changes in the transmission rate \(\beta \), through demographic or evolutionary means, may induce a transcritical bifurcation of Eq. (2). For example, transmission may decrease as a result of slow increases in vaccination uptake among the population, leading to extinction of the pathogen, or it may increase due to slow decreases in vaccination uptake, potentially causing a transition to endemicity. To formally incorporate slow changes in transmission into the model, we rewrite Eq. (2) as a fast–slow system:
$$ \begin{array}{rll} \dot{I} &=& \beta (1-y) y + \eta (1-y) -\gamma y,\\ \dot{\beta} &=& \epsilon f(y, \beta) , \end{array} $$
(3)
where \(0< \epsilon \ll 1\) and the function f describes the change in transmission rate. In this paper, we assume that transmission is a slowly changing linear function of time, i.e., \(\beta = \beta (t) =\beta (1-p(t))\). The proportion of the population that are vaccinated is modeled by the function \(p(t)=p_{s}+p_{0} t\), with \(p_{0}\) an incremental change in \(\beta \). Hence, \(\dot {\beta } = p_{0}\). If \(p_{0}>0\), then the rate of transmission is slowly declining over time, and if \(p_{0}<0\), it is slowly increasing. The limit case \(\epsilon \rightarrow 0\) of Eq. (3) is Eq. (2), and thus, the endemic equilibrium of the limit case assumes \(\beta = \beta (1-p_{s})\), which arises from the level of vaccination uptake \(p(t)\) (the bifurcation parameter). Figure 1 shows the transcritical bifurcation diagram for the SIS model. For small \(\eta \), the plot of the stable endemic equilibrium \(y^{\ast }\) as a function of vaccination uptake is similar to the bifurcation diagram for \(\eta =0\), except in the vicinity of the critical point \(p^{\ast }\) (inset figure of Fig. 1). For \(p > p^{\ast }\), the disease-free equilibrium is stable in the limiting case, but for the model with immigration, the infectious population is sustained at a low level due to importation of the disease from external sources (Fig. 1).
Stochastic description of SIS model
To investigate the effect of noise on this transition, we assume that fluctuations in the infectious population are caused by demographic stochasticity (intrinsic noise). To understand these fluctuations, we require an individual-level description rather than a population-level description. We assume that all individuals have identical attributes, and individuals may move from the susceptible state \(S(t)\) to the infectious state \(I(t)\), or from the infectious state to the susceptible state. We assume that the population size is constant, i.e., \(S(t)+I(t)=N\). Since the population size is constant, we need only model the transitions into and out of the infectious state. Furthermore, since the SIS system moving towards a bifurcation is a fast–slow system, the transmission rate \(\beta \) is not constant in time but is a slowly changing function of time. Because the probability of transmission between an infectious and susceptible individual is slowly changing over time, we treat the transmission rate as constant for each small time increment dt, \(\overline {\beta }\). Table 1 shows the variables and parameters of the SIS model.
Table 1 Variables and parameters of the SIS model
We assume that in a sufficiently small time increment dt, the number of infectious individuals I can either (a) increase by 1, (b) decrease by 1, or (c) not change in number. Infection and removal are the events in an SIS process that lead to these changes in state. We denote the number of individuals I at time t or the state of the system by \(\alpha = I\) at time t and the alternative states by \(\tilde {\alpha }\), where either \(\tilde {\alpha } = I+1\) or \(\tilde {\alpha }=I-1\) in the SIS model. The system of individuals goes from a state with \(\alpha =I\) individuals at time t to a state with \(\tilde {\alpha }\) individuals at time \(t+dt\) with transition probability per unit time \(T_{i}(\tilde {\alpha }|\alpha )\). The system can also go from a state with \(\tilde {\alpha }\) individuals at time t to I individuals at time \(t+dt\) with a probability flux \(\tilde {T_{i}}(\alpha |\tilde {\alpha })\). Table 2 outlines the events, changes in state, and the transition probability fluxes into and out of state I that can occur in a stochastic SIS process with a slowly changing transmission rate.
Table 2 Transition probability fluxes for SIS model into and out of state \(\alpha =I\)
Since the transitions in Table 2 describe a Markov process, we can write down a master equation describing how the probability of there being I individuals at time t evolves with time. The master equation represents the continuous time version of the dynamics of the Markov process. Derivations of master equations can be found in Van Kampen (1981) and Renshaw (1991). Letting \(P(I,t)=\text {Prob}\)
\((I(t)=I)\) be the probability that the infectious state variable \(I(t)\) is equal to some nonnegative integer I, the master equation for the evolution of the probability of the infectious population being in state \(\alpha = I\) at time t is
$$ \frac{dP(\alpha,t)}{dt} = \sum\limits_{\alpha \neq \tilde{\alpha}} \tilde{T_{i}}(\alpha|\tilde{\alpha})P(\tilde{\alpha},t)- \sum\limits_{\alpha \neq \tilde{\alpha}} T_{i}(\tilde{\alpha}|\alpha)P(\alpha,t), $$
(4)
where \(\tilde {\alpha }\) describes all other states (Table 3), and the probability fluxes into and out of state I are as described in Table 2. This is a system of \(N+1\) ordinary differential equations, \(\alpha =0, 1, \dots N\), which can be solved with an initial condition of \(I_{0}\) individuals at time \(t=0\), i.e., \(P(\alpha , 0) = 1\) for \(\alpha = I_{0}\) and \(P(\alpha , 0) = 0\) for all \(\alpha \neq I_{0}\). Thus, the probability distribution at time \(t=0\) is a delta function. Table 3 summarizes the quantities in Eq. (4).
Table 3 Master equation terms. The transition probability of infection per unit time is given as an example of a transition into and out of state \(\alpha \) for the SIS and SIR stochastic processes. The other transition probability fluxes can be found in Tables 2 and 6
We will apply the van Kampen system size expansion to the master equation. The system size expansion takes advantage of the fact that, for large population size N, the demographic fluctuations in the population are small, and so N is the expansion parameter. Thus, the key assumption for the expansion to be valid is that the system size N is sufficiently large (see Appendix A for details). Here, we pause to note that that the SIS model is equivalent to a simple logistic process, and hence, there are analytical expressions for the mean and variance (Allen 2003). The \(I=0\) state is absorbing, and the expected time to extinction is finite (Allen 2003). Nevertheless, if the population size is large, the probability distribution is approximately stationary for a very long time. It is the statistics of this quasi-stationary distribution that we are interested in.
Following expansion of the master equation, the leading order and next-to-leading order terms are collected, giving rise to a deterministic system that describes the evolution of the trend and its stochastic correction, respectively, see Appendix A for details. The deterministic system turns out to be the fast–slow system (3). The distribution of the fluctuations about the solution of Eq. (3) is given by a linear Fokker–Planck equation. Thus the fluctuations are Gaussian distributed about the mean, and thus, \(I(t) \sim \text{Normal}(N \varphi(t), N \sigma^2)\), see Appendix A for details and definitions. The Fokker–Planck equation is equivalent to a stochastic differential equation (Gardiner 2004). This is the key advantage of applying the system size expansion. Using the stochastic differential equation, we can obtain analytical expressions for statistical signatures of leading indicators and early warning signals, including the power spectrum and autocorrelation function (see Appendix A for details). Table 4 summarizes the indicator statistics calculated using the methods described in Appendix A for the stable limiting case \(\epsilon \rightarrow 0\) for the SIS model, in terms of the eigenvalue \(\lambda \). We will examine how the statistics change over a range of vaccination uptake values in section “Results”.
Table 4 Analytical expressions for quasi-stationary statistics about the endemic infectious quasi-steady state expressed in terms of the eigenvalues
Robustness of the van Kampen approximation
Since the system size expansion involves approximating the discrete random variable I with a normal random variable, its assumptions may break down when I is small. For small numbers of infectious individuals, e.g., if the system is subcritical and, thus, incidence of a disease is low, the assumption that the fluctuations are normally distributed about the mean is likely to be inappropriate. The van Kampen expansion is valid when the system is far from the absorbing boundary at \(I=0\). However, the approximation upon which the approach is built (Eq. 10 in Appendix A) cannot describe chance extinctions, which can occur if I is small. If \(\eta = 0\) and \(\overline {p} > 1- 1/R_{0}\), the disease-free equilibrium is stable, and so the approximation given by Eq. (10) is not valid. Due to the absorbing boundary, the probability distribution about the disease-free state for a given time t will be one-sided. On the the other hand, if \(\eta >0\), then, close to the system boundary, the normal distribution approximation about the deterministic mean may be poor because the distribution is bimodal due to extinction events. However, the models with immigration do not have a disease-free state; just a small infectious population when \(\overline {p}\) is approximately greater than \(1/R_{0}\). Therefore, predictions for the quasi-stationary statistics about this state can be obtained.
SIR model
Mean field theory of SIR model
Denoting the proportions of susceptible, infectious, and recovered populations by \(x(t)\), \(y(t)\), and \(z(t)\) respectively, the SIR model with immigration in a closed population of size N is
$$\begin{array}{rll}\label{eqn:SIRmodelimmigration}\dot{x} &=& \mu(1-p) -\beta x y - \eta x -\mu x \\\dot{y} &= &\beta x y + \eta x -(\gamma+\mu) y, \\\dot{z} &=& \mu p+ \gamma y - \mu z,\end{array}$$
(5)
where \(\beta \) is the transmission rate, \(\gamma \) is the recovery rate, \(\eta \) is the immigration rate, and \(\mu \) is the per capita birth rate. To maintain a constant population size, the per capita death rate is set equal to the birth rate. A proportion p of the population are chosen at random for vaccination at birth and are recruited into the recovered class. The remaining unvaccinated proportion of the population enter the susceptible class at birth. Since the population size N is constant, \(x+y+z=1\), system (5) is equivalent to the following system of ordinary differential equations
$$\begin{array}{rll}\label{eqn:SIRmodel}\dot{x} &=& \mu(1-p) -\beta x y - \eta x -\mu x \\\dot{y} &=& \beta x y + \eta x -(\gamma+\mu) y.\end{array} $$
(6)
The SIR model is particularly appropriate for acute immunizing infections such as measles and pertussis.
In the absence of vaccination and immigration, the basic reproduction number is given by \(R_{0} =\beta /(\gamma +\mu )\). If \(\eta = 0\), the SIR model (6) has two equilibria: the disease free state \((x_{0}, y_{0})=(1-p, 0)\) and the endemic equilibrium \((x^{\ast }, y^{\ast })=(1/R_{0}, \mu (1-p-1/R_0) /(\gamma+\mu)\), respectively. The SIR model undergoes a transcritical bifurcation at \(R_{0} =1\). The endemic equilibrium is locally stable if \(R_{0}>1\) and is not biologically feasible if \(R_{0}<1\), when the disease-free equilibrium is locally stable. As the vaccination uptake p increases, the basic reproduction number reduces by a factor \((1-p)\), i.e., the effective reproduction number is \(R_{0}(1-p)\). The vaccination uptake p for the effective reproduction number is 1 at the critical vaccination threshold, \(p^{\ast } = 1 - 1/R_{0}\) (Anderson and May 1991), and it is at this critical threshold that the transcritical bifurcation occurs. Again, the SIR model with no immigration is the relevant limiting case.
Temporary importation of pathogen often occurs in infectious disease systems (Keeling and Rohani 2008). Assuming that a low level of immigration occurs, i.e., \(\eta >0\), only a single positive equilibrium is biologically feasible. This equilibrium is a stable spiral when the square of the trace of the Jacobian matrix evaluated about the equilibrium is less than four times its determinant and is a stable node if it is greater than or equal to this quantity. Complex eigenvalues of the Jacobian matrix of Eq. (6) characterize a stable spiral, and the eigenvalues are real and negative if the equilibrium is a stable node. The stable node equilibrium can be thought of as a “disease-free” equilibrium if \(\eta \) is small, in the sense that the infectious population is sustained at a low level due to importation of the disease from external sources. Therefore, while the transcritical bifurcation occurs in a suitably constructed limiting case (\(\eta =0\)), the limit case may give an indication for the behavior of the fluctuations for the stochastic version of the model with immigration, provided that \(\eta \) is small.
Gradual changes in the vaccination uptake rate p may induce a transition from endemicity or to extinction. Recruitment of susceptibles may slowly vary over long time scales as a result of demographic or evolutionary changes. Vaccination uptake rates are often not constant over time and may exhibit trends, e.g., percentage uptake of pertussis vaccine in the USA (Rohani and Drake 2011). Mathematically, we may express the SIR model approaching a transcritical bifurcation as a fast–slow system:
$$\begin{array}{rll}\label{eqn:fastslowSIRmodel}\dot{x} &=& \mu(1-p) -\beta x y - \eta x -\mu x \\\dot{y} &= &\beta x y + \eta x -(\gamma+\mu) y, \\\dot{p} &= &\epsilon f(x, y, p), \end{array}$$
(7)
where \(0<\epsilon \ll 1\) and the function f describes the change in vaccination uptake p. Again, we model vaccination uptake as a linear function of time \(p(t)=p_{s}+p_{0} t\), with \(p_{0}\) an incremental change in p. Consequently, \(\dot {p} = p_{0}\). If \(p_{0} >0\), recruitment into the susceptible class is slowly declining over time and if \(p_{0}<0\), then recruitment is slowly increasing over time. In the limit \(\epsilon \rightarrow 0\), the vaccination uptake is fixed at a constant rate \(p_{s}\), and the system is stable. Figure 2 shows the transcritical bifurcation diagram for the SIR model. The infectious equilibrium \(y^{\ast }\) is plotted as a function of vaccination uptake p. The bifurcation diagram indicates that the endemic infectious equilibrium declines linearly with p in the models with and without immigration. However, the inset plot indicates that in the vicinity of the \(\eta = 0\) critical point, the infectious equilibrium of the immigration model is elevated relative to the disease-free equilibrium, since the infectious population is sustained at a low level due to immigration.
Stochastic description of SIR model
Model (7) is the deterministic description of the SIR system approaching a transition, but there will be stochastic fluctuations in the state of the system as the transition is approached. As before, we assume that these fluctuations result from demographic stochasticity. To quantify these fluctuations, we assume individuals are identical. Individuals may be recruited into the susceptible state and can transition out of this state through infection or death. Infectious individuals may recover or die. Assuming that the population size \(S(t) + I(t) +R(t) = N\) is constant, we need only consider the transitions into and out of the susceptible and infectious states. Furthermore, since the SIR system moving towards a bifurcation is a fast–slow system, the vaccination uptake p is not constant in time but is a slowly changing function of time. Therefore, the recruitment rate into the susceptible class is slowly changing, but, for each small time increment dt, we can treat the vaccination uptake as constant \(\overline {p}\). Table 5 presents the variables and parameters of the SIR model.
Table 5 Variables and parameters of the SIR model
Events that occur in an SIR process in a small time increment dt moving slowly towards a transition include infection, recovery, recruitment to the susceptible class, and death due to natural causes. Table 6 outlines the events, changes in state, and the transition probabilities per unit time for each change in state. We can construct a master equation in the same manner as for the SIS model. Letting \(P(S, I,t)=\text {Prob}((S(t),I(t)) = (S, I))\) be the probability that the state vector \((S(t),I(t))\) is equal to some nonnegative integer \((S,I)\), the master equation for the evolution of the probability of the population being in state \(\alpha = (S,I)\) at time t is
$$ \frac{dP(\alpha,t)}{dt} = \sum\limits_{\alpha \neq \tilde{\alpha}} \tilde{T_{i}}(\alpha|\tilde{\alpha})P(\tilde{\alpha},t)- \sum\limits_{\alpha \neq \tilde{\alpha}} T_{i}(\tilde{\alpha}|\alpha)P(\alpha,t), $$
(8)
where \(\tilde {\alpha }\) describes all other states (Table 3).
Table 6 Transition probability fluxes for SIR model. Transitions can occur into and out of state \(\alpha = (S,I)\)
The master Eq. (8) is nonlinear and, therefore, cannot be solved analytically. To make analytical progress with Eq. (8), again, we can use the van Kampen system size expansion (see Appendix B for details). The approach gives rise to the deterministic SIR fast–slow system (7) and a linear Fokker–Planck equation that describes the evolution of the fluctuations, which may be written as a system of stochastic differential equations. These equations can be analyzed using Fourier transformation. The solution of the Fokker–Planck equation is a bivariate normal distribution. Table 4 summarizes the indicator statistics in the stable system limiting case \(\epsilon \rightarrow 0\) for the SIR model, in terms of its eigenvalues.
Simulations
The preceding sections present an analytical theory of early warning signals for emergence and leading indicators of elimination. To investigate the results of this theory for a particular parameter set (Table 7), we calculated leading indicators of elimination and emergence, assuming alternatively that (a) the mean proportion of infectious individuals is given by the deterministic endemic equilibrium (\(\epsilon \rightarrow 0\) theory) or (b) assuming it is given by the current state of the fast–slow system approaching a transition. We selected parameters consistent with sexually transmitted SIS diseases with long infectious period and large \(R_{0}\) and parameters typical for childhood infectious diseases with SIR dynamics. To examine how different changes in vaccination uptake \(p_{0}\) affect the statistics, we varied \(p_{0}\) by 1/500 and \(1/30~\text {year}^{-1}\) in the model with immigration. We also compared the elimination indicators with those calculated assuming that the mean proportion of infectious individuals was given by the deterministic endemic equilibrium from the limiting case models with no immigration. The early warning signals of emergence were not calculated for the limiting case because the disease-free equilibrium is stable.
Table 7 Parameter values for the simulations
To test the robustness of this theory to the range of approximations that were introduced (fast–slow approximation, continuum description, van Kampen expansion), we simulated the approach to elimination and emergence in a variety of cases. To simulate the approach to elimination, we followed the “bottom-up” approach of Allen (2003) to derive stochastic differential equations that incorporated demographic stochasticity. This approach uses the rates in Tables 2 and 6 to build a system of stochastic differential equations. Stochastic differential equations formulated in this way are appropriate, provided that the population size is sufficiently large, because then changes in the state variables are assumed to be normally distributed. Simulations were compared with output from Gillespie’s direct method (Gillespie 1977). The simulations were qualitatively similar for population sizes greater than 50,000. To simulate the approach to emergence, we used Gillespie’s direct method as this is most appropriate for small population sizes.
We simulated the SIS and SIR stochastic models with immigration approaching elimination and emergence 500 times. To compare the statistics to the theoretical predictions in Table 4, the infectious time series approaching elimination were sampled at yearly intervals. The transcritical bifurcation in these scenarios was approached over a long time frame (e.g., 470 years when \(p_{0} = 1/500\) in the SIR model). However, the transcritical bifurcation to emergence was approached over a relatively short time frame for the SIR model (10 years). To obtain a better sampled time series, the data from the Gillespie simulations were aggregated over monthly intervals for the SIR model. The infectious time series were aggregated over yearly intervals for the SIS model approaching emergence because events did not always occur at a monthly frequency. Thus, time series obtained from the SIS system approaching emergence allowed us to examine the issues that arise from poorly sampled time series.
Analysis over a moving window
To investigate the robustness of the early warning predictions over a moving window, i.e., as they would be used in online analysis of surveillance data, the influence of the slowly varying trend must be removed. The van Kampen approach (Appendices A and B) leads to a natural expression for the fluctuations from the quasi-stationary state \(N \varphi (t)\),
$$ \zeta(t) = N^{-1/2}(I -N \varphi(t) ), $$
(9)
where ϕ(t) is determined by the mean field equations (11)(SIS) and (22) (SIR) in Appendices A and B respectively. To obtain the fluctuations, we subtracted the current mean, which we assumed to be determined by the current state of the fast–slow system \(N \varphi (t)\) , from the state of the system at the start of each year and divided this quantity by the square root of the population size. We refer to this as Van Kampen detrending.
Gaussian filtering is another, more common, method used to remove the influence of a slowly varying mean of a data series. To compare the performance of Gaussian smoothing to van Kampen detrending, for each time series, we fit a Gaussian kernel smoothing function across the entire infectious case record up to the time that the transcritical bifurcation was predicted using a fixed bandwidth. Lenton et al. (2012) have shown that the results obtained from applying the Gaussian filter across the entire time series do not differ significantly from detrending within windows. To obtain the residuals, we subtracted the fit from each time series and divided by the square root of the population size to be consistent with van Kampen detrending. The choice of bandwidth was informed by the resemblance of the Gaussian residuals to the fluctuations obtained from the van Kampen approach. To study the changes in the statistics up to the critical transition, we calculated the lag-1 autocorrelation and the variance of the fluctuations obtained using the two detrending methods over a moving window half the length of the time series. We calculated the lag-1 autocorrelation coefficient of each replicate using the acf function in R. The coefficient of variation (CV) was calculated by calculating, over a moving window, the mean and standard deviation of each infectious replicate. The median and 95 % prediction intervals for each of the statistics were calculated over the 500 replicates of each model. The prediction intervals were calculated using the quantile function in R. To quantify trends in each statistic for each replicate, we used Kendall’s correlation coefficient \(\tau \). To determine the distribution of Kendall’s \(\tau \), we calculated the coefficient for the trend in the test statistic for each realization.
To assess the performance of the leading indicators, we followed the approach of Boettiger and Hastings (2012) and calculated receiver operating characteristic (ROC) curves from the distribution of Kendall’s \(\tau \) calculated from realizations of the models with and without transitions. The model without a transition is quasi-stationary, and we refer to it as the baseline or null model. The null models for elimination assume vaccination uptake \(p=0\) and are simulated beginning from the deterministic equilibrium at \(p=0\). The null models for emergence assume vaccination uptake \(p=0.96\) and are simulated beginning from the deterministic equilibrium at \(p=0.96\). The baseline models were simulated for the same length of time as it takes for the transition to be approached in the test models. The model with a transition is the test model. An ROC curve enables investigation of the sensitivity of leading indicators to detect differences between quasi-stationary systems and those approaching a critical transition. We simulated the baseline models 500 times and obtained fluctuations using the van Kampen approach and from Gaussian filtering. We then quantified the trend in each indicator using Kendall’s \(\tau \) for each baseline simulation. In our results, indicator statistics typically exhibited increasing trends or no trend, but for those that decreased, we multiplied Kendall’s \(\tau \) for each realization by -1 to calculate the ROC curve. The area under the ROC curve (AUC) was also calculated. An AUC close to one indicates near-perfect detection.