Abstract
After the pioneering theoretical studies of Lotka and Volterra, Gause and his coworkers replace the previously used linear functional response by using a saturating functional response with a discontinuity at a threshold prey density. Here we assume that prey density at below this threshold value is effectively and successfully in a refuge patch. In this situation there is no food option for predator and go to extinction. But above this threshold value surplus density of prey is available to predator for its diet. But the system does not show any future activities when prey density is in the vicinity of the threshold density, system is ill posed because the trajectories are not well defined here. In the present study, we redefine and analyze the model by using Filippov regularization method. By this continuation method, the system becomes well posed and gives more results as predicted by Gause. Also predator fully depends upon alternative diet to survive from extinction risk when prey is in refuge patch and system largely varies with the availability of alternative diet resource but in the later case predator again switches to its primary (essential) food. When prey density is in the vicinity of the threshold density, then predator may choose its deit preferentially from essential or alternative resources according to its profit. Numerical examples support these hypothesis and analytical results.
Similar content being viewed by others
Avoid common mistakes on your manuscript.
Introduction
Theoretical ecology is the scientific study and esthetic analysis of interactions among organisms and their environment. Ecosystem is composed of dynamically interacting parts including organisms, the communities they make up and the nonliving components of their environment. The dynamic relationship between predator and its prey has long been and will continue to be one of the dominant themes in both ecology and mathematical ecology due to its universal existence and importance (Berryman 1992). After the pioneering theoretical studies of Lotka (1925) and Volterra (1931), their predictions are experimentally tested by Gause (1934) and his coworkers (Gause et al. 1936). Gause focuses on three experiments consisting of prey (Aleuroglyphus agilis) and predatory (Cheyletus eruditus) mites, prey (Paramecium caudatum) and predator (Didinium nasatum) protists and yeast (Saccharomyces exiguus) and protists (Paramecium bursaria). The results of these experiments are not consistent with the Lotka–Volterra neutrally stable limit cycles, such as a prolong coexistence of prey and predator are obtained by regularly adding protists and mites where as, in a completely homogeneous environment Didinium destructs all prey and it collapses subsequently.
When the environment is not homogeneous and there is a structural diversity of habitat, then the propagation of life of both prey and predator and also their interrelationship (Alstad 2001; Anderson 1984) largely depends and varies upon the structure of habitat and more prominently their habitat selection (Anderson 2001; Johnson 2006). Refuge means a place or state of safety, provides in the literature is rather elastic. An implicit definition of “refuge” is often tied into the concept of barrier for population dispersal. Metapopulation, population makes up of sub populations of weak dispersal moving through weakly connected (due to, for example, the presence of population barriers) networks of patches, face limited intraspecific competition and as a consequence tend to be stable. Predator’s functional response, defines as the amount of prey catch per predator per unit of time, is affected by the structure of prey habitat and predator’s hunting ability (Alstad 2001; Anderson 1984). The use of spatial refuge by prey is one of the more relevant behavioral traits that decreases predation rate, e.g., spatial or temporal refuge, behavioral or physical refuge, prey aggregation or reducing search activity by prey. The smell, color, injection of some poisonous agent, size, skin, body cover etc may be used as different form of refuges, which can be used to save the prey species from its predator. Environmental heterogeneity enhances spatial refuges of prey which provides less accessible sites for predator to capture prey. By using refuge, some numbers of the prey population are protected against predator. The work assumes physical (nonadaptive) refuge used by prey  either a constant number or proportion of prey stays in the refuge patch (GonzalezOlivares and RamosJiliberto 2003; Kar 2005; Chen et al. 2010; Jana 2013, 2014). However, due to the decreasing resource, prey fitness (food intake or mating opportunity) gradually decreases. In the refuge patch there is no predation risk for the prey population and if their resources are not limiting the survival rate is increased. So, it has been documented that due to increasing foraging efficiency in an open patch, prey can reduce their activities and change their suitable habitat adaptively (Sih 1980, 1986, 1987; Ives and Dobson 1987; Lima and Dill 1990; Ruxton 1995; Křivan 1997, 1998, 2011, 2013; Peacor and Werner 2001; Brown and Kotler 2004). There has been a great deal of research on the effect of prey refuge (physical/ nonadaptive and behavioral/ adaptive) on the population dynamics (Kar 2005; Křivan 1997, 1998, 2011, 2013; Jana 2013, 2014).
Extinction is part of natural world. If we are to understand the causes of extinction, we shall have to comprehend many aspects of ecology and evolution. Population viability analysis (PVA) is a process of determining whether a particular recovery or restoration strategy will lead to success. PVA involves the consideration of information from all aspects of life history of the population (Ricklefs and Miller 2000). So, to rescue the population at any tropic level from extinction risk, for example, due to the lack of essential food in a diverse habitat, is a very important PVA. Alternative food source is a very important and prominent strategy when any population is suffering from lack of essential food. When a population moves through a habitat in search of food, it sequentially encounters potential or more available food with each encounter, the population is confronted with the choice of pursuing and eating the profitable food (Ricklefs and Miller 2000). Prey switching is a frequencydependent predation, where the predator preferentially consumes the most common type of prey. The phenomenon has also been described as apostatic selection. The term ‘switching’ is first coined by the ecologist Murdoch (1969) to describe the situation where a predator eats disproportionately more of the most common type of prey. Clarke (1962) describes a similar phenomenon. The diet choice models (Charnov 1976; Stephens and Krebs 1986) assume that predator feeds potential prey type on the basis of their profitability and also predict that at low preferred prey density, the interaction possibility between prey and predator sharply decreases due to their switching tendency to alternative food temporarily or permanently. Since then the term prey switching has mainly been used by ecologists. Different aspects of frequency dependent selection is also very important for prey population. Such a situation is more complex as prey fitness not only depends on predator abundance but also their own abundance. This phenomena makes the frequency dependent prey fitness (Colombo and Křivan 1993; Cressman and Křivan 2006, 2013; Křivan 1997, 1998, 2011, 2013) and finally the predator’s optimal foraging strategy can create a behavioral prey refuge (Oaten and Murdoch 1975; Charnov 1976; Holt 1983; Fryxell and Lundberg 1994, 1997; Abrams and Matsuda 1996; Křivan 1997, 2011, 2013; Abrams 1999; Křivan and Eisner 2003; Ma et al. 2003).
Ray and Straškraba (2001) notice that the prey species, i.e., detritivorous fish and their predator species, carnivorous fish coexist in Sundarban Mangrove ecosystem. In the pristine part of the ecosystem where forest is dense, the production of detritus is high and the detritivorous fish can easily take refuge in the densely inundated bushy part of the forest to avoid predation by carnivorous fish. In this part the production and densities of both prey and predator fish are high and coexist with each other. But in the reclaimed part of the forest where there is huge anthropogenic stress, production of detritus is less and due to lack of bushy part of the forest the area for refuge of prey species is minimum. In this reclaimed area prey density (detritivorous fish) is reduced at an alarming level but predator fish population is also slightly reduced because this fish population has switched its food habit to other fish and animals (Roy et al. 2008). In a completely homogeneous environment Didinium nasatum destructs all prey and it collapses subsequently. When the environment is not homogeneous and there is a refuge of total prey population, the population survives in the refuge but the predator population collapses. The situation is different when protists fed on the yeast, here a fraction of yeast population undergoes refuse. This strong experimental evidence suggests that in this type of situation population dynamics of both prey and predator tend to a limit cycle. The above observations lead Gause et al. (1936) to search for discrepancies in assumptions of the Lotka–Volterra preypredator model. First, they observe that protists are not able to feed on yeast at low densities because at low yeast densities the population hide into the sediment at the bottom which is not accessible to predator inhabiting the water column. Thus, prey is effectively and successfully in a refuge when at low concentrations (Kar 2005; Křivan 1997, 1998, 2011, 2013; Jana 2013, 2014; Jana et al. 2015). When prey reaches above the critical density, it reappears in the water column and becomes accessible to predator (Kar 2005; Křivan 1997, 1998, 2011, 2013; Jana 2013, 2014; Jana et al. 2015). So, when prey density is below its critical density, then there is a possibility of predator’s collapse. In that condition predator can switch their diet resources to another prey in the context of behavioral prey refuge (Oaten and Murdoch 1975; Charnov 1976; Holt 1983; Fryxell and Lundberg 1994, 1997; Abrams and Matsuda 1996; Křivan 1997, 2011, 2013; Abrams 1999; Křivan and Eisner 2003; Ma et al. 2003). This phenomena can avoid predator extinction possibility. But when prey density reaches above its critical density, predator switches to its essential (primary) food. At the vicinity of critical prey density predator starts switching from primary to alternative food sources or vice versa.
In the next section, we construct the model on the basis of Gause’s observation where the prey population is protected by a refuge of constant strength, when prey density below this threshold strength they are effectively and successfully in the refuge, above which only the surplus density is available for predation. In the vicinity of the critical value of prey density predator switches its primary to alternative food and vice versa. Mathematically till now the system is not defined in this vicinity because the trajectories are not well studied. In “Filippov regularization”, we analyze the system at this vicinity by applying Filippov regularization method for better understanding. Numerical examples support all the assumptions and their corresponding results throughout the study.
Model and analysis
Gause et al. (1936) consider the following form of Lotka–Volterra preypredator system as
Here, x and y are the densities of logistically growing prey [with intrinsic growth rate r and carrying capacity k in the form \(f=r(1\frac{x}{k})\)] and its predator population (with food conversion efficiency e and mortality rate m) respectively. Predator’s functional response is defined as the rate of prey captured by a predator per unit of time and here g is the Gause functional response which is specified bellow.
A constant number of the prey population is always protected from the hunting ability of predator, this is the refuge area and this subhabitat conceives mostly a low density of prey (say, \(x_c\)) (Gause et al. 1936; Maynard Smith 1974). These models assume that prey prefer to be in the refuge and only when the refuge is fully occupied, the surplus of prey moves outside. So, there are three prominent distinct dynamical features of the predation process.

1.
Up to the critical prey population threshold (\(x<x_c\)), prey is not consumed by predator. This situation corresponds to the refuge of prey in a constant size \(x_c\).

2.
When the prey density is above its critical value (\(x>x_c\)), then surplus of the prey population (\(xx_c\)) is accessible for predation.

3.
Consumption saturates with increasing prey density (Gause et al. 1936).
The jump at the critical prey density is motivated by their observation (Gause et al. 1936). This suggests that the functional response in the vicinity of \(x=x_c\) is quite steep and can be approximated by a functional response with a jump discontinuity at \(x=x_c\). Such a functional response is given by (see Fig. 1a)
here \(\alpha\) and h describe the attack rate and the handling time that a predator needs to process one unit of prey. \(x_c\) is the intensity of prey refuge, i.e., a critical prey density below which prey are not accessible to predator. Thus, for above the critical prey density, g is the Holling type II functional response (Holling 1959). For prey population density below the threshold (\(x<x_c\)), prey is not being eaten by predator, grows logistically while predator dies exponentially, i.e., model (1) becomes
The nullclines define domains in the phase plane (xyplane) in which the population rate of change does not change sign. The nullclines provide some insights regarding existence of equilibrium points and their stability behaviors. The equilibrium point, formed by the intersection of the prey and predator nullclines, is stable if \(\frac{dy}{dx}<0\), unstable if \(\frac{dy}{dx}>0\) and Hopf bifurcation occurs at local maxia and minima (Robert 1976; Rosenzweig 1969; Rosenzweig and MacArthur 1963). For small prey density which satisfying \(x<x_c\) (grey regions in the xyplane of Fig. 2a–c), there is only trivial prey nullcline, \(x=0\) and trivial predator nullcline, \(y=0\) and their intersection defines the trivial equilibrium point of the system (\(E_0(0,0)\)) and dynamically it is unstable node (UN). Thus, at each point of the preypredator density phase plane to the left of the critical prey density \(x=x_c\), trajectories move in the southeast direction (grey regions in the xyplane of Fig. 3i–iv).
For prey population density above the threshold (\(x>x_c\)), population dynamics (1) is given by the Lotka–Volterra model with the Holling typeII functional response [functional response (2)]
There is only one non trivial prey nullcline \(y=\frac{rx(kx)\{1+\alpha h (xx_c)\}}{\alpha k (xx_c)}\), this is a sigmoid curve with vertex at \(x=\underline{x}\) which is the positive root/roots of
and intersects positive xaxis at \(x=k\). The predator nullclines, where \(\frac{dy}{dt}=0\), are \(y=0\) and the vertical line \(x = \frac{m}{\alpha (emh)}+x_c\). If we vary intensity of prey refuge parameter \(x_c\) with \(h=0.17\) (other parameters are as in Fig. 1), we observe that when \(x_c<184.27\), then \(\frac{dy}{dx}>0\) and \(E^{*}\) is unstable; \(184.27<x_c<k\), then \(\frac{dy}{dx}<0\) and \(E^{*}\) is stable and \(x_c=184.27\) there is a Hopfbifurcation at \(E^{*}\) (Fig. 2a). Similarly if we vary h with \(x_c=200\) (other parameters are as in Fig. 1), then system is stable in \((0,h_1)\bigcup (h_2,h_3)\), unstable in \((h_1,h_2)\), Hopfbifurcation occurs at \(h_1\) and \(h_2\) and transcritical bifurcation occurs between \(E^{*}\) and \(E_k\) at \(h_3\), where \(h_1=0.1708\) (Fig. 2b), \(h_2=0.15\) (Fig. 2c), \(h_3=0.199\) . Thus, the dynamical features of the model system (4) are given by Table 1.
Existence of limit cycle and its uniqueness
It is known for preypredator system that the existence and stability of a limit cycle is related to the existence and stability of the interior equilibrium. If the interior equilibrium point \(E^{*}\) exists and any limit cycle does not exist, then the equilibrium point \(E^{*}\) is globally asymptotically stable. On the other hand, if an interior point exists and is unstable, there must occur at least one limit cycle. Using the Theorem 4.3 of Kuang and Freedman (1988), one can see that the system (4) has exactly one limit cycle which is globally asymptotically stable (green closed loop in Fig. 3iii, all trajectories with different initial condition (x(0), y(0)) converges to the stable limit cycle) with respect to the set \(\{(x, y)/ x >0, y>0\} {\setminus } {E^{*}}\) if \(0< x_c < \bar{x}_c\) (region \(R_3\) in Fig. 3a), where \(\bar{x}_c\) is the positive root (minimum root, if both roots are positive) of \(x_c^2+px_c+q=0\) \(\left[p=\frac{2m}{\alpha (emh)}k\frac{2e m}{(emh)^2}, q=\frac{e m k}{(emh)^2}\frac{2e m^2}{\alpha (emh)^3}\frac{m}{\alpha (emh)}(k\frac{m}{\alpha (emh)})\right]\). It is interesting to observe whether prey refuge intensity is capable of preventing the cyclic behavior of system population. Suppose the system parameters are such that the system (4) admits a limit cycle. This limit cycle can be prevented if all solutions of the system approach an equilibrium point instead of a limit cycle. Let \(\hat{x}_c\) be the value of \(x_c\) for which the system (4) admits a limit cycle. Then definitely \(\hat{x}_c\in (0, \bar{x}_c)\). Now, let \((\bar{x}, \bar{y})\) be the required limiting value for the solutions of the system. Let \(\hat{x}_c\) be such that \((x^{*}(\hat{x}_c), y^{*}(\hat{x}_c))= (\bar{x}, \bar{y})\). Clearly, \((\bar{x}, \bar{y })\) will be asymptotically stable if we select \(\hat{x}_c\) in such a way that \(\bar{x}_c<\hat{x}_c<k\frac{m}{\alpha (emh)}\) along with \(h<\frac{e}{m}\frac{1}{\alpha k}, e\alpha k>m\) (region \(\bigcup _{i=1}^2R_i\) in Fig. 3a and corresponding phase plane diagrams are in Fig. 3i, ii, here all trajectories with different initial condition (x(0), y(0)) with \(x(0)>x_c\) converge to the coexistence equilibrium point \(E^{*}\)). Therefore, it is possible to prevent oscillation of a preypredator system to a stable state varying the intensity of prey refuge. This shows that intensity of prey refuge may increase stability of a preypredator interaction.
Bifurcation analysis
To obtain a complete classification of the qualitative behavior of model (4), we analyze the bifurcation pattern (Jana 2013; Jana and Bairagi 2014; Jana et al. 2015; Solisa and KuCarrillo 2014) and illustrate the results with two parameters, say intensity of prey refuge\(x_c\) and handling timeh. The classification requires up to two codimensionone bifurcations: Hopf bifurcation points in which coexistence equilibrium point \(E^{*}\) exchanges stability, limit points of cycles in which a stable and an unstable cycles collide and bifurcation points tracking a transcritical bifurcation between coexistence equilibrium point \(E^{*}\) and prey only equilibrium \(E_k\), where these two equilibria coincide and exchange their stability to each other.
Hopf bifurcation
Onedimensional bifurcation analysis reveals the behavior of the system when a particular system parameter is varied for a long range. Here we observe the behavior of the system when \(x_c\), the prey refuge intensity is varied. It can be observed from the Eq. (4) that the characteristic roots become purely imaginary when \(x_c=\bar{x}_c\). In this case, Re\((\lambda )_{x_c=\bar{x}_c}=0\), Im\((\lambda )_{x_c=\bar{x}_c}\ne 0\) and \(\frac{d}{dc}\)Re\((\lambda )_{x_c=\bar{x}_c}<0\) (we use standard package of Mathematica to get these results) and hence the transversality condition for a Hopfbifurcation is satisfied (Perko 2001). Therefore, there exists a Hopfbifurcation at \(x_c=\bar{x}_c\). The negative sign of \(\frac{d}{dc}\)Re\((\lambda )_{x_c=\bar{x}_c}<0\) implies that the oscillations in the population densities dampen as \(x_c\) passes from lower value to higher value through \(\bar{x}_c\) (Fig. 4).
Similarly, if we analyze Hopf bifurcation with respect to h, we see that Hopfbifurcation occurs at \(h=h_1\) and \(h=h_2\) and the transversality conditions for a Hopfbifurcation are \(\frac{d}{dh}\)Re\((\lambda )_{h=h_1}>0\) and \(\frac{d}{dh}\)Re\((\lambda )_{h=h_2}<0\) (Fig. 5). Also, the red line in Fig. 3a indicates the Hopfbifurcation points in the \(x_ch\)plane.
Transcritical bifurcation
The system (4) undergoes a transcritical bifurcation involving the two equilibria \(E_k\) and \(E^{*}\). It has been shown that when \((0,h_1)\bigcup (h_2,h_3)\) (stability subregion \(\bigcup _{i=1}^{3}R_i\) in Fig. 3a), the coexistence equilibrium \(E^{*}\) is stable but the prey only equilibrium \(E_k\) is unstable node. The two equilibria coincide at \(h=h_3=\frac{e}{m}\frac{1}{\alpha (kx_c)}\) (green line in Fig. 3a) and exchange their stability. Once \(h>h_3\), \(E^{*}\) becomes unstable node (in this case this equilibrium point is biologically infeasible) and \(E_k\) becomes stable node (stability subregion \(R_4\) in Fig. 3a). Here, we will prove that the system undergoes a transcritical bifurcation at \(h=h_3=\frac{e}{m}\frac{1}{\alpha (kx_c)}\) using \(Sotomayer's\) theorem (Perko 2001). For \(h=h_3=\frac{e}{m}\frac{1}{\alpha (kx_c)}\), there is only one prey only equilibrium point \(E_k\) of the system. The Jacobian matrix evaluated at \(E_k\) is
J has an eigenvalue \(\xi =0\). Let V and W be the eigenvectors corresponding to the eigenvalue \(\xi =0\) for J and \(J^T\), respectively. Then one can calculate,
Using the expressions for V and W, we get

\(W^Tf_h(x_1,y_1;h_3)=0,\)

\(W^T[Df_h(x_1,y_1;h_3)V]=\frac{m^2 k}{\alpha e(kx_c)^2}<0\) and

\(W^T[D^2f(x_1,y_1;c_1)(V,V)]=\frac{2m^2}{rke}<0\).
Thus, by Sotomayor’s theorem, we conclude that the model system undergoes a transcritical bifurcation as the parameter h passes through the critical value \(h_3\). We draw a bifurcation diagram with respect to \(x_c\) when h is fixed (Fig. 4) to demonstrate the system behavior more succinctly. We take initial condition as \(x(0)>x_c,~y(0)>0\) always, because system (4) is not define for \(x<x_c\). Observe that population oscillate (blue for prey and black for predator) around the unstable coexistence equilibrium point (\(E^{*}\)) (red solid lines for both \(x^{*}\) and \(y^{*}\)) at the lower range of degree of prey refuge (\(x_c\)), at lower threshold value of \(x_c\) (here \(x_c=212\)) system experiences Hopfbifurcation and becomes stable up to the upper threshold value (here \(x_c=810\)) (green solid line for \(x^{*}\) and blue solid line for \(y^{*}\)) and till now the prey only equilibrium point \(E_k\) remains unstable (red dash lines). At upper threshold value of \(x_c\) two equilibrium points \(E^{*}\) and \(E_k\) coincide and become \(E_k\) and transcritical bifurcation occurs. If further we increase the value of \(x_c\), \(E^{*}\) becomes unstable node (here it is biologically infeasible and indicated by grey solid line) and system converges to prey only equilibrium point \(E_k\) (green dash lines). It is very interesting to observe that prey density (x) never comes less the threshold density (\(x=x_c\), here the black dot line represents \(x=x_c\)). Similarly we draw bifurcation diagram with respect to h with fixed \(x_c=200\) in the similar fashion (Fig. 5).
Filippov regularization
The most interesting phenomena of Gause model is the fact when a trajectory falls on the vertical part of the prey nullcline, then it cannot be continued any further. Indeed, as the “trajectory” cannot leave the line \(x=x_c\) and it follows that \(\frac{dx}{dt}=0\). In other words, the Gause model is not well posed because trajectories are not defined when they fall on the vertical part of the prey nullcline. This is a consequence of the fact that the Gause functional response has a “jump” at the critical prey density, because such models may not have solutions. After Gause et al. (1936), (Filippov 1960, 1988) introduces a new solution concept for such models. The crucial step is provided through suitable definition of the vector field at the critical prey density, which we briefly describe now. The Filippov solution concept (Filippov 1960, 1988; Colombo and Křivan 1993; Utkin et al. 2009; Yang et al. 2013) applied to the Gause model defines a new vector field at the critical prey density \(x_c\) as the line segment with end points given by the two adjacent vector fields \(F^1\) and \(F^2\). Here \(F^1=(rx_c(1\frac{x_c}{k}),my)\) stands for the vector field is defined by the righthand side of model (3) and \(F^2=(rx_c(1\frac{x_c}{k})\frac{\alpha (xx_c) y}{1+\alpha h (xx_c)},\frac{e\alpha (xx_c) y}{1+\alpha h (xx_c)}my)\) for the vector field is defined by the righthand side of model (4). This new (multi valued) vector field is given by
where \(0\le \beta \le 1\). In other words, this vector field associates to every point along the vertical part of prey nullcline, a whole set of possible directions is given by F. The definition of the Filippov field redefines the functional response (2) at the critical prey density to
It follows that under this new definition \(g(x_c)\) is the line segment \(\bigg [0,\frac{\alpha x_c}{1+\alpha h x_c}\bigg ]\) that fills the gap in Gause functional response (Fig. 1b). This definition is very natural as it reflects the fact that at the critical prey density the functional response does not specify exactly the prey consumption by predators.
Existence and uniqueness of trajectories of the Gause model
Let \(n(n_x, n_y)=(1,0)\) be the vector perpendicular to the line \(x=x_c\) in the preypredator density phase plane. Projection of the two vector fields are given by the right hand sides of (3) (denoted as \(F^1\)) and (4) (denoted as \(F^2\)) are \(\langle n,F^1\rangle =rx_c(1\frac{x_c}{k})\) and \(\langle n,F^2\rangle =rx_c(1\frac{x_c}{k})\frac{\alpha (xx_c)}{1+\alpha h (xx_c)}\). The existence of trajectories for the Gause model follows from general existence theorem that can be found in Filippov (1988) (Colombo and Křivan 1993). Uniqueness of trajectories for the Gause model follows from the fact that \(\langle n,F^1\rangle =\langle n,F^2\rangle +\frac{\alpha (xx_c)}{1+\alpha h (xx_c)}\).
and the manifold
In order to investigate the global dynamics of Filippov system [system (1) with functional response (6)], the qualitative behaviors of the subsystems which are determined by vector field \(F^1\) or \(F^2\) and the dynamics which is defined on the discontinuity manifold \(\sum\) are crucial. The dynamics for vector fields \(F^1\) and \(F^2\) can be analyzed by using classical qualitative techniques of differential dynamic systems previously. However, the dynamics on the switching manifold \(\sum\) can be complex and can be studied by using the wellknown Filippov’s convex method (Filippov 1988) or Utkin’s equivalent control method (Utkin et al. 2009).
Let \(\sigma =\langle n,F^1\rangle \langle n,F^2\rangle\), then \(\sum _s=\{x\in \sum : \sigma <0\}\). The sliding mode domain \(\sum _s\) can be distinguished by the following regions (Kuznetsov et al. 2003; Buzzi et al. 2006, 2010):

1.
Escaping region: if \(\langle n,F^1\rangle <0\) and \(\langle n,F^2\rangle >0\) which would imply nonuniqueness of trajectories (Filippov 1988; Colombo and Křivan 1993).

2.
Sliding region: if \(\langle n,F^1\rangle >0\) and \(\langle n,F^1\rangle >0\), trajectories are pushed from both below and above to the line \(x=x_c\).
Thus, the state space is (\(S_1\bigcup S_2\bigcup \sum\)) and the model equations are
where, \(F^i\) are described previously. Sliding occurs on the segment \(\sum\) which is delimited by two intersections \(S_1\) and \(S_2\) of \(\sum\) with two nullclines \(\frac{dy}{dt}=0\). As first pointed out by Filippov, these points which are called the tangent points (Kuznetsov et al. 2003) at which \(F^2\) is tangent to \(\sum\) is very important for bifurcation analysis. Sliding motion on \(\sum\) obeys the smooth scalar differential equation
where F is the unique convex combination of \(F^1\) and \(F^2\), which is given by (5) and \(\beta =1\frac{r(kx_c)\{1+\alpha h x_c\}}{k\alpha y}\). Now the system is well posed and all the trajectories are defined throughout the range of x. This phenomena [system (1) with functional response (6)] is shown by Fig. 6 (each figures corresponds to the same in Fig. 3).
To analyze the Gause model (Gause 1934), it is very important to know the dynamics of the system along the vertical part of prey nullcline where the trajectories of system (1) falls on the vertical part of the prey nullcline and the predator density satisfies \(y>\frac{r(kx_c)\{1+\alpha hx_c\}}{\alpha k}=y_c\), the trajectories cannot leave the nullcline and moves vertically, i.e., \(\frac{dx}{dt}=0\), i.e., along \(\beta =1\frac{r(kx_c)\{1+\alpha h x_c\}}{k\alpha y}\). This trajectory corresponds to the vector \(f^{F}=(0,erx_c(1\frac{x_c}{k})my)\in F\) and the population dynamics is:
This equation depicts the dynamics of the system (1) along the line \(x=x_c\) as long as \(y>y_c\), then trajectories are forced to leave the line \(x=x_c\) and enter the region \(x>x_c\). This also follows from the considerations of the existence and uniqueness of trajectories of the Gause model.
Adaptive refuge use by the prey
The adaptation of an organism cannot easily be separated from the environment in which it lives. Insect larvae from stagnant aquatic environments in ditches and sloughs can survive longer without oxygen than can related species from wellaerated streams and rivers; species of marine snails that occurs high in the intertidal zone, where they are frequently exposed to air, tolerate desiccation better than do species from lower levels. These are examples of specializations that suit organisms to particular, restricted ranges of environmental conditions (Ricklefs and Miller 2000). It is very convenient that when predators are present in the system, naturally prey decreases its activities or changes suitable habitat to avoid predation risk (Werner and Gilliam 1984; Holbrook and Schmitt 1988; Brown and Alkon 1990; Brown 1998; Lima 1998a, b; Sih 1998; Peacor and Werner 2001; Preisser et al. 2005). Reduction in prey activities is a prominent example of a behavioral refuge whether prey reduces their activities or moves to a physical refuge leads to a tradeoff due to live in the refuge (behavioral or physical) which increases the survival rate and decreases the predation rate and other components of prey fitness. The model of this tradeoff is:
Here, \(u_1\) is the proportions of prey out of the refuge and \(u_2\) is so for refuge and \(u_1+u_2=1\). Maximum refuge size is \(x_c\), so in the reserve area prey grows logistically up to the size \(x_c\) and naturally in the unreserve area prey grows logistically up to \((kx_c)\), because k is the maximum strength of prey in the total habitat in absence of predation. \(r_1\) and \(r_2\) are the intrinsic growth rates of prey out and inside the refuge. Figure 7 depicts the behavior of the system with increasing the proportion of prey out of the refuge \(u_1\) (decreasing the proportion of prey in the refuge \(u_2\)) as the bifurcation parameter. When \(u_1<u_2\), i.e., maximum population can avoid the foraging efficiency of predator, then predator either goes to extinction or survive with very low density and at the intermediate range of \(u_1,~ u_2\) and \(u_1>u_2\), then both population can survive at their commensurate equilibrium density or at fluctuating mode, this adaptive nature is validated by Fig. 7. Now the question: What is the optimal prey strategy \((u_1,u_2)\)?
If proportion \(u_1\) of prey is out of the refuge, so their payoff is \(V_1=r_1(1\frac{u_1 x}{(kx_c)})\frac{\alpha y}{1+\alpha hu_1 x}\). As the proportion of prey out side the refuge \((u_1)\) increases, payoff will increase, because the chance that a single prey will be captured by a predator decreases due to the risk dilution effect (Foster and Treherne 1981). As we assume that prey in the refuge is completely protected from predation, the payoff in the refuge is \(V_2=r_2(1\frac{u_2 x}{x_c})\). Thus, fitness of a mutant prey (with strategy \(\tilde{u}=(\tilde{u}_1,\tilde{u}_2)\)) in a monomorphic resident population (with distribution \(u=(u_1,u_2)\)) is given by the mean payoff
Mutant fitness is not only density dependent but also frequency dependent, because it depends on the resident strategy.
Adaptive alternative food selection by predator
Optimization of diet choice by predator has been explored theoretically by number of authors (Schoner 1969, 1971; Rapport 1971; Cody 1974; Orians and Pearson 1979; Townsend and Hughes 1981; Charnov and Stephens 1988; Krebs and Kacelnik 1991). These treatments agree with MacArthur and Pianka’s prediction (MacArthur and Pianka 1966). Mathematically, the problem of optimal diet choice has been characterized by the switching phenomena of “classic” model of diet choice (Ricklefs and Miller 2000). Prey switching is frequencydependent predation, where the predator preferentially consumes the most common type of prey. The reason a predator may switch from eating one prey to another, is because it may increase an individual’s foraging efficiency and therefore its inclusive fitness (Hughes and Croy 1993; Cornell 1976). It has been argued that frequencydependent predation is predicted from optimal foraging theory (Hubbard et al. 1982). Predator’s preference for prey might stabilize preypredator dynamics (Oaten and Murdoch 1975). Oaten and Murdoch (1975) assume that the proportion of prey in the predator’s diet choice increases with increasing prey proportion in the environment faster than linearly what they call switching. So, the Holling type II functional response for the primary resource is (Holt 1983; Abrams 1999)
where \(u~(u_a=1u)\) is the predator’s preference for the primary (alternative) prey \(x~(x_a)\). At the extreme case where predators attack on the more abundant species and the functional response is piecewise continuous (Fig. 1b)
and corresponding numerical response is
where \(x_a\) is the alternative prey abundance and \(\alpha _a,~h_a,~e_a\) are the attack rate, handling time and food conversion efficiency of predator for their alternative prey. So, our model system (1) becomes when \(x<x_a\)
at each point to the left of \(x=x_c\) in the preypredator phase space, prey population grows logistically and predator increases (decreases) if \(x_a>\frac{m}{\alpha _a(e_ah_a m)}\) \((x_a<\frac{m}{\alpha _a(e_ah_a m)})\). When prey population density above the critical prey density (\(x>x_c\)), then system (1) becomes
The prey population decreases if predator population is very large (\(y>y^c=\frac{rx(kx)\{1+\alpha h(xx_c)\}}{\alpha (xx_c)}\)) in the vicinity and to the right of \(x=x_c\). For \(x=x_c\) population dynamics is
Here the trajectories can not leave the switching line \(x=x_c\) when predator density is higher than \(y^c\), then corresponding preference is
Then Eq. (17) becomes
This system has the equilibrium as
There are three distinguish features depending upon the abundance of alternative food source (\(x_a\)).

When alternative resource is very small, i.e., \(x_a<x^{*}\) [\(x^{*}\) is the equilibrium density of prey of model system (4)], i.e., when predator feeds on primary prey only, then both population coexist in limit cycle (Fig. 8i).

When alternative food size is intermediate, i.e., \(x^{*}<x_a<X_a=x_c+\frac{m}{\alpha _a (e_amh_a)}\), then there exists a stable coexistence equilibrium point \(E_a^{*}(x_a^{*},y_a^{*})\) where \(x_a^{*}=x_a\) and \(y_a^{*}=\frac{erx(kx)\{1+\alpha h u (xx_c)+\alpha _a h_a u_a x_a\}}{k\{m+(mh_ae_a)\alpha _au_ax_a+m\alpha hu(xx_c)\}}\) (Fig. 8ii).

If alternative food resource approaches to \(X_a\), then predator equilibrium is tending to infinity and when \(x_a>X_a\), then there coexistence is not possible because it is very unrealistic to assume that alternative resource can reach such a high density and not be influenced by predation or competition with essential resource directly or apparently.
Discussion
In the present study, we have analyzed a preypredator system which is introduced by Gause (1934) and his coworkers (Gause et al. 1936) to examine some cycles of preypredator interaction in some experiments. They replace the linear functional response of Lotka and Volterra by some saturating functional response which is zero below a critical prey density and is saturating above which with respect to prey density. At the critical prey density there is discontinuity as well as a “jump” of the functional response of predator. So, trajectories are not well defined throughout the range of prey density and also not well posed. So, at the vicinity of the critical prey density there is some confusion. But in nature any population may increase or decrease their density over time. So, naturally it is very convenient that prey density must cross this critical density at any time, so analysis at that critical prey density is very essential not only the purpose of ecological clarification but also for mathematical interest. Natural habitat, mostly, is patchy. Different patches or sites characterize differently. Some sites are suitable for prey or some are for predator. So patch selection by both population is a very essential and important ecological phenomena. In this present study, we consider that there is two patches, one is reserve for prey and other is non reserve. In the reserve patch prey population can successfully refuse the foraging efficiency of predator. Here we consider the critical prey density, the size of refuse limit of Gause model. After successful refusing, the remaining prey population is not able to take shelter in the reserve patch and then they come into the interaction with predator. Here at this critical prey density the system is not well defined. Below this critical density predator is forced to go extinction due to lack of food and prey population grows logistically without any predation risk. But when prey density is above this critical density, then surplus of prey and predator populations can survive in the fashion of classical preypredator relation and can coexist in either limit cycle mode or at equilibrium. Where as Gause et al. (1936) show that a prey refuge can lead to preypredator coexistence as the same model without refuge. Also, here the cyclic behavior of both population coincides with coexistence equilibrium or vice versa through Hopfbifurcation due to the size of refuse patch or food handling ability of predator and also the decreasing availability of food can able to make predator extinction from the coexistence of population through transcritical bifurcation. For searching the answer of discontinuity of Gause model in this article, an approach is developed by Filippov (1960) is used, we show how the trajectories in the Gause model can be defined at the vicinity of prey density. This technique can explain mathematically and ecologically the movement of trajectories in the vicinity of prey density.
Generally, predator depends on more than a single prey population. Predator population’s growth under a given set of conditions behaves uniquely to the level of each of its prey resources (Ricklefs and Miller 2000; Murdoch 1969; Clarke 1962; Charnov 1976; Stephens and Krebs 1986). According to Liebig’s law of minimum (Liebig 1840) whichever the resources is reduced to the value of critical density, first it limits the growth of predator population. Here, in this model the primary prey is referred ecologically as essential resources, an interaction of the predator and the prey resources is considered when one of the prey resource is essential and other is not. If essential prey resource density drops at the critical level, the predator population for their survival switches to non essential resources (Ricklefs and Miller 2000; Murdoch 1969; Clarke 1962; Charnov 1976; Stephens and Krebs 1986). Our model depicts that the survivability and stability of predator depend on switching behavior from essential to non essential prey resources. Ecologically it is established that if essential resource is decreased to the critical level and if predator switch to the non essential one, then more non essential resources will be consumed because non essential resource will provide less nutrient than that of essential resource. So, the condition i.e., the dependency of predator on alternative prey (non essential resource) does not persist for the long time. But during this time essential prey density will again be increased due to the absence of predation pressure on them and exceeds the critical density and predator’s dependency on alternative prey is again switched back to primary prey population.
References
Abrams PA (1999) The adaptive dynamics of consumer choice. Am Nat 153:83–97
Abrams PA, Matsuda H (1996) Fitness minimization and dynamic instability as a consequence of predatorprey coevolution. Evol Ecol 10:167–186
Alstad D (2001) Basic populations models of ecology. Prentice Hall Inc, New Jersey
Anderson O (1984) Optimal Foraging by largemouth bass in structured environments. Ecology 65:851–861
Anderson TW (2001) Predator responses, prey refuges and densitydependent mortality of a marine fish. Ecology 82(1):245–257
Berryman AA (1992) The origins and evolutions of predatorprey theory. Ecology 73:1530–1535
Brown JS (1998) Game theory and habitat selection. In: Dugatkin LA, Hudson KR (eds) Game theory & animal behavior. Oxford University Press, New York, pp 188–220
Brown JS, Alkon PA (1990) Testing values of crested porcupine habit by experimental food patches. Oecologia 83:512–518
Brown JS, Kotler BP (2004) Hazardous duty pay and the foraging cost of predation. Ecol Lett 7:999–1014
Buzzi CA, Silva PR, Teixeira MA (2006) A singular approach to discontinuous vector fields on the plane. J Differ Equ 231:633–655
Buzzi CA, Carvalho TD, Silva PR (2010) Canard cycles and Poincaré index of nonsmooth vector fields on the plane. J Dyn Control Syst 2:173–193
Charnov EL (1976) Optimal foraging: attack strategy of a mantid. Am Nat 110:141–151
Charnov EL, Stephens DW (1988) On the evolution of host selection in solitary parasitoids. Am Nat 132:707–722
Chen L, Chen F, Chen L (2010) Qualitative analysis of a predator prey model with Holling type II functional response incorporating a constant prey refuge. Nonlinear Anal Real World Appl 11(1):246–252
Clarke BC (1962) Balanced polymorphism and the diversity of sympatric species. In: Nichols D (ed) Taxonomy and geography. Systematics Association Publication, Oxford, pp 47–70
Cody ML (1974) Optimization in ecology. Science 183:1156–1164
Colombo R, Křivan V (1993) Selective strategies in food webs. IMA J Math Appl Med Biol 10:281–291
Cornell H (1976) Search strategies and the adaptive significance of switching in some general predators. Am Nat 110:317–320
Cressman R, Křivan V (2006) Migration dynamics for the ideal free distribution. Am Nat 168:384–397
Cressman R, Křivan V (2013) Twopatch population models with adaptive dispersal: the effects of varying dispersal speeds. J Math Biol 67:329–358
Filippov AF (1960) Differential equations with discontinuous righthand side. Matematicheskii sbornik 51:99–128 (in Russian English translation published in American Mathematical Society Translations, Series 2, 199–231, 1964)
Filippov AF (1988) Differential equations with discontinuous righthand sides. Kluwer Academic Publishers, Dordrecht
Foster WA, Treherne JE (1981) Evidence for the dilution effect in the selfish herd from fish predation on a marine insect. Nature 293:466–467
Fryxell JM, Lundberg P (1994) Diet choice and predatorprey dynamics. Evol Ecol 8:407–421
Fryxell JM, Lundberg P (1997) Individual behavior and community dynamics. Chapman & Hall, London
Gause GF (1934) The struggle for existence. Williams and Wilkins, Baltimore
Gause GF, Smaragdova NP, Witt AA (1936) Further studies of interaction between predators and prey. J Animal Ecol 5:1–18
GonzalezOlivares E, RamosJiliberto R (2003) Dynamic consequences of prey refuges in a simple model system: more prey, fewer predators and enhanced stability. Ecol Model 166:135–146
Holbrook SJ, Schmitt RJ (1988) The combine effects of predation risk and food reward on patch selection. Ecology 69:125–134
Holling CS (1959) Some characteristics of simple types of predation and parasitism. Can Entomol 91:385–398
Holt RD (1983) Optimal foraging and the form of the predator isocline. Am Nat 122:521–541
Hubbard SF, Cook RM, Glover JG, Greenwood JJD (1982) Apostatic selection as an optimal foraging strategy. J Animal Ecol 51:625–633
Hughes RN, Croy MI (1993) An experimental analysis of frequencydependent predation (switching) in the 15spines Stickleback, Spinachia spinachia. J Animal Ecol 62:341–352
Ives AR, Dobson AP (1987) Antipredator behaviour and the population dynamics of simple predatorprey systems. Am Nat 130:431–447
Jana D (2013) Chaotic dynamics of a discrete predatorprey system with prey refuge. Appl Math Comput 224:848–865
Jana D (2014) Stabilizing effect of prey refuge and predator’s interference on the dynamics of prey with delayed growth and generalist predator with delayed gestation. Int J Ecol 12 (Article ID 429086)
Jana D, Bairagi N (2014) Habitat complexity, dispersal and metapopulations: macroscopic study of a predatorprey system. Ecol Complex 17:131–139
Jana D, Agrawal R, Upadhyay RK (2015) Dynamics of generalist predator in a stochastic environment: effect of delayed growth and prey refuge. Appl Math Comput 268:1072–1094
Johnson WD (2006) Predation, habitat complexity and variation in density dependent mortality of temperate reef fishes. Ecology 87(5):1179–1188
Kar T (2005) Stability analysis of a preypredator model incorporating a prey refuge. Commun Nonlinear Sci Numer Simul 10(6):681–691
Krebs JR, Kacelnik A (1991) Decisionmaking. In: Krebs JR, Davies NB (eds) Behavioural ecology: an evolutionarily approach. Blackwell Scientific Publications, Oxford, pp 105–136
Křivan V (1997) Dynamic ideal free distribution: effects of optimal patch choice on predatorprey dynamics. Am Nat 149:164–178
Křivan V (1998) Effects of optimal antipredator behavior of prey on predatorprey dynamics: role of refuges. Theor Popul Biol 53:131–142
Křivan V (2011) On the Gause predatorprey model with a refuge: a fresh look at the history. J Theor Biol 274:67–73
Křivan V (2013) Behavioral refuges and predatorprey coexistence. J Theor Biol 339:112–121
Křivan V, Eisner J (2003) Optimal foraging and predatorprey dynamics III. Theor Popul Biol 63:269–279
Kuang Y, Freedman HI (1988) Uniqueness of limit cycles in Gausetype models of predatorprey systems. Math Biosci 88:67–84
Kuznetsov YA, Rinaldi S, Gragnani A (2003) One parameter bifurcations in planar Filippov systems. Int J Bifurc Chaos 13:2157–2188
Liebig J (1840) Chemistry in its application to agriculture and physiology. Taylor and Walton, London
Lima SL (1998a) Nonlethal effects in the ecology of predatorprey interactions. Bioscience 48:25–34
Lima SL (1998b) Stress and decision making under the risk of predation: recent developments from behavioral, reproductive and ecological perspectives. Stress Behav 27:215–290
Lima SL, Dill LM (1990) Behavioral decisions made under the risk of predation: a review and prospectus. Can J Zool 68:619–640
Lotka AJ (1925) Elements of physical biology. Williams & Winlkins, Baltimore
Ma B, Abrams P, Brassil C (2003) Dynamic versus instantaneous models of diet choice. Am Nat 162:668–684
MacArthur RH, Pianka ER (1966) On optimal use of patchy environment. Am Nat 100:603–609
Maynard Smith J (1974) Models in ecology. Cambridge University Press, Cambridge
Murdoch WW (1969) Switching in generalist predators: experiments on prey specificity and stability of prey populations. Ecol Monogr 39:335–354
Oaten A, Murdoch WW (1975) Switching, functional response and stability in predatorprey systems. Am Nat 109:299–318
Orians GH, Pearson NE (1979) On the theory of central place foraging. In: Horn DJ, Mitchell R, Stair GR (eds) Analysis of ecological systems. Ohio State University Press, Columbus, pp 155–177
Peacor SD, Werner EE (2001) The contribution of traitmediated indirect effects to the net effects of a predator. Proc Natl Acad Sci USA 98:3904–3908
Perko L (2001) Differential equations and dynamical systems. Springer, New York
Preisser EL, Bolnick DI, Benard MF (2005) Scared to death? The effects of intimidation and consumption in predatorprey interactions. Ecology 86:501–509
Rapport DJ (1971) An optimization model of food selection. Am Nat 105:575–587
Ray S, Straškraba M (2001) The impact of detritivorous fishes on the mangrove estuarine system. Ecol Model 140:207–218
Ricklefs RE, Miller GL (2000) Ecology, 4th edn. W. H, Freeman and Company, New York
Roy M, Mandal S, Ray S (2008) Detrital ontogenic model including decomposer diversity. Ecol Model 215:200–206
Robert AA (1976) The effect of predator functional response and prey productivity on predatorprey stability: a graphical approach. Ecology 57:609–612
Rosenzweig ML (1969) Why the prey curve has a hump. Am Nat 103:81–87
Rosenzweig ML, MacArthur RH (1963) Graphical representation and stability conditions of predatorprey interactions. Am Nat 97:209–223
Ruxton GD (1995) Short term refuge use and stability of predatorprey models. Theor Popul Biol 47:1–17
Schoener TW (1969) Models of optimal size for solitary predators. Am Nat 103:277–313
Schoner TW (1971) Theory of feeding strategies. Annu Rev Ecol Syst 2:369–404
Sih A (1980) Optimal behavior: can forages balance two conflicting demands? Science 210:1041–1043
Sih A (1986) Antipredator responses and the perception of danger by mosquito larvae. Ecology 67:434–441
Sih A (1987) Prey refuges and predatorprey stability. Theor Popul Biol 31:1–12
Sih A (1998) Game theory and predatorprey response races. In: Dugatkin LA, Hudson KR (eds) Game theory & animal behavior. Oxford University Press, New York, pp 221–238
Solisa FJ, KuCarrillo RA (2014) Generic predation in age structure predatorprey models. Appl Math Comput 231:205–213
Stephens DW, Krebs JR (1986) Foraging theory. Princeton University Press, Princeton
Townsend CT, Hughes RN (1981) Maximizing net energy returns from foraging. In: Townsend CR, Calow P (eds) Physiological ecology: an evolutionary approach to resource use. Blackwell, Oxford, pp 86–108
Utkin VI, Guldner J, Shi JX (2009) Sliding mode control in electromechanical systems, 2nd edn. Taylor and Francis, New York
Volterra V (1931) Lecons sur la theorie mathematique de la lutte pour la vie. GauthierVillars, Paris
Werner EE, Gilliam JF (1984) The ontogenetic niche and species interaction sizestructured populations. Annu Rev Ecol Syst 15:393–425
Yang J, Tang S, Cheke RA (2013) Global stability and sliding bifurcations of a nonsmooth Gause predatorprey system. Appl Math Comput 224:9–20
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Jana, D., Ray, S. Impact of physical and behavioral prey refuge on the stability and bifurcation of Gause type Filippov preypredator system. Model. Earth Syst. Environ. 2, 24 (2016). https://doi.org/10.1007/s408080160077y
Received:
Accepted:
Published:
DOI: https://doi.org/10.1007/s408080160077y