Nutrient retention by predators undermines predator coexistence on one prey

Abstract

Contemporary theory of predator coexistence through relative non-linearity in their functional responses strongly relies on the Rosenzweig and MacArthur (1963) equations in which the (autotrophic) prey exhibits logistic growth in the absence of the predators. This implies that the prey is limited by a resource such as light or space, which availability is independent of the predators. This assumption does not hold under nutrient limitation where both prey and predators bind resources such as nitrogen and phosphorus in their biomass. Furthermore, the prey’s resource uptake rate is assumed to be linear and the predator-prey system is considered to be closed. All these assumptions are unrealistic for many natural systems. Here, we show that nutrient retention in predator biomass strongly hampers the coexistence of two predators on one prey because it stabilizes the dynamics. In contrast, a non-linear resource uptake rate of the prey slightly promotes predator coexistence. Our study highlights that predator coexistence does depend not only on differences in the curvature of their functional responses but also on the type of resource constraining the growth of their prey. This has far-reaching consequences for the relative importance of fluctuation-dependent and fluctuation-independent mechanisms of species coexistence in natural systems where autotrophs experience light or nutrient limitation.

Appendices

Appendix 1. Necessary conditions and parameter range of potential species coexistence

Following the general methodology of Hsu et al. (1978) and Xiao and Fussmann (2013), we here derive ecological conditions for species coexistence that determine a range of parameters for which species coexistence might be possible. Although these conditions are not sufficient, they are necessary, implying that outside this parameter range species coexistence is impossible. For simplicity, we will assume predator one (C₁) to exhibit a linear functional response and predator two (C₂) to have a (non-linear) functional type II response.

Persistence of predator-prey systems in the absence of exploitative competition

Predator coexistence necessarily requires persistence of the two different predator-prey systems comprising either C₁ or C₂ next to the prey. Hence, each predator has to be able to invade the system that only includes the prey. In the absence of C₁ and C₂, the prey is approaching its single-species equilibrium${R}_S^{\ast }=1-\frac{\delta }{\left(1-{h}_R\delta \right)}$. The two predators thus exhibit positive invasion growth rates when their grazing rates for ${R}_S^{\ast }$ are larger than the sum of their different loss rates through mortality $\left({d}_{C_i}\right)$ and dilution (δ), respectively, i.e. ${a}_{C_1}{R}_S^{\ast }>{d}_{C_1}+\delta$ for C₁ and $\frac{a_{C_2}{R}_S^{\ast }}{1+{h}_{C_2}{a}_{C_2}{R}_S^{\ast }}>{d}_{C_2}+\delta$ for C₂. Consequently, the death rates that allow persistence of the two different predator-prey systems are limited to ${d}_{C_1}<{a}_{C_1}\left(1-\frac{\delta }{\left(1-{h}_R\delta \right)}\right)-\delta$ for C₁ and to ${d}_{C_2}<\frac{a_{C_2}\left(1-\left({h}_R+1\right)\delta \right)}{\left(1-{h}_R\delta \right)+{h}_{C_2}{a}_{C_2}\left(1-\left({h}_R+1\right)\delta \right)}-\delta$ for C₂. These conditions put natural limits on the two-dimensional parameter space in the ${d}_{C_1}-{d}_{C_2}$ plane for which predator coexistence might be possible. The first necessary condition for predator coexistence is thus given by the following:

$$ {N}_1:{\displaystyle \begin{array}{c}{d}_{C_1}<{d}_{C_1}^U\\ {}{d}_{C_2}<{d}_{C_2}^U\end{array}} $$

with ${d}_{C_1}^U={a}_{C_1}\left(1-\frac{\delta }{\left(1-{h}_R\delta \right)}\right)-\delta$ and ${d}_{C_2}^U=\frac{a_{C_2}\left(1-\left({h}_R+1\right)\delta \right)}{\left(1-{h}_R\delta \right)+{h}_{C_2}{a}_{C_2}\left(1-\left({h}_R+1\right)\delta \right)}-\delta$ denoting the upper limits of the death rates of the two different predators C₁ and C₂, respectively.

Invasion of C₂

The linear functional response of C₁ ensures that the C₁ − R system will approach a stable equilibrium with prey density ${R}_{C_1}^{\ast }=\frac{d_{C_1}+\delta }{a_{C_1}}$. A necessary condition for predator coexistence is that C₂ is able to invade the C₁ − R system. Hence, the invasion growth rate of C₂ for ${R}_{C_1}^{\ast }$ has to be larger than 0, i.e. $\frac{1}{C_2}\frac{d{C}_2}{dt}=\frac{a_{C_2}{R}_{C_1}^{\ast }}{1+{h}_{C_2}{a}_{C_2}{R}_{C_1}^{\ast }}-{d}_{C_2}-\delta >0$, which corresponds to $\frac{a_{C_2}\left({d}_{C_1}+\delta \right)}{a_{C_1}+{h}_{C_2}{a}_{C_2}\left({d}_{C_1}+\delta \right)}-{d}_{C_2}-\delta >0$. Therefore, the second necessary condition for predator coexistence is given by the following:

$$ {N}_2:{d}_{C_2}<\frac{a_{C_2}\left({d}_{C_1}+\delta \right)}{a_{C_1}+{h}_{C_2}{a}_{C_2}\left({d}_{C_1}+\delta \right)}-\delta . $$

This implies that the equilibrium density of the prey in the C₂ − R system has to be smaller than ${R}_{C_1}^{\ast }$, i.e. ${R}_{C_2}^{\ast }=\frac{d_{C_2}+\delta }{\left({a}_{C_2}-\left({d}_{C_2}+\delta \right){h}_{C_2}{a}_{C_2}\right)}<{R}_{C_1}^{\ast }$. In other words, the minimum prey requirement of C₂ has to be lower than the minimum prey requirement of C₁. In contrast, when ${R}_{C_2}^{\ast }>{R}_{C_1}^{\ast }$, C₂ cannot invade the C₁ − R system.

Competitive exclusion of C₁

Since the curvature of the type II functional response is concave, the time-averaged per capita net growth rate of C₂ will always be larger than or equal to the time-averaged per capita net growth rate of a corresponding predator that has a linear functional response with the same per capita net growth rates at the minimum (R_min = 0) and the maximum $\left({R}_{\mathrm{max}}={R}_S^{\ast }=1-\frac{\delta }{\left(1-{h}_R\delta \right)}\right)$ feasible prey densities as C₂. Hence, the following inequality holds:

$$ \int \frac{1}{C_2}\frac{d{C}_2}{dt} dt=\int \frac{a_{C_2}R(t)}{1+{h}_{C_2}{a}_{C_2}R(t)} dt-{d}_{C_2}-\delta \ge \frac{a_{C_2}\int R(t) dt\kern0.5em }{1+{h}_{C_2}{a}_{C_2}{R}_S^{\ast }}-{d}_{C_2}-\delta =\frac{a_{C_2}{\overline{R}}_{C_2}}{1+{h}_{C_2}{a}_{C_2}{R}_S^{\ast }}-{d}_{C_2}-\delta $$

with ${\overline{R}}_{C_2}=\int R(t) dt$being the time-averaged prey density of the C₂ − R system. Since $\int \frac{1}{C_2}\frac{d{C}_2}{dt} dt=0$ at equilibrium, the following inequality holds: $\frac{a_{C_2}{\overline{R}}_{C_2}}{1+{h}_{C_2}{a}_{C_2}{R}_S^{\ast }}-{d}_{C_2}-\delta \le 0$. Setting the term $\frac{a_{C_2}}{1+{h}_{C_2}{a}_{C_2}{R}^{\ast }}{\overline{R}}_{C_2}-{d}_{C_2}-\delta$ to 0 and solving for ${\overline{R}}_{C_2}$ thus gives us an upper limit for the temporal average of the prey density in the corresponding predator-prey system that is denoted by${\overline{R}}_{C_2}^U=\frac{\left({d}_{C_2}+\delta \right)}{a_{C_2}}\left(1+{h}_{C_2}{\mathrm{a}}_{C_2}{R}_S^{\ast}\right)$. Therefore, whenever C₁ has a minimum prey requirement ${R}_{C_1}^{\ast }$higher than ${\overline{R}}_{C_2}^U$, it will inevitably be outcompeted by C₂. Consequently, the following inequality has to be met for the two predators to be able to coexist:

$$ \frac{1}{C_1}\frac{d{C}_1}{dt}={a}_{C_1}{\overline{R}}_{C_2}^U-{d}_{C_1}-\delta >0\leftrightarrow {a}_{C_1}\frac{\left({d}_{C_2}+\delta \right)}{a_{C_2}}\left(1+{h}_{C_2}{a}_{C_2}{R}_S^{\ast}\right)-{d}_{C_1}-\delta >0 $$

This leads to the following third necessary condition of predator coexistence:

$$ {N}_3:{d}_{C_2}>\frac{\left({d}_{C_1}+\delta \right){a}_{C_2}}{a_{C_1}\left(1+{h}_{C_2}{a}_{C_2}{R}_S^{\ast}\right)}-\delta . $$

Nevertheless, ${R}_{C_1}^{\ast }<{\overline{R}}_{C_2}^U$does not ensure that C₁ can invade the C₂ − R system. For this to happen, the minimum prey requirement of C₁ has to be lower than the actual temporal average of the prey density in the corresponding predator-prey system, i.e. ${R}_{C_1}^{\ast }<{\overline{R}}_{C_2}$.

Hopf bifurcation in predator-prey models with type II functional response

Predator coexistence through relative non-linearity in their functional responses requires oscillations in the abundances of the different species. It is well known from the theory of non-linear dynamics that limit cycles only emerge in systems of ordinary differential equations with more than 2 dimensions when there is some degree of non-linearity, i.e. non-linear density-dependence, in the per capita rates of the corresponding state variables. Hence, in the predator-prey system defined by Eqs. 9a, 9b and 9c, oscillations can only occur in the presence of C₂. Therefore, we performed a linear stability analysis of the two-dimensional R − C₂ system, in order to evaluate the influence of nutrient retention by the predators, the effect of a non-linear resource uptake rate of the prey and the impact of a shared loss rate for all species on the stability of the predator-prey dynamics. For simplicity, we will neglect the index of predator two for its state variable and parameters and thus consider the following predator-prey model:

$$ \frac{dR}{dt}=\left[\frac{\left(1-\left(R+\eta C\right)\right)}{1+{h}_R\left(1-\left(R+\eta C\right)\right)}-\frac{aC}{1+ haR}-\delta \right]R $$

(11a)

$$ \frac{dC}{dt}=\left[\frac{aR}{1+ haR}-d-\delta \right]C $$

(11b)

Throughout the analysis, we will focus on parameter values where the predator is able to persist, i.e. d < d^U. To start, we calculate the interior equilibrium of the prey (R^∗) and predator (C^∗) species for the different model parametrizations used in the main text by setting Eqs. 11a and 11b to 0, i.e. $\frac{dR}{dt}=0$ and $\frac{dC}{dt}=0$, and solving for the corresponding values of the state variables satisfying these two conditions (for the calculated equilibria see Appendix Table 1). Subsequently, we will calculate the entries of the Jacobian matrix of the linearized predator-prey system and evaluate them at the interior equilibrium in order to describe the local behaviour of the predator-prey system closely around its equilibrium state and to determine its local stability. The stability of the interior equilibrium depends on the properties of the Jacobian matrix:

$$ J={\left.\left(\begin{array}{cc}\frac{\partial \left(\frac{dR}{dt}\right)}{\partial R}& \frac{\partial \left(\frac{dR}{dt}\right)}{\partial C}\\ {}\frac{\partial \left(\frac{dC}{dt}\right)}{\partial R}& \frac{\partial \left(\frac{dC}{dt}\right)}{\partial C}\end{array}\right)\right|}_{R={R}^{\ast },C={C}^{\ast }} $$

The predator-prey model defined by Eqs. 9a, 9b and 9c (main text) does not account for interference or cooperation among individual predators or any other kind of direct density-dependence in C. Consequently, the lower right entry of J is 0, i.e. ${\left.\partial \dot{C}/\partial C\right|}_{R={R}^{\ast },C={C}^{\ast }}=0$. Furthermore, an increase in the density of the prey has always a positive effect on the predator density due to its type II functional response, which is a monotonically increasing function in R. Hence, the lower left entry of J is strictly positive, i.e. ${\left.\partial \dot{C}/\partial R\right|}_{R={R}^{\ast },C={C}^{\ast }}>0$. In contrast, an increase in the predator density will always have a negative effect on the prey density through grazing and nutrient retention. Therefore, the upper right entry of J is strictly negative, i.e. ${\left.\partial \dot{R}/\partial C\right|}_{R={R}^{\ast },C={C}^{\ast }}<0$. Finally, the sign of the upper left entry of J, i.e. ${\left.\partial \dot{R}/\partial R\right|}_{R={R}^{\ast },C={C}^{\ast }}$, will depend on the actual parameter values, e.g. the death rate of the predator. Hence, the entries of J possess the following signs:

$$ J=\left(\begin{array}{cc}\mp & -\\ {}+& 0\end{array}\right) $$

According to the Routh-Hurwitz criterion for a two-dimensional system of ordinary differential equations, the interior equilibrium is stable when the determinant of the Jacobian matrix evaluated at the interior equilibrium (J) is positive and the corresponding trace is negative (Murray 2002). The determinant of the Jacobian matrix is given by the following:

$$ \mathrm{Det}(J)=\left(\frac{\partial \left(\frac{dR}{dt}\right)}{\partial R}\bullet \frac{\partial \left(\frac{dC}{dt}\right)}{\partial C}\right)-\left(\frac{\partial \left(\frac{dC}{dt}\right)}{\partial R}\bullet \frac{\partial \left(\frac{dR}{dt}\right)}{\partial C}\right) $$

In line with our reasoning above, the sign of Det(J) is as follows:

$$ \operatorname{sign}\left(\mathrm{Det}(J)\right)=\left(\mp \bullet 0\right)-\left(+\bullet -\right)=+ $$

Therefore, the determinant of the Jacobian will always be positive. Consequently, the stability of the interior equilibrium entirely depends on the trace of the Jacobian matrix, which is given by the following:

$$ Tr(J)=\frac{\partial \left(\frac{dR}{dt}\right)}{\partial R}+\frac{\partial \left(\frac{dC}{dt}\right)}{\partial C}=\mp +0=\mp $$

Since ${\left.\partial \dot{C}/\partial C\right|}_{R={R}^{\ast },C={C}^{\ast }}=0$, the trace of the Jacobian is given by the derivative of the rate of change of the prey with respect to its own density. Hence, we calculated ${\left.\partial \dot{R}/\partial R\right|}_{R={R}^{\ast },C={C}^{\ast }}$ for the different model parametrizations used in the main text and generally solved for the death rate at which the trace of the Jacobian is 0. At this particular point (d^H), a Hopf bifurcation occurs, separating two regions for which the system exhibits either a stable equilibrium (d > d^H) or a limit cycle (d < d^H) (cf. Fig. 4c, d). In general, the Hopf bifurcation point d^H has to be calculated numerically. However, in some cases, explicit expressions for ${d}_{C_2}^H$ can be obtained (for details, see Appendix Table 1).

For instance, in the absence of nutrient retention by the predators (η = 0) and in case of a linear resource uptake rate of the prey (h_R = 0) and batch culture conditions (δ = 0), the Hopf bifurcation point d^H of the classical predator-prey system (Rosenzweig and MacArthur 1963) with logistically growing prey in the vacancy of the predator is given by the following (see also Abrams and Holt 2002):

$$ {d}^H=\left( ah-1\right)/h\left( ah+1\right) $$

Since d > 0, a Hopf bifurcation only occurs when ah > 1, so that d^H is also positive.

Furthermore, nutrient retention by the predators generally reduces the Hopf bifurcation point of the corresponding C − R system and thus stabilizes its population dynamics. This can be seen when choosing a particular value for the nutrient to carbon ratio η of the predator, e.g. η = 1. In this case, the Hopf bifurcation point d^H can be written as follows:

The stabilizing effect of nutrient retention on the predator-prey dynamics is stronger, when the non-linearity of the type II functional response is weaker. In contrast to the stabilizing effect of nutrient retention by the predators, a non-linear resource uptake rate of the prey has a destabilizing effect on the predator-prey dynamics. When accounting for non-linearity in the preys’ resource uptake rate (h_R > 0), the Hopf bifurcation point of the corresponding predator-prey system is given as follows:

Hence, d^H increases with an increase in h_R. The impact of h_R on the overall stability of the predator-prey dynamics depends on the curvature of the type II functional response of the predator. The larger the product ah is, the larger the factor ${\left(1+{h}_R\left(1+\frac{1}{ah}\right)\right)}^{-\frac{1}{2}}$ becomes, and thus, the smaller the term $1-{\left(1+{h}_R\left(1+\frac{1}{ah}\right)\right)}^{-\frac{1}{2}}$ will be. Hence, the destabilizing effect of h_R on the predator-prey dynamics is larger when the non-linearity of the type II functional response of the predator is weaker.

Finally, a common death rate of all species through the dilution rate δ is stabilizing the predator-prey dynamics. The Hopf bifurcation point of the corresponding C − R system, i.e. when δ > 0, is given by the following:

Hence, d^H decreases when δ is increasing. The stabilizing effect of δ on the predator-prey dynamics is larger when the non-linearity of the type II functional response of the predator is stronger and, thus, when the handling time h is higher. This happens, because the dilution rate does affect not only the stability of the predator-prey dynamics but also the persistence of the corresponding C − R system.

Quantifying the size of the parameter range of potential species coexistence

The necessary conditions N₂, N₃ and N₄ of predator coexistence derived above determine a range of parameters that may permit the coexistence of the two different predators (Fig. 6a). In order to quantify the size of this parameter region, we first calculated three different threshold values of the death rate of C₁, ${d}_{C_1}^{T_i}$, that allows piecewise integration of the relevant functions. The first threshold value ${d}_{C_1}^{T_1}$ is given by the death rate ${d}_{C_1}$ for which the function ${N}_2\left({d}_{C_1}\right)=\frac{a_{C_2}\left({d}_{C_1}+\delta \right)}{a_{C_1}+{h}_{C_2}{a}_{C_2}\left({d}_{C_1}+\delta \right)}-\delta$ (reflecting the second necessary condition) intersects with the Hopf bifurcation point ${d}_{C_2}^H$ (denoting the fourth necessary condition), i.e. ${N}_2\left({d}_{C_1}^{T_1}\right)={d}_{C_2}^H$ (cf. Fig. 6b). Solving this equation with respect to ${d}_{C_1}^{T_1}$ leads to the following expression of the first threshold value:

$$ {d}_{C_1}^{T_1}=\frac{\left({d}_{C_2}^H+\delta \right){a}_{C_1}+\delta \left({d}_{C_2}^H+\delta \right){h}_{C_2}{a}_{C_2}-{a}_{C_2}\delta }{\left({a}_{C_2}-\left({d}_{C_2}^H+\delta \right){h}_{C_2}{a}_{C_2}\right)} $$

Accordingly, the second threshold value ${d}_{C_1}^{T_2}$ is given by the intersection point of the function ${N}_3\left({d}_{C_1}\right)=\frac{\left({d}_{C_1}+\delta \right){a}_{C_2}}{a_{C_1}\left(1+{h}_{C_2}{a}_{C_2}{R}_S^{\ast}\right)}-\delta$ (representing the third necessary condition) with the Hopf bifurcation point ${d}_{C_2}^H$, i.e. ${N}_3\left({d}_{C_1}^{T_2}\right)={d}_{C_2}^H$ (cf. Fig. 6c). Evaluating this equation with respect to ${d}_{C_1}^{T_2}$ results into the following expression of the second threshold value:

$$ {d}_{C_1}^{T_2}=\frac{a_{C_1}}{a_{C_2}}\left({d}_{C_2}^H+\delta \right)\left(1+{h}_{C_2}{a}_{C_2}{R}_S^{\ast}\right)-\delta . $$

Finally, the third threshold value ${d}_{C_1}^{T_3}$ is given by the value of ${d}_{C_1}$ at which the function ${N}_3\left({d}_{C_1}\right)$ equals 0, i.e. ${N}_2\left({d}_{C_1}^{T_3}\right)=0$ (cf. Fig. 6d). Solving the latter for ${d}_{C_1}^{T_3}$ results into the following expression of the third threshold value:

$$ {d}_{C_1}^{T_3}=\delta \left(\frac{a_{C_1}}{a_{C_2}}\left(1+{h}_{C_2}{a}_{C_2}{R}_S^{\ast}\right)-1\right) $$

Using these three different threshold values, the absolute size of the parameter region of potential predator coexistence is given by the following sum of integrals (cf. Fig. 6):

$$ {P}_a={\int}_0^{d_{C_1}^{T_1}}\frac{a_{C_2}\left({d}_{C_1}+\delta \right)}{a_{C_1}+{h}_{C_2}{a}_{C_2}\left({d}_{C_1}+\delta \right)}-\delta\ \boldsymbol{d}{d}_{C_1}+{d}_{C_2}^H\ \left({d}_{C_1}^{T_2}-{d}_{C_1}^{T_1}\right)-{\int}_{d_{C_1}^{T_3}}^{d_{C_1}^{T_2}}\frac{\left({d}_{C_1}+\delta \right){a}_{C_2}}{a_{C_1}\left(1+{h}_{C_2}{a}_{C_2}{R}_S^{\ast}\right)}-\delta \boldsymbol{d}{d}_{C_1} $$

After having solved the first integral with Wolfram Alpha, we obtain the following expression of P_a:

$$ {P}_a=\frac{a_{C_2}{h}_{C_2}{d}_{C_1}^{T_1}-{a}_{C_1}\ \mathit{\ln}\left(\frac{a_{C_2}{h}_{C_2}{d}_{C_1}^{T_1}}{a_{C_1}+{a}_{C_2}{h}_{C_2}\delta }+1\right)}{a_{C_2}{h_{C_2}}^2}-\delta {d}_{C_1}^{T_1}+{d}_{C_2}^H\left({d}_{C_1}^{T_2}-{d}_{C_1}^{T_1}\right)-\left(\frac{a_{C_2}\left(\frac{1}{2}{d}_{C_1}^{T_2}+\delta \right){d}_{C_1}^{T_2}}{a_{C_1}\left(1+{h}_{C_2}{a}_{C_2}{R}_S^{\ast}\right)}-\frac{a_{C_2}\left(\frac{1}{2}{d}_{C_1}^{T_3}+\delta \right){d}_{C_1}^{T_3}}{a_{C_1}\left(1+{h}_{C_2}{a}_{C_2}{R}_S^{\ast}\right)}\right)+\delta \left({d}_{C_1}^{T_2}-{d}_{C_1}^{T_3}\right). $$

Note, although it is not explicitly stated, P_a also depends on ${\eta}_{C_2}$ through the dependency of the Hopf bifurcation point ${d}_{C_2}^H$ on ${\eta}_{C_2}$. However, P_a does not depend on ${\eta}_{C_1}$. This strongly contrasts with the fact that the parameter range of realized species coexistence will likely also depend on ${\eta}_{C_1}$ and potential differences between ${\eta}_{C_1}$and ${\eta}_{C_2}$.

Table 1 Summary of the linear stability analysis of the predator-prey model defined by Eqs. 11a and 11b for the interior equilibrium under different model parameterizations, showing the influence of different ecological processes on the stability of the corresponding predator-prey dynamics. The colours highlight instances where the calculated terms of the particular expressions, i.e. equilibrium values, entrees of the Jacobian matrix or Hopf bifurcation point, differ from the ones of the reference scenario, i.e. the terms of the Rosenzweig-MacArthur predator-prey model (Rosenzweig-MacArthur 1963)

Fig. 6

Parameter range, i.e. death rates ${d}_{C_1}$ and ${d}_{C_2}$, that potentially enables coexistence of predator one, C₁, and predator two, C₂ (grey shaded area) (for details, see legend of Fig. 1). The parameter range of potential species coexistence marked in a is given by the sum of parameter ranges shown in b and c minus the parameter range shown in d. Striped areas in c and d indicate parameter regions where predator coexistence is impossible

Appendix 2. Fluctuations in the prey abundance act as a stabilizing mechanism of predator coexistence

Contemporary theory suggests that species coexistence depends on the balance between stabilizing niche differences and destabilizing fitness differences among species (Chesson 2000; Barabás et al. 2018). Niche differences enable species coexistence by increasing intra- relative to interspecific competition, thereby facilitating the regrowth of species from low densities (Chesson 2018). In contrast, fitness differences promote competitive exclusion of inferior species in the absence of niche differentiation (Letten et al. 2017). Hence, two or more predators cannot coexist on one prey in the absence of niche differences, because the presence of inevitable fitness differences among the different predators will drive all but one of them extinct (Chesson 2000). Stabilizing niche differences between two different predators may arise through relative non-linearity in their functional responses and associated fluctuations in the prey abundance. We therefore here derive corresponding terms reflecting average fitness differences and stabilizing niche differences in our predator-prey model defined by Eqs. 9a, 9b and 9c. Following Chesson (2018), average fitness differences between the different predators can be inferred from their invasion growth rates in the absence of stabilizing niche differences. Hence, we calculated the invasion growth rates, i.e. temporal averages of the per capita net growth rates, of both predators for the corresponding resident system that comprises the other predator and the prey at its long-term equilibrium, which is assumed to be stable. Hence, the invasion growth rate of predator one for the equilibrium density of the prey in the corresponding resident predator-prey system comprising predator two is given by the following:

Similarly, the invasion growth rate of predator two for the equilibrium density of the prey in the corresponding resident predator-prey system comprising predator one is given by the following:

Comparing the invasion growth rates of both predators reveals that the competitive outcome entirely depends on the difference between ${R}_{C_2}^{\ast }$ and ${R}_{C_1}^{\ast }$. Hence, the average fitness differences (FD) between the two predators are given by the difference between the prey requirements of predator two $\left({R}_{C_2}^{\ast}\right)$ and predator one $\left({R}_{C_1}^{\ast}\right)$, i.e. ${R}_{C_2}^{\ast }-{R}_{C_1}^{\ast }$.

According to the coexistence mechanism of relative non-linearity in the predator’s functional responses, stabilizing niche differences between the two different predators only arise through predator-prey oscillations. Given that the linear functional response of predator one ensures that the equilibrium of the corresponding predator-prey system is always stable, additional terms cannot arise in the invasion growth rate of predator two. However, the type II functional response of predator two may destabilize the equilibrium of the corresponding predator-prey system. Under such conditions, the invasion growth rate of predator one is given as follows:

$$ {I}_{C_1}={\left.\frac{d{C}_1}{C_1 dt}\right|}_{{\overline{R}}_{C_2}}={a}_{C_1}\left({\overline{R}}_{C_2}-{R}_{C_1}^{\ast}\right)={a}_{C_1}\left({\overline{R}}_{C_2}-{R}_{C_2}^{\ast }+{R}_{C_2}^{\ast }-{R}_{C_1}^{\ast }\ \right) $$

Comparing the invasion growth rate of predator one for the resident system of predator two in the absence of predator-prey oscillations, which prevents coexistence, with the one in the presence of predator-prey oscillations, which potentially allows coexistence, shows that the stabilizing niche differences (ND) between the two predators are given by the difference between ${\overline{R}}_{C_2}$ and ${R}_{C_2}^{\ast }$, i.e. ${\overline{R}}_{C_2}-{R}_{C_2}^{\ast }$. Hence, the net-stabilizing effect of relative non-linearity in the predators’ functional response on predator coexistence is thus given by the sum of stabilizing niche differences (+) and destabilizing fitness differences (−), i.e. $ND+ FD={\overline{R}}_{C_2}-{R}_{C_2}^{\ast }+{R}_{C_2}^{\ast }-{R}_{C_1}^{\ast }={\overline{R}}_{C_2}-{R}_{C_1}^{\ast }$.

Appendix 3. Fourier analysis of predator-prey oscillations

We found two different types of population dynamics in case of predator coexistence. For most parameter combinations, simple predator-prey oscillations emerged with the increase of the prey followed by an increase in the two predators (e.g. Fig. 5e). For some other parameter combinations, the competition between C₁ and C₂ for the shared prey generated much more complex predator-prey dynamics (e.g. Fig. 5f). To evaluate the differences in the population dynamics in more detail, we performed a discrete Fourier transformation by using the fft function implemented in MATLAB, version 7.13 (The MathWorks Inc., Natick, MA, 2011).

Our results show that for the simple predator-prey oscillations, most of the variation in the species’ abundances can be attributed to a single dominate time scale (Fig. 7a, c, e). Here, the two predators are strongly synchronized, exhibiting a negligible phase difference of 0.03 π (cf. Fig. 7b, d). In contrast, for the more complex population dynamics, our analysis reveals two dominant time scales (Fig. 8a, c, e), which can be most clearly seen from the amplitude spectrum of C₁ (Fig. 8c). At the low frequency, the population dynamics of the two predators exhibit a phase difference of 0.74 π (cf. Fig. 8b, d). Hence, their abundances cycle almost anti-synchronously, meaning that an increase in the abundance of C₁ coincides with a decrease in the abundance of C₂ and vice versa (Fig. 8d). In contrast to the prey (R) and C₁, C₂ showed hardly any variation in its abundance at the high frequency (Fig. 8f), although this shorter time scale in the population dynamics is generated by the predator-prey interaction between R and C₂. This discrepancy can be explained by the fact that the destabilization of the predator-prey oscillations by C₂ selects immediately for the recovery of C₁ (Fig. 8b).

Fig. 7

Frequency domain and time domain of the population dynamics showing simple predator-prey oscillations (cf. Fig. 4c). Amplitude spectrum (grey) of the time series of the prey (R), predator one (C₁) and predator two (C₂) are shown in a, c and e, respectively. The orange vertical line indicates the dominate time scale of the population dynamics. b The full time series of R (black solid line), C₁ (grey solid line) and C₂ (grey dashed-dotted line). The population dynamics at the dominant time scale are plotted in d based on the calculated discrete Fourier transform

Fig. 8

Frequency domain and time domain of the population dynamics showing complex predator-prey oscillations (cf. Fig. 4d). Amplitude spectrum (grey) of the time series of the prey (R), predator one (C₁) and predator two (C₂) are shown in a, c and e, respectively. The orange vertical lines indicate the two dominate time scales of the population dynamics. b The full time series of R (black solid line), C₁ (grey solid line) and C₂ (grey dashed-dotted line). The population dynamics at the longer and shorter time scales are plotted in d and f, respectively, based on the calculated discrete Fourier transform

Appendix 4. The opposing effects of nutrient retention and nutrient recycling on the stability of predator-prey dynamics

Our results presented in the main text and Appendix 1 demonstrate that the retention of nutrients in the predator biomass stabilizes predator-prey dynamics because it effectively reduces the carrying capacity of the prey. In contrast, previous studies have shown that the recycling of nutrients from dead organisms destabilizes predator-prey dynamics (e.g. DeAngelis 1992; Kooi et al. 2002). Hence, non-consumptive effects on the prey may enhance or reduce the stability of food webs, depending on whether the stabilizing effects of nutrient retention or the destabilizing effects of nutrient recycling prevail.

To clarify the relative importance of nutrient retention and nutrient recycling for system stability, we extended the predator-prey model used in the main text by two detrital components, thereby incorporating a time lag in the recycling of nutrients from dead predators and prey (cf. DeAngelis 1992). We track the recycling from dead predators and prey separately because they generally contain different amounts of nutrients per unit of biomass. For convenience, we consider here one predator species only. Hence, the amount of dissolved inorganic nutrients (N) and the biomasses of the dead predators (D_C) and prey (D_R) and the living predators (C) and prey (R) are changing over time according to the following set of equations:

$$ \frac{dN}{dt}=\underset{\mathrm{replenishment}}{\underbrace{\delta \left({N}_I-N\right)}}-\underset{\mathrm{uptake}}{\underbrace{\frac{\eta_R{a}_R NR}{1+{h}_R{a}_RN}}}+\underset{\mathrm{excretion}}{\underbrace{\eta_R\left(1-\frac{\eta_C}{\eta_R}{e}_C\right)\frac{a_C RC}{1+{h}_C{a}_CR}}}+\underset{\mathrm{recycling}}{\underbrace{\eta_R{m}_R{D}_R+{\eta}_C{m}_C{D}_C}} $$

(12a)

$$ \frac{d{D}_R}{dt}={d}_RR-{m}_R{D}_R-\delta {D}_R $$

(12b)

$$ \frac{d{D}_C}{dt}={d}_CC-{m}_C{D}_C-\delta {D}_C $$

(12c)

$$ \frac{dR}{dt}=\left[\frac{a_RN}{1+{h}_R{a}_RN}-\frac{a_CC}{1+{h}_C{a}_CR}-{d}_R-\delta \right]R $$

(12d)

$$ \frac{dC}{dt}=\left[\frac{e_C{a}_CR}{1+{h}_C{a}_CR}-{d}_C-\delta \right]C $$

(12e)

In contrast to Eqs. 1a, 1b, 1c and 1d of the main text, nutrient recycling does not occur immediately but rather passes through a detrital component. The time needed for the recycling of nutrients contained in the dead organisms into the inorganic nutrient pool scales with $\frac{1}{m_R}$ and $\frac{1}{m_C}$ for the prey and predators, respectively. A complete description of the other terms and parameters can be found in “Methods”.

The predator-prey system described by Eqs. 12a, 12b, 12c, 12d and 12e is mass balanced. Hence, the total amount of nutrients is given by N_T = N + η_R(R + D_R) + η_C(C + D_C). As a result, we can follow the general approach described in the methods and reduce the number of state variables of Eqs. 12a, 12b, 12c, 12d and 12e. The dimensionally reduced predator-prey system is then given as follows:

$$ \frac{d{D}_R}{dt}={d}_RR-{m}_R{D}_R-\delta {D}_R $$

(13a)

$$ \frac{d{D}_C}{dt}={d}_CC-{m}_C{D}_C-\delta {D}_C $$

(13b)

$$ \frac{dR}{dt}=\left[\frac{a_R\left({N}_I-\left({\eta}_R\left(R+{D}_R\right)+{\eta}_C\left(C+{D}_C\right)\right)\ \right)}{1+{h}_R{a}_R\left({N}_I-\left({\eta}_R\left(R+{D}_R\right)+{\eta}_C\left(C+{D}_C\right)\right)\ \right)}-\frac{a_CC}{1+{h}_C{a}_CR}-{d}_R-\delta \right]R $$

(13c)

$$ \frac{dC}{dt}=\left[\frac{e_C{a}_CR}{1+{h}_C{a}_CR}-{d}_C-\delta \right]C $$

(13d)

To further simply the analysis of our predator-prey system described by Eqs. 13a, 13b, 13c and 13d, we now consider a separation of time scales between the predator-prey dynamics (slow dynamics) and the recycling of nutrients from dead organisms (fast dynamics) (cf. Kooi et al. 1998b). Hence, we set $\frac{d{D}_R}{dt}$ and $\frac{d{D}_C}{dt}$ always equal to 0 and then substitute the resulting algebraic expression into Eq. 13c (cf. O’Dwyer 2018). This results in the following approximation of our predator-prey system:

$$ \frac{dR}{dt}=\left[\frac{a_R\left({N}_I-\left({\upeta}_R\left(1+\frac{d_R}{\left({m}_R+\delta \right)}\right)R+{\eta}_C\left(1+\frac{d_C}{\left({m}_C+\delta \right)}\right)C\right)\ \right)}{1+{h}_R{a}_R\left({N}_I-\left({\eta}_R\left(1+\frac{d_R}{\left({m}_R+\delta \right)}\right)R+{\eta}_C\left(1+\frac{d_C}{\left({m}_C+\delta \right)}\right)C\right)\ \right)}-\frac{a_CC}{1+{h}_C{a}_CR}-{d}_R-\delta \right]R $$

(14a)

$$ \frac{dC}{dt}=\left[\frac{e_C{a}_CR}{1+{h}_C{a}_CR}-{d}_C-\delta \right]C $$

(14b)

In the main text and Appendix 1, we showed that an increase in η_R and η_C has a stabilizing effect on the predator-prey dynamics. Consequently, an increase in m_R and m_C will have a destabilizing effect on the population dynamics of our predator-prey system described by Eqs. 14a and 14b because m_R and m_C scale inversely with η_R and η_C, respectively. Hence, our results confirm previous findings by e.g. DeAngelis (1992) and Kooi et al. (2002) that nutrient recycling has a destabilizing effect on the predator-prey dynamics (Fig. 9). This happens because nutrient recycling effectively enhances the carrying capacity of the prey (cf. Eqs. 14a). In fact, in the absence of nutrient recycling (endogenously driven nutrient replenishment), i.e. m_R = 0 and m_C = 0, and dilution (exogenously driven nutrient replenishment), i.e. δ = 0, the amount of nutrients in the living predator and prey biomasses will decrease over time because the nutrients accumulate in the detrital compartments.

Fig. 9

The predator (grey dashed-dotted line) and prey (black solid line) dynamics of the model defined by Eqs. 14a and 14b in dependence of the rate of nutrient recycling (m_C) and the nutrient to carbon ratio of the predators (η_C). The parameter m_C equals 0.1 in a and c and 0.8 in b and d, whereas the parameter η_C equals 0.25 in a and b and 1 in c and d. Other parameter values are as follows: d_R = 0, d_C = 0.15, a_R = 1, a_C = 2, h_R = 0, h_C = 2, e_C = 1, η_R = 0.25, δ = 0.05, and N_I = 1