1 Highlights

  • A fractional-order predator–prey system is proposed.

  • The local behavior of the solution is studied.

  • An efficient numerical scheme is used for investigating the solution of the fractional system.

  • Some graphical representations with their biological interpretations are provided.

  • The numerical method also works to solve other similar problems in mathematical biology.

2 Introduction

Mathematical modeling of the real-world phenomenon is a potent tool for predicting some ecological and biological components. The validity of this mathematical approximation depends on the model itself. The crucial component that describes the interaction between different species in a certain environment is the interaction functional. There are many types of these functionals in the literature. Each one describes a specific manner of intermingling between two species. More precisely, let us assume that \(x(t)\) and \(y(t)\) are the densities of the prey and predator populations at the time t, respectively. One of the first interaction functionals is the Holling I interaction functional \(\mathit{ax}(t)y(t)\), where a is the hunting (predation) rate of the prey population by a predator. This functional response is one of the most used tools in the literature. For instance, we refer to [27, 2938, 42, 43, 4951]. The issue with this interaction functional is the unboundedness of the consumption of the prey by a predator, which is inconsistent with the actual conditions in nature. To fix this defect, Holling [26] constructed the saturated functional response \(\frac{\mathit{ax}(t)y(t)}{1+\mathit{at}_{h} x(t)}\), where \(t_{h}\) is the average handling time of prey by a predator. This functional resolves the unboundedness of the prey consumption by a predator. There are many other functional responses in the literature. In each case a specific manner on the intermingling between two species has been assumed.

The following items are some related examples:

  • Holling III interaction functional \(\frac{a(x(t))^{2} y(t)}{1+\mathit{at}_{h} (x(t))^{2}}\) [30].

  • Generalized Holling III interaction functional \(\frac{a(x(t))^{2} y(t)}{1+b x(t)+ c (x(t))^{2}}\) [32].

  • Beddington–DeAngelis interaction functional \(\frac{\mathit{ax}(t) y(t)}{1+b x(t)+ c y(t)}\) [28].

  • Ratio-dependent interaction functional \(\frac{\mathit{ax}(t) y(t)}{ x(t)+ y(t)}\) [42, 55].

  • Hassel–Varley interaction functional \(\frac{\mathit{ax}(t) y(t)}{1+b x(t)+ c y(t)}\) [6].

Here all the parameters are assumed to be positive. The reason for this great diversity in functionals is due to the variety of environmental conditions in the problem. Some of the factors that influence the selection of these parameters are the behavior of the prey and predator, and the studied area. For the last factor, many components play a crucial role such as rivers (water availability), food (for the prey), and the density of prey and predator. Overall, the functional selection depends on many factors.

The predator–prey models with three species have been attracted many researchers. In the environment the intermingling is not limited to just two populations, but interactions can be defined between more than two species in one single place. The scientists interested in this point of view have put efforts to model such complex interactions in the last few decades. We can take as an example two types of prey and one predator [15], where the predator has the capability of hunting both prey populations. Moreover, in prey–predator–superpredator models [35] the predator feeds the prey only, and the superpredator feeds both prey and predator. In some models, we study the interaction between two predators and one prey model where two types of predators are fed the same prey. Due to the intrinsic nature of the predators, there will always be a constant struggle to capture this one prey. In real situations, it is seen that one predator determines its own hunting territory. The presence of other predators in such territories is entirely unacceptable. This situation is called competition. The models in which competition is found have also received much attention in many research papers such as [2, 38]. In this paper, we are interested in studying the intermingling and competition between two competitive predators on one prey with a generalized class of interaction functionals in the presence of the time-fractional derivative. In [14], it is highlighted that the fractional-time derivative explains the memory effect of a dynamical system, where the order of the derivative is called the memory rate, and the kernel of the factional derivative is called the memory function. This point of view has been applied in many disciplines such as mathematics, engineering, signal proceeding, mechanics, and especially in mathematical biology [12, 2024, 3941, 44]. By summarizing all the previously mentioned components let us focus on the following time-fractional formulation with a generalized consumption functional:

$$ \textstyle\begin{cases} \frac{d^{\varepsilon }x}{\mathit{dt}^{\varepsilon }}=\mathit{rx}(1-\frac{x}{k})-f(x) y - g(x) z, \\ \frac{d^{\varepsilon }y}{\mathit{dt}^{\varepsilon }}=e_{1} f(x) y -\mu _{1} y- \beta y z, \\ \frac{d^{\varepsilon }z}{\mathit{dt}^{\varepsilon }}=e_{2} g(x)z - \mu _{2} z - \gamma y z,\end{cases} $$
(1)

where \(\frac{d^{\delta }}{\mathit{dt}^{\varepsilon }}\) represents Caputo’s derivative in terms of time, which is defined by

$$ \frac{d^{\varepsilon }\varphi (t)}{\mathit{dt}^{\varepsilon }} = \frac{1}{\Gamma (m-\varepsilon )}\displaystyle \int _{0}^{t}\frac{\varphi ^{m}(s)}{(t-s)^{\varepsilon +s-i}} ds,\quad m-1< \varepsilon < m, m \in \mathbb{N} . $$

The conditions on the functionals f and g are defined as

\((A_{1})\):

\(f(0)=0\), \(g(0)=0\),

\((A_{2})\):

\(f'(x)>0\), \(g'(x)>0\) for \(x>0\).

In model (1), \(x(t)\), \(y(t)\), and \(z(t)\) are the densities of prey, first predator, and other predator populations at time t, respectively. We assume that the prey population reproduces logistically with the increasing rate r and the carrying capacity of the space k, \(e_{1}\) and \(e_{2}\) are respectively the conversion rate of the prey biomass into the first predator population and the diversion of the prey biomass into the second predator biomass, \(\mu _{1}\) and \(\mu _{2}\) are the mortality rates of the first and second predators, respectively, β (resp., γ) is the competition rate of the first predator with the second one (resp., of the second predator with the first one). The functionals f and g are respectively the interaction functionals for the first and second predator populations with the prey population. In the literature, there are a few papers that deal with a generalization of an interaction functional in a three-species model; we refer, for instance, to [29, 37, 45, 4951], which give an additional motivation to our research. Furthermore, it is been also applied in understanding some epidemiological interactions; we refer, for example, to [3, 4, 16, 17, 31, 36, 4648]. In nature the interaction between animals is affected by many factors, such as the weather, animal nature, environmental structure, natural resources (water, food), which can affect the interaction between the three studied populations. Hence it is wise to consider a wide class of interaction functionals, which provides a wide choice of applications of the obtained results in predicting the evolution of the species. For more reading about some recent methods of modeling ecological interactions, we refer the readers to [52, 53]. The applicability of model (1) is an additional motivation for us to present this paper. There are many functionals compatible with conditions \((A_{1})\) and \((A_{2})\), including Holling I–III interaction functional. Also, numerous other functionals fit with these functionals (we consider only the functional f, and the same can be assumed for the functional g):

  • \(f(x)=a\sqrt{x}\) [1],

  • \(f(x)=\frac{a\sqrt{x}}{1+\mathit{at}_{h} \sqrt{x}}\) [5],

  • \(f(x)=\mathit{ax}^{\alpha }\) (\(0<\alpha <1\)) [54],

  • \(f(x)=\frac{\mathit{ax}}{1+b\sqrt{x}+\mathit{cx}}\) [8],

  • \(f(x)=\frac{\mathit{ax}^{\alpha }}{1+\mathit{at}_{h} x^{\alpha }}\) [9].

This demonstrates the broad class of interaction functionality that we can consider in this study. For further details, e see [7, 10, 11, 13]. Based on the above-mentioned mathematical and biological backgrounds, the paper is structured as follows:

  • In Sect. 2, we offer some tools that will be useful in dealing with the fractional operator.

  • Sect. 3 is devoted to the mathematical investigation of model (1), where the asymptotic behavior of the solutions is investigated.

  • Sect. 4 offers a numerical scheme for the fractional system (1). The graphical representations of the solution for different parameters are also involved.

  • The concluding section of the paper is intended to highlight the biological meanings of the acquired numerical results.

3 Mathematical analysis and asymptotic behavior of the solution

3.1 Equilibria of the model

In this subsection, we determine the local behavior of system (1). First, we determine the equilibria of system (1), which are the solutions of the following system:

$$ \begin{gathered} 0=\mathit{rx}\biggl(1-\frac{x}{k} \biggr)-f(x) y - g(x) z, \\ 0=e_{1} f(x) y -\mu _{1} y-\beta y z, \\ 0=e_{2} g(x)z - \mu _{2} z -\gamma y z.\end{gathered} $$
(2)

As a first remark, we deduce that system (2) has the following particular cases:

  1. (i)

    \(O=(0,0,0)\), which represents the extinction of the three populations.

  2. (ii)

    \(\Gamma _{0}=(k,0,0)\), which implies the extinction of two types of predators. The point is called the predator-free equilibrium (PFE).

  3. (iii)

    Searching for the first predator-free equilibrium (FPFE) as \(\Gamma _{1}=(x_{1} ,0,z_{1} )\), we insert \(y=0\). By replacing this result in the third equation of system (2) we get \(x_{1}=g^{-1} (\frac{\mu _{2}}{e_{2}} )\), where \(g^{-1}\) is the reciprocal function of g, which exists since g is a bijective function. Substituting this last result into the first equation of (2) yields

    $$ z_{1}=\frac{\mathit{rx}_{1} e_{2}}{\mu _{2}} \biggl(1-\frac{x_{1}}{k} \biggr), $$

    which is positive if \(x_{1}< k\). Summarizing all the results, we can conclude that FPFE \(\Gamma _{1}=(x_{1},0,z_{1})\) exists if \(x_{1}< k\).

  4. (iv)

    Seeking for the second predator-free equilibrium (SPFE) \(\Gamma _{1}=(x_{2}, y_{2} ,0)\) by replacing \(z=0\) in (2). By substituting this result into the second equation of system (2) we get \(x_{2}=f^{-1} (\frac{\mu _{1}}{e_{1}} )\), where \(f^{-1}\) is the inverse function of f, which exists since f is a bijective function. Taking this last result along with the first equation of (2), we get

    $$ y_{2}=\frac{\mathit{rx}_{2} e_{1}}{\mu _{1}} \biggl(1-\frac{x_{2}}{k} \biggr), $$

    which is biologically relevant if \(x_{2}< k\). Summarizing all the results, we can deduce that SPFE \(\Gamma _{2}=(x_{2},y_{2},0)\) exists if \(x_{2}< k\).

Remark 1

It is assumed that both functional f and g are increasing in x. From \(x_{2}\) and \(x_{1}\), if \(\lim_{x \rightarrow +\infty } f(x)=a\) (resp., \(\lim_{x \rightarrow +\infty } g(x)=b\)), then another condition on the parameters arises, \(\frac{\mu _{1}}{e_{1}}< a\) (resp., \(\frac{\mu _{2}}{e_{2}}< b\)), which is a necessary condition for having a solution for the equation \(f(x)=\frac{\mu _{1}}{e_{1}}\) (resp., \(g(x)=\frac{\mu _{2}}{e_{2}}\)). This feature can be seen in the Holling II–III interaction functional.

  1. (v)

    Now we are in a position to seek the coexistence equilibrium, which is the positive solution of the following system:

    $$ \begin{gathered} 0=\mathit{rx}\biggl(1-\frac{x}{k} \biggr)-f(x) y - g(x) z, \\ 0=e_{1} f(x) -\mu _{1} -\beta z, \\ 0=e_{2} g(x) - \mu _{2} -\gamma y .\end{gathered} $$
    (3)

    From \(e_{2} g(x) - \mu _{2} -\gamma y =0\) we obtain

    $$ y=\frac{e_{2}}{\gamma }g(x)-\frac{\mu _{2}}{\gamma }. $$
    (4)

    Moreover, from \(e_{1} f(x) -\mu _{1} -\beta z=0\) we find that

    $$ z=\frac{e_{1}}{\beta }f(x)-\frac{\mu _{1}}{\beta }. $$
    (5)

    Substituting (4) and (5) into the first equation of (3), we get \(F_{1} (x)=F_{2} (x)\), where

    $$ F_{1} (x)= \mathit{rx}\biggl(1-\frac{x}{k}\biggr),\qquad F_{2} (x)= f(x) g(x) \biggl( \frac{e_{2}}{\gamma }+ \frac{e_{1}}{\beta } \biggr)- \biggl( \frac{\mu _{2}}{\gamma }f(x)+ \frac{\mu _{1}}{\beta }g(x) \biggr). $$

    Some straightforward calculations suggest that

    $$ F_{1} (0)=F_{1} (k)=0, \qquad F_{1} (x)= \textstyle\begin{cases} >0 &\text{for } x< \frac{k}{2}, \\ < 0& \text{for } x> \frac{k}{2}. \end{cases} $$

    To guarantee at least one nontrivial intersection between two curves of the functionals \(F_{1}\) and \(F_{2}\), we introduce the following assumption:

    $$ F_{1} (\tilde{x})>F_{2} (\tilde{x}), \qquad F_{2} (k)>0\quad \text{with } \tilde{x}=\max \{x_{1},x_{2} \}, $$

    which it can be rewritten as

    $$\begin{aligned} (H_{1}): \quad &\tilde{x}< k,~e_{2} >e_{2} ^{*} := \frac{f(k) g(k) (\frac{e_{2}}{\gamma }+ \frac{e_{1}}{\beta } )- ( \frac{\mu _{2}}{\gamma }f(k)+\frac{\mu _{1}}{\beta }g(k) )}{\beta f(k)g(k)}-e_{1} \gamma , \\ &r>r_{E}:= \frac{k [f(\tilde{x}) g(\tilde{x}) (\frac{e_{2}}{\gamma }+ \frac{e_{1}}{\beta } )- ( \frac{\mu _{2}}{\gamma }f(\tilde{x})+\frac{\mu _{1}}{\beta }g(\tilde{x}) ) ]}{\tilde{x}(k-\tilde{x})}. \end{aligned}$$

    Under condition \((H_{1})\), we get the existence of at least one nonnegative solution of system (3).

Remark 2

  1. (i)

    If \(\tilde{x}>k\), then system (3) has no solution.

  2. (ii)

    The condition \(F_{1} (\tilde{x})>F_{2} (\tilde{x})\) came from the fact that \(F_{1} '(0)=- ( \frac{\mu _{2}}{\gamma }f(0)+ \frac{\mu _{1}}{\beta }g(0) )<0\) (thus we can have \(F_{1} '(\upsilon )<0\) for υ sufficiently small).

3.2 Asymptotic behavior of (1)

In this part, we are interested in determining the asymptotic stability of the equilibria obtained in the previous section.

Remark 3

Consider the following fractional-order system:

$$ \frac{d^{\varepsilon }U}{\mathit{dt}^{\varepsilon }}=\Lambda U(t)+\varphi (U),\quad U(0)=U_{0} \in \mathbb{R}^{i}, $$
(6)

where \(\varepsilon \in (0,1)\), \(A\in \mathbb{R}^{i\times i}\), and \(\varphi \in C^{1}(\mathbb{R}^{i},\mathbb{R}^{i})\) with \(D\varphi (0)=0\). For the time-fractional-order derivative, the concept of the local stability is very different from the first-order derivative, where in this case, we have an expansion of the stability region in comparison with the first-order derivative.

Let \((x,y,z)\) be an equilibrium for system (1). The Jacobian matrix of system (1) at \((x,y,z)\) is expressed as

$$ J(x,y,z)= \begin{pmatrix} \mathit{rx}(1-\frac{2x}{k})-f'(x)y-g'(x)z & -f(x) & -g(x) \\ e_{1} f'(x)y & e_{1} f(x)-\mu _{1} -\beta z & -\beta y \\ e_{2} g'(x) z & -\gamma z & e_{2} g(x) -\mu _{2} -\gamma y \end{pmatrix} , $$
(7)

At the predator-free equilibrium the Jacobian matrix (7) reduces into

$$ J(k,0,0)= \begin{pmatrix} -r & -f(k) & -g(k) \\ 0 & e_{1} f(k)-\mu _{1} & 0 \\ 0 & 0 & e_{2} g(k) -\mu _{2} \end{pmatrix} . $$
(8)

The Jacobian matrix (8) has the eigenvalues \(\vartheta _{1}=-r<0\), \(\vartheta _{2}=e_{1} f(k)-\mu _{1}\), and \(\vartheta _{3}=e_{2} g(k) -\mu _{2} \). Hence

$$ \vartheta _{2}= \textstyle\begin{cases} < 0 &\text{for } k< x_{2} , \\ >0 &\text{for } k>x_{2} , \end{cases} $$

and

$$ \vartheta _{3}= \textstyle\begin{cases} < 0&\text{for }k< x_{1} , \\ >0&\text{for }k>x_{1} , \end{cases} $$

which means that the eigenvalues \(\vartheta _{i}\), \(i=1,2,3\), satisfy \(|\arg (\vartheta _{i})|>\frac{\varepsilon \pi }{2}\) if and only if \(x<\bar{x}:=\min \{x_{1} ,x_{2}\}\); in this case the predator-free equilibrium is locally stable, and it is unstable for \(x>\bar{x}\). We summarize the obtained results in the following lemma.

Lemma 1

The predator-free equilibrium is locally stable if \(x<\bar{x}\) and unstable if \(x>\bar{x}\).

Now we analyze the linear stability of FPFE point of \(\Gamma _{1}\). The Jacobian matrix corresponding to the equilibrium FPFE is evaluated as

$$ J(x,y,z)= \begin{pmatrix} \mathit{rx}_{1} (1-\frac{2x_{1} }{k})-g'(x_{1} )z_{1} & -f(x_{1} ) & -g(x_{1} ) \\ 0 & e_{1} f(x_{1} )-\mu _{1} -\beta z_{1} & 0 \\ e_{2} g'(x_{1} ) z_{1} & -\gamma z_{1} & 0 \end{pmatrix} . $$
(9)

As a first look, we can deduce that \(\vartheta _{2}= e_{1} f(x_{1} )-\mu _{1} -\beta z_{1}\) is an eigenvalue of the Jacobian matrix (9). By replacing the explicit formula of \(z_{1}\) we obtain \(\vartheta _{2}= e_{1} f(x_{1} )-\mu _{1} - \frac{\beta \mathit{rx}_{1} e_{2}}{\mu _{2}} (1-\frac{x_{1}}{k} )\). Obviously, if \(e_{1} f(x_{1} )-\mu _{1}<0\) (equivalent to \(x_{1} < x_{2}\)), then \(|\arg (\vartheta _{2})|>\frac{\varepsilon \pi }{2}\). Now we presume that if \(e_{1} f(x_{1} )-\mu _{1}>0\) (equivalent to \(x_{1} >x_{2}\)), then

$$ \vartheta _{2}= \textstyle\begin{cases} >0 &\text{for } r< r_{1}:= \frac{k\mu _{2}( e_{1} f(x_{1} )-\mu _{1} )}{\beta x_{1} e_{2} (k-x_{1})}, \\ < 0 &\text{for } r>r_{1}. \end{cases} $$

Under the condition \(\vartheta _{2}>0\), we get \(|\arg (\vartheta _{2})|<\frac{\varepsilon \pi }{2}\). This means that FPFE is an unstable equilibrium point. Besides, from \(\vartheta _{2}>0\) we conclude that \(|\arg (\vartheta _{2})|<\frac{\varepsilon \pi }{2}\). This means that two remaining eigenvalues of the Jacobian matrix (9) determine the stability (resp., instability) of this equilibrium. Note that these significant eigenvalues are the eigenvalues of the matrix

$$ \tilde{J}= \begin{pmatrix} \mathit{rx}_{1} (1-\frac{2x_{1} }{k})-g'(x_{1} )z_{1} & -g(x_{1} ) \\ e_{2} g'(x_{1} ) z_{1} & 0 \end{pmatrix} . $$
(10)

To determine the nature of the eigenvalues of the reduced matrix (10), we define the characteristic equation of (10) as

$$ \vartheta ^{2}-\tilde{\Lambda }_{1}\vartheta +\tilde{ \Lambda }_{2}=0, $$
(11)

where

$$\begin{aligned} \tilde{\Lambda }_{1} =&\mathit{rx}_{2} \biggl(1-\frac{2x_{2} }{k}\biggr)-f'(x_{2} )y_{2} \\ =&r \biggl[ -(1+A)+\frac{x_{1}}{k} A \biggr], \\ \Lambda _{2} =&e_{2} g'(x_{1} ) g(x_{1} )z_{1}>0 \\ =&\mathit{re}_{2} g'(x_{1})x_{1} \biggl(1-\frac{x_{1} }{k}\biggr)>0, \end{aligned}$$

where \(A=-2+\frac{g'(x_{1})x_{1} e_{2}}{\mu _{2}}\). To determine the sign of \(\Lambda _{1}\), the following results arise.

Lemma 2

Assuming \(x_{1} < k\), we obtain the following results:

  1. (i)

    \(\Lambda _{1}\leq 0\) if (\(-1\leq A\leq 0\)) or (\(A>0\) and \(k\leq k_{1} :=\frac{Ax_{1}}{1+A}\)) or (\(A<-1\) and \(k\leq k_{1} \)).

  2. (ii)

    \(\Lambda _{1} >0\) if (\(A>0\) and \(k< k_{1}\)) or (\(A<-1\) and \(k> k_{1} \)).

Proof

The proof can be easily deduced from the explicit formula of Λ. □

From Lemma 2, we can see that \(\Lambda _{1} \leq 0\) next to \(\Lambda _{2}>0\) gives \(|\arg \{\vartheta _{1,2}\}|>\frac{\varepsilon \pi }{2}\), which means that this equilibrium is stable. Now assume that \(\Lambda _{1} >0\) (which means that the second assumption of Lemma 2 holds). In this situation, (11) admits two complex roots of \(\vartheta _{\pm }=a_{1} \pm a_{2} i\), \(a_{1}, a_{2} \in \mathbb{R}\). Then these roots satisfy

$$ \tan ^{2} \bigl( \arg \{ \vartheta _{i} \} \bigr) = \frac{4a_{2}}{a_{1} ^{2}}-1= \frac{e_{2} g'(x_{1})x_{1} (1-\frac{x_{1} }{k})}{r (-(1+A)+\frac{x_{1}}{k} A )^{2}}-1,\quad i=1,2. $$

To guarantee the stability of the equilibrium, we must have \(\tan ^{2} ( \arg \{ \vartheta _{i} \} ) > \tan ^{2} ( \frac{\varepsilon \pi }{2} ) \), \(i=1,2\), which implies that

$$ r< r^{*} := \frac{4g'(x_{1}) x_{1} e_{2} (k-x_{1} )}{k (1+\tan ^{2} ( \frac{\varepsilon \pi }{2} ) ) (-(1+A)+\frac{x_{1}}{k} A )^{2}}. $$

Obviously, it is unstable for \(r>r^{*}\). Hence we resume the stability conditions for the equilibrium \((x_{1},0,z_{1})\) by the following theorem.

Theorem 1

If \(x_{1}< k\), then we have:

  1. (i)

    If \(x_{1} > x_{2}\) and \(r< r_{1}\), then the FPFE is unstable.

  2. (ii)

    For (\(x_{1} < x_{2}\) or (\(x_{1} > x_{2}\) and \(r>r_{1}\))), under condition (i) in Lemma 2, we get the local stability of FPFE.

  3. (iii)

    (\(x_{1} < x_{2}\) or (\(x_{1} > x_{2}\) and \(r>r_{1}\))), condition (ii) in Lemma 2, and \(r< r^{*}\) give the local stability of FPFE.

  4. (vi)

    (\(x_{1} < x_{2}\) or (\(x_{1} > x_{2}\) and \(r>r_{1}\))), condition (ii) in Lemma 2, and \(r>r^{*}\) imply that FPFE is unstable.

To study the stability of the SPFE of \(\Gamma _{2}\), we construct the Jacobian matrix

$$ J(x_{2} ,y_{2} ,0)= \begin{pmatrix} \mathit{rx}_{2} (1-\frac{2x_{2} }{k})-f'(x_{2} )y_{2} & -f(x_{2} ) & -g(x_{2} ) \\ e_{1} f' (x_{2}) y_{2} & 0 & -\beta y_{2} \\ 0 & 0 & e_{2} g(x_{2}) -\mu _{2} -\gamma y_{2} \end{pmatrix}. $$
(12)

Obviously, \(\vartheta _{3}= e_{2} g(x) -\mu _{2} -\gamma y_{2}\) is an eigenvalue of the Jacobian matrix (12). Replacing \(y_{2}=\frac{\mathit{rx}_{2} e_{1}}{\mu _{1}} (1-\frac{x_{2}}{k} )\) in \(\vartheta _{3}\), we obtain \(\vartheta _{3}= e_{2} g(x_{2}) -\mu _{2} - \frac{\gamma \mathit{rx}_{2} e_{1}}{\mu _{1}} (1-\frac{x_{2}}{k} )\). Obviously, if \(e_{2} g(x_{2}) -\mu _{2}<0\) (equivalent to \(x_{1} >x_{2}\)), then \(|\arg (\vartheta _{3})|>\frac{\varepsilon \pi }{2}\). Now we suppose that \(e_{2} g(x_{2}) -\mu _{2}>0\) (equivalent to \(x_{1} < x_{2}\)). Then

$$ \vartheta _{3}= \textstyle\begin{cases} >0 &\text{for } r< r_{2}:= \frac{k\mu _{1}( e_{2} g(x_{2} )-\mu _{2} )}{\gamma x_{2} e_{1} (k-x_{2})}, \\ < 0 &\text{for } r>r_{2}. \end{cases} $$

For \(\vartheta _{3}>0\), we obtain \(|\arg (\vartheta _{3})|<\frac{\varepsilon \pi }{2}\), which means that SPFE is unstable. In fact, for \(\vartheta _{3}>0\), we get \(|\arg (\vartheta _{3})|<\frac{\varepsilon \pi }{2}\). This means that the other two eigenvalues of the Jacobian matrix (12) determine the stability (resp., instability) of this equilibrium. These eigenvalues are the eigenvalues of the matrix

$$ \bar{J}= \begin{pmatrix} \mathit{rx}_{2} (1-\frac{2x_{2} }{k})-f'(x_{2} )y_{2} & -f(x_{2} ) \\ e_{1} f' (x_{2}) y_{2} & 0 \end{pmatrix} . $$
(13)

The nature of the eigenvalues of the reduced matrix (13) can be determined through the quadratic equation

$$ \vartheta ^{2}-\Xi _{1}\vartheta +\Xi _{2}=0, $$
(14)

where

$$\begin{aligned} \Xi _{1} =&\mathit{rx}_{1} \biggl(1- \frac{2x_{2} }{k}\biggr)-f'(x_{2} )y_{2} \\ =&r \biggl[ B-\frac{x_{k}}{k} (1+B) \biggr], \\ \Xi _{2} =&e_{1} f'(x_{2} ) f(x_{2} )y_{2}>0 \\ =&\mathit{re}_{1} f'(x_{2})x_{2} \biggl(1-\frac{x_{2} }{k}\biggr)>0, \end{aligned}$$

where \(B=1-\frac{f'(x_{2})x_{2} e_{2}}{\mu _{1}}\).

To prove the positivity of \(\Xi _{1}\), we use the following lemma.

Lemma 3

Let \(x_{2} < k\). Then:

  1. (i)

    \(\Xi _{1}\leq 0\) if (\(-1\leq B\leq 0\)) or (\(B>0\) and \(k\leq k_{2} :=\frac{1+B}{Bx_{1}}\)) or (\(B<-1\) and \(k\geq k_{1} \)).

  2. (ii)

    \(\Xi _{1} >0\) if (\(A>0\) and \(k> k_{2}\)) or (\(A<-1\) and \(k< k_{2} \)).

Proof

Form the explicit formula of Ξ we can determine its positivity. The proof is completed. □

From Lemma 2, we can see that \(\Xi _{1} \leq 0\) and \(\Xi _{2}>0\) lead to \(|\arg \{\vartheta _{1,2}\}|>\frac{\varepsilon \pi }{2}\), and hence the SPFE is stable. Now assume that \(\Lambda _{1} >0\) (which means that the second assumption of Lemma 3 holds). In this case, equation (14) has complex roots \(\vartheta _{\pm }=b_{1} \pm b_{2}\), \(b_{1} b_{2} \in \mathbb{R}\). Then we get

$$ \tan ^{2} \bigl( \arg \{ \vartheta _{i} \} \bigr) = \frac{4b_{2}}{b_{1} ^{2}}-1= \frac{e_{1} f'(x_{2})x_{2} (1-\frac{x_{2} }{k})}{r (B-\frac{x_{1}}{k} (1+B) )^{2}}-1,\quad i=1,2. $$

To ensure the stability of SPFE, we must have \(\tan ^{2} ( \arg \{ \vartheta _{i} \} ) > \tan ^{2} ( \frac{\varepsilon \pi }{2} ) \), \(i=1,2\), which implies that

$$ r< r^{**} := \frac{4f'(x_{2}) x_{2} e_{1} (k-x_{2} )}{k (1+\tan ^{2} ( \frac{\varepsilon \pi }{2} ) ) (B+\frac{x_{2}}{k} B )^{2}},$$
(15)

and it is unstable for \(r>r^{**}\). We summarize the obtained results in the following theorem.

Theorem 2

Let \(x_{2}< k\). Then:

  1. (i)

    If \(x_{1} < x_{2}\) and \(r< r_{2}\), then the SPFE is unstable.

  2. (ii)

    If ((\(x_{1}>x_{2}\)) or (\(x_{1} < x_{2}\) and \(r>r_{2}\))) and condition (i) in Theorem 1is satisfied, then the SPFE is stable.

  3. (iii)

    ((\(x_{1}>x_{2}\)) or (\(x_{1} < x_{2}\) and \(r>r_{2}\))), condition (ii) in Theorem 1, and \(r< r^{**}\) imply that the SPFE is stable.

  4. (vi)

    ((\(x_{1}>x_{2}\)) or (\(x_{1} < x_{2}\) and \(r>r_{2}\))), condition (ii) in Theorem 1, and \(r>r^{**}\) imply that the SPFE is unstable.

Now we are in a position to focus on studying the local behavior of the coexistence equilibrium. For this equilibrium, we suppose that assumption \((H_{1})\) holds. The Jacobian matrix at this equilibrium is expressed as

$$ J\bigl(x^{*} ,y^{*} ,z^{*} \bigr)= \begin{pmatrix} \mathit{rx}^{*} (1-\frac{2x^{*} }{k})-f'(x^{*} )y^{*} -g'(x^{*} )z^{*} & -f(x^{*} ) & -g(x^{*} ) \\ e_{1} f'(x^{*} )y^{*} & 0 & -\beta y^{*} \\ e_{2} g'(x^{*} ) z^{*} & -\gamma z^{*} & 0 \end{pmatrix} , $$
(16)

The characteristic equation associated with Jacobian (16) is

$$ \Delta =\vartheta ^{3}+\Phi _{2}\vartheta ^{2}+\Phi _{1}\vartheta + \Phi _{0}, $$

where

$$\begin{aligned} \Phi _{2} =& \mathit{rx}^{*} \biggl(1- \frac{2x^{*} }{k}\biggr)-f'\bigl(x^{*} \bigr)y^{*} -g'\bigl(x^{*} \bigr)z^{*}, \\ \Phi _{1} =& e_{1} f' \bigl(x^{*} \bigr) g'\bigl(x^{*} \bigr)z^{*} y^{*} + e_{2} g \bigl(x^{*} \bigr) g'\bigl(x^{*} \bigr) z^{*} - \beta \gamma z^{*} y^{*} , \\ \Phi _{0} =& -e_{2} \beta g' \bigl(x^{*} \bigr)f\bigl(x^{*} \bigr) y^{*} z^{*} + \gamma \beta y^{*} z^{*} \biggl[ \mathit{rx}^{*} \biggl(1-\frac{2x^{*} }{k}\biggr)-f' \bigl(x^{*} \bigr)y^{*} -g' \bigl(x^{*} \bigr)z^{*} \biggr]\\ &{}-e_{1} \gamma z^{*} y^{*} f' \bigl(x^{*} \bigr) g\bigl(x^{*} \bigr). \end{aligned}$$

Let

$$ D=18\Phi _{2}\Phi _{1}\Phi _{0}+ ( \Phi _{2}\Phi _{1} ) ^{2}-4 \Phi _{0} \Phi _{2} ^{3}-4 \Phi _{1} ^{3}-27 \Phi _{0} ^{3}. $$

Using the Routh–Hurwitz stability criterion for fractional calculus defined in [19], we get the stability conditions for the nontrivial equilibrium.

Theorem 3

The positive equilibrium is stable if one of the following assumptions holds:

  1. (i)

    \(D>0\), \(\Phi _{2}>0\), \(\Phi _{0}>0\), \(\Phi _{2}\Phi _{1}>\Phi _{0}\).

  2. (ii)

    \(D<0\), \(\Phi _{2}\geq 0\), \(\Phi _{1}\geq 0\), \(\Phi _{0}\geq 0\), and \(\varepsilon <\frac{2}{3}\).

4 Numerical analysis of system (1)

4.1 A numerical scheme

The main purpose of this section is to numerically solve the following fractal problem:

$$ \frac{d^{\delta }V}{\mathit{dt}^{\varepsilon }}= {P}\bigl(t, V(t)\bigr). $$
(17)

By applying the fundamental theorem of fractional calculus on (1) we get

$$ V(t) - V(0) = \frac{1}{\Gamma (\varepsilon )} \int _{0}^{t} {P}\bigl( \omega , V(\omega ) \bigr) (t-\omega )^{\varepsilon -1}\,d\omega . $$
(18)

Letting \(t=t_{n}=n\hslash \) in (18), we arrive at

$$ V(t_{n}) = V(0)+\frac{1 }{\Gamma (\varepsilon )}\sum _{i=0}^{n-1} \int _{t_{i}}^{t_{i+1}} {P}\bigl(\omega , V(\omega ) \bigr) (t_{n}-\omega )^{ \varepsilon -1}\,d\omega . $$
(19)

Now we can approximate the function \({P}(t, V(t))\) by the following linear approximation:

$$ {P}\bigl(t, U(t)\bigr)\approx {P}(t_{i+1}, V_{i+1} )+\frac{t-t_{i+1}}{\hslash } \bigl( {P}(t_{i+1}, V_{i+1} )\bigr)- {P}(t_{i}, V_{i} ) ) ,\quad t \in [t_{i},t_{i+1}], $$
(20)

with the notation \(V_{i}= V(t_{i})\).

By substituting Eq. (20) into (19) and applying some algebra (for more detail, see [18]) we get

$$ V_{n}= V_{0} +\hslash ^{\delta } \Biggl(\Phi _{n} {P}(t_{0}, V_{0} )+ \sum_{i=1}^{n} \Psi _{n-i} {P}(t_{i}, V_{i} ) \Biggr) $$
(21)

with

$$\begin{aligned}& \Phi _{n}= \frac{(n-1)^{\varepsilon +1}-n^{\varepsilon }(n-\varepsilon -1)}{\Gamma (\varepsilon +2)}, \\& \Psi _{n}=\textstyle\begin{cases} \frac{1}{{\Gamma (\varepsilon + 2)}},\quad n = 0, \\ \frac{{{{(n - 1)}^{\varepsilon }} - 2{n^{\varepsilon }} + {{(1+n)}^{\varepsilon }}}}{{\Gamma ( + 2)}}, \quad n = 1,2, \ldots . \end{cases}\displaystyle \end{aligned}$$

Using the numerical method presented in formula (21) to solve problem (1), we obtain the following iterative schemes:

$$\begin{aligned}& x_{n} = x_{0} +\hslash ^{\delta } \Biggl(\Phi _{n} {P}_{1}( x_{0},y_{0},z_{0})+ \sum_{i=1}^{n}\Psi _{n-i} {P}_{1}( x_{i},y_{i},z_{i}) \Biggr), \\& y_{n} = y_{ 0} +\hslash ^{\delta } \Biggl(\Phi _{n} {P}_{2}( x_{0},y_{0},z_{0})+ \sum_{i=1}^{n}\Psi _{n-i} {P}_{2}( x_{i},y_{i},z_{i}) \Biggr), \\& z_{n} = z_{0} +\hslash ^{\delta } \Biggl(\Phi _{n} {P}_{3}( x_{0},y_{0},z_{0})+ \sum_{i=1}^{n}\Psi _{n-i} {P}_{3}( x_{i},y_{i},z_{i}) \Biggr), \end{aligned}$$

where

$$\begin{aligned}& {P}_{1} ( x,y,z ) = \mathit{rx}\biggl(1-\frac{x}{k} \biggr)-f(x) y - g(x) z, \\& {P}_{2} (x,y,z ) =e_{1} f(x) y -\mu _{1} y-\beta y z, \\& {P}_{3} ( x,y,z )=e_{2} g(x)z - \mu _{2} z -\gamma y z. \end{aligned}$$

4.2 Applications and graphical representations

4.2.1 Example 1

In this section, we numerically investigate system (1) to ensure the obtained results in the previous sections. First, we choose the functional f to be a Holling II interaction functional \(f(x)=\frac{\mathit{ax}}{1+\mathit{at}_{h} x}\). Then \(f^{-1}(x)=\frac{x}{a(1-t_{h} x)}\), and based on Remark 3.1 for the existence of \(x_{2}\), we must have \(\frac{\mu _{1}}{e_{1}}<\frac{1}{t_{h}}\), and the functional g to be a Holling I interaction functional.

Figure 1 We consider the following parameters:

$$\begin{aligned}& r=0.5,\qquad k=10.5,\qquad a=1,\qquad b=0.5,\qquad e_{1}=0.5,\qquad e_{2}=0.5,\\& \mu _{1}=0.5,\qquad \mu _{2}=0.5,\qquad \beta =0.5, \\& \gamma =0.5,\qquad t_{h}=0.5,\qquad x(0)=0.04,\qquad y(0)=0.01,\qquad z(0)=0.01. \end{aligned}$$

For these values, we obtain the local stability of FPFE. Indeed, these values satisfy \(x_{1} =x_{2}\) and \(\Lambda _{1}= -1.309<0\) (condition (i) in Lemma 2), which means that condition (ii) in Theorem 2 holds.

Figure 1
figure 1

The local stability of the equilibrium \(\Gamma _{2} =(0.0497,0.0346,0)\)

Figure 2 We consider the following parameters:

$$\begin{aligned}& r=0.15,\qquad k=10.5,\qquad a=5,\qquad b=1,\qquad e_{1}=0.7,\qquad e_{2}=0.5,\\& \mu _{1}=0.15,\qquad \mu _{2}=0.15,\qquad \beta =0.35, \qquad \gamma =0.75,\qquad t_{h}=0.65, \qquad\varepsilon =0.9. \end{aligned}$$

For these values, we get the local stability of the SPFE \(\Gamma _{2} =(0.0497,0.0346,0)\). More precisely, these values satisfy \(r< r_{1} = 1.12\), which means that \(\Gamma _{1} =(0.3,0,0.145)\) is unstable and satisfy the second condition of Theorem 3 (\(x1>x2\) and \(\Psi _{1}=0.148>0\)), which means that the SPFE is stable.

Figure 2
figure 2

The local stability of the equilibrium \(\Gamma _{1}=(2.01,0,0.8)\)

Figure 3 We set the following values of parameters:

$$\begin{aligned}& r=0.15,\qquad k=10.5,\qquad a=5,\qquad b=1,\qquad e_{1}=0.7,\qquad e_{2}=0.5,\\& \mu _{1}=0.15,\qquad \mu _{2}=0.15,\qquad \beta =0.35, \qquad \gamma =0.75,\qquad t_{h}=0.65, \qquad\varepsilon =0.9. \end{aligned}$$
Figure 3
figure 3

The instability of the equilibrium \(\Gamma _{1}=(2.01,0,0.8)\)

Figure 4 We consider

$$\begin{aligned}& r=3.05,\qquad k=10.5,\qquad a=0.5,\qquad b=0.1,\qquad e_{1}=0.7,\qquad e_{2}=0.6,\\& \mu _{1}=0.15,\qquad \mu _{2}=0.15,\qquad \beta =0.35, \qquad \gamma =0.75,\qquad t_{h}=0.65, \end{aligned}$$

and multivalues of the memory rate. In this example, we realize the effect of the memory on the stability of SPFE. More precisely, this figure highlights that the memory has a huge influence on the condition of the stability obtained in (15), which shows a great agreement between the theoretical findings and numerical ones, and thus we conclude the efficiency of the obtained numerical method.

Figure 4
figure 4

The memory effect on the stability of the SPFE

4.2.2 Example 2

In this example, we consider another behavior of the prey population known by herd behavior (see [1, 34, 56]). In fact, we consider the functionals \(f(x)=a x^{\alpha }\) and \(g(x)=b x^{\alpha }\), where \(0<\alpha <1\) represents the herd shape rate. Thus \(f^{-1} (x)= (\frac{x}{a} )^{\frac{1}{\alpha }}\) and \(g^{-1} (x)= (\frac{x}{b} )^{\frac{1}{\alpha }}\). For the graphical representations, we take the following values of parameters.

Figure 5 In this figure, we consider the values

$$\begin{aligned}& r=3.05,\qquad k=10.5,\qquad a=0.5,\qquad b=0.1,\qquad e_{1}=0.7,\qquad e_{2}=0.6, \\& \mu _{1}=0.15,\qquad \mu _{2}=0.15,\qquad \beta =0.35, \qquad \gamma =0.75,\qquad \varepsilon =0.9,\qquad \alpha =0.95. \end{aligned}$$

These values provide the local stability of the SPFE.

Figure 5
figure 5

The local stability of the equilibrium \(\Gamma _{2} =(0.41,5.6,0)\)

Figure 6 We considered the same values as in Fig. 5, except \(\alpha =0.8\). This leads to the instability of the SPFE and the occurrence of oscillations of the solution.

Figure 6
figure 6

The local stability of the equilibrium \(\Gamma _{2} =(0.40,5.58,0)\)

Figure 7 In this figure, we have taken

$$\begin{aligned}& r=1.05,\qquad k=7.5,\qquad a=0.2,\qquad b=0.5,\qquad e_{1}=0.7,\qquad e_{2}=0.7, \\& \mu _{1}=0.15,\qquad \mu _{2}=0.15,\qquad \beta =0.75, \qquad \gamma =0.75,\qquad \varepsilon =0.9,\qquad \alpha =0.8. \end{aligned}$$

These values provide the instability of the FPFE.

Figure 7
figure 7

The local stability of the equilibrium FPFE

Figure 8 In this figure, we consider the following choices:

$$\begin{aligned}& r=1.5,\qquad k=10.5,\qquad a=1.5,\qquad b=0.5,\qquad e_{1}=0.5,\qquad e_{2}=0.5, \\& \mu _{1}=0.5,\qquad \mu _{2}=0.5,\qquad \beta =0.5, \qquad \gamma =0.5,\qquad \varepsilon =0.9,\qquad \alpha =0.7. \end{aligned}$$

These values provide the instability of the SPFE.

Figure 8
figure 8

The existence of oscillation generated by the instability of SPFE

Figure 9 We consider the values

$$\begin{aligned}& r=1.5,\qquad k=10.5,\qquad a=1.5,\qquad b=0.5,\qquad e_{1}=0.5,\qquad e_{2}=0.5, \\& \mu _{1}=0.5,\qquad \mu _{2}=0.5,\qquad \beta =0.5, \qquad \gamma =0.5,\qquad \alpha =0.7 \end{aligned}$$

For these values, we have studied the effect of the memory rate on the stability of the SPFE.

Figure 9
figure 9

The influence of the memory rate on the stability of the SPFE \(\Gamma _{2} =(0.56, 0.79,0)\)

Figure 10 The following values are used in numerical simulations

$$\begin{aligned}& r=5.5,\qquad k=3.5,\qquad a=0.5,\qquad b=1.5,\qquad e_{1}=0.5,\qquad e_{2}=0.5,\\& \mu _{1}=0.5,\qquad \mu _{2}=0.5,\qquad \beta =0.5, \qquad \gamma =0.5,\qquad \alpha =0.7. \end{aligned}$$

Using these values, we can investigate the effect of the memory rate on the stability of the FPFE.

Figure 10
figure 10

The influence of the memory rate on the stability of the SPFE \(\Gamma _{2} =(0.56, 0.79,0)\)

5 Conclusions

In this research, we studied an ecological model with two predators fighting on one prey with a generalized functional response. The reason behind considering a comprehensive generalized class of functional interaction is to model the diversity in predator–prey interaction with the environment. These interactions can be affected by many factors, such as the environment and the adaptation of the three species. In the first section, we studied the existence of the equilibria of system (1), where we can have many equilibrium points next to the predator-free equilibrium. By analyzing the existence of the equilibria we obtained that these populations may have many scenarios. They include the extinction of three populations, two types of predators, the extinction of each population of predators, and finally the coexistence of the three populations. For the coexistence stage, we provided some conditions on the model parameters for the existence of this equilibrium. To determine which scenario will prevail, we have utilized the local asymptotic stability using the Jacobian matrix. Further, we proposed a numerical scheme to find an approximate solution of the fractional-derivative system (1), which is based on the trapezoidal product-integration rule. For the application part, we have utilized two examples. The first example models the interaction of the three populations in the case of the solitary behavior for the prey population. We obtained an excellent agreement between the mathematical results and graphical representation, which proves the accuracy of the obtained approximations by proposed iterative schemes. In fact, in the case of the extinction of the second predator, two sceneries may happen. The first case is to stabilize on the SPFE as it has been highlighted in Fig. 1 for the prey solitary behavior (PSB) of the prey population. Moreover, Fig. 5 displays the case of the prey herd behavior (PHB). The second case corresponds to the instability of the SPFE and the presence of oscillations. The numerical simulations for this case have been plotted in Fig. 7, and also Fig. 8 for PHB. The same remark is obtained for the extinction of the first predator population, presented in Figs. 2, 3, 7, in different cases (the stability and instability of SPFE). Furthermore, the effect of the memory rate measured by the time-fractional derivative order ε is established numerically in Figs. 4, 9, 10. In these figures, we proved that the memory rate can destabilize the equilibrium states, which demonstrates the importance of considering it in the model construction, where in the real-world, the influence of the time-fractional-order derivative can represent the influence of these two populations on the hunting. Further, we can highlight that system (1) undergoes Hopf bifurcation at FPFE and SPFE at \(r=r^{*}\) for FPFE. Moreover, at \(r=r^{**}\) system undergoes Hopf bifurcation for FPFE. The oscillatory behaviors generated by these equilibria are presented in Figs. 3, 4, 6, 7, 8, 9. For more detail about the proof, we refer to the readers to [25]. Based on the obtained results in this contribution, the diversity of the ecological systems can be measured by the choice of the functional response.