1 Introduction

The study of the ecosystem using mathematical models has been one of the most crucial joint research fields between applied mathematicians, engineers, economists, ecologists, biologists, and epidemiologists for decades. It should be noted that differential equations have always been used in modeling and studying phenomena. The reason for this efficiency is that the future of systems is influenced by the past and present situations of understudied phenomena. In recent years there has been a strong tendency to use differential equations to model real-world problems. Mathematical biology is one of the most important and growing branches whose studies rely more on the use of differential equations than other fields [1–12]. One of the widespread applications of dynamic systems in ecology is to model the growth of monogamous and multicultural populations. The systems proposed in ecology determine the interaction of species with each other and with their environment, as well as the distribution and structure of populations. Such interactions may take place over a wider range of spatial and temporal scales. One of the most important types of interactions that affect the qualitative behavior of all species is hunting. For this reason, predator and prey models have been the focus of all ecological studies. In any ecosystem, different species of animals are usually in constant interaction with each other. In this case, the population dynamics of each species is affected by other species. In fact, there is a general set of interacting species called the food web [13–16]. One of the most important models in these prey and predator species is Lotka–Volterra systems [17]. In conventional models, in order to simplify the surveys, the interaction between a limited number of populations is usually examined. In any ecosystem, there are usually many species of animals in constant interaction with each other. In this case, the population dynamics of each species is affected by other species. The diversity of animals in the area in question always complicates the dynamic systems. Therefore, in conventional models, in order to simplify the analysis, the interaction between a limited number of populations is usually considered.

As we know, fractional calculus can be considered as an advanced generalization for classic differential calculus of integer order [18–31]. Nowadays, fractional differential calculus is one of the most widely used aspects of differential calculus, which is less used in practical aspects of real-life problems compared to the classical concepts. Perhaps the most important reason for this shortcoming can be considered the lack of objective background to describe practical phenomena, as well as the lack of tools and definitions needed in this area. However, this trend has accelerated in recent years [32–37]. And these days, its use in various branches of science, including mathematics, physics, and engineering has been highlighted [29, 38, 39]. One of the most important factors in the efficiency of such derivatives concerning derivatives is the correct equipping of these definitions with the concept of memory, which causes the basic information related to the phenomenon to be preserved and used during the study time. This fundamental advantage is very important when applying them to the structure of dynamic systems.

A review of the literature reveals that the dynamic models in biology are one of the most fundamental applications of fractional differential calculus with a wide range of potential theoretical and practical aspects. For instance, in [40] the authors have employed the Caputo fractional derivative to propose a new fractional-order predator–prey model with group defense. In [41], the authors have proposed an efficient numerical method to handle a predator–prey model with Beddington–DeAngelis functional response and fractional derivatives with Mittag-Leffler kernel. In [42] one of the Kolmogorov model variants called a Gauss-type predator–prey model involving the Allee effect and Holling type-III functional response in a fractional perspective has been studied. The authors of [43] have utilized a novel discretization based on the numerical discretization of the Riemann–Liouville integral to manipulate a fractional predator–prey model with the harvesting rate. To see more models, please refer to [44–49].

These research activities give us the idea of using the tools in fractional differential calculus in computational biological models. In this paper, we aim to incorporate the Atangana–Baleanu fractional derivative [50] in the structure of a prey and predator model. One of the key reasons for this choice is that this fractional order derivative has the ability to store and use the necessary system information from the beginning to the desired time [29, 38, 39]. This useful feature can play a key role in the study of dynamic systems related to computational biology. The paper is structured as follows. Some mathematical background, mainly about recent definitions on fractional calculus model, is formulated in Sect. 2. The proposed system is analyzed in Sect. 3. In Sect. 4, we investigate some mathematical analysis aspects of the model, including the calculation of equilibrium points of the system, the existence and uniqueness of the solution for the model different equilibrium points, and their stability. Then, we present the corresponding numerical simulations in Sect. 5. Finally, some concluding remarks are stated in the last section of the paper.

2 Mathematical backgrounds

In this section, we have an overview of some definitions and basic features of fractional differential calculus.

Definition 1

The Caputo (C) fractional operators of order ν for \(n\in \mathbb{N}\) are respectively defined as [19]

$$\begin{aligned}& \mathscr{D}_{\mathsf{C}} ^{{\nu }}{{p(t) }} = \frac{1}{\Gamma (n- {\nu })} \int _{0}^{t} (t-{\theta })^{m- {\nu }-1} {p}^{(k)}({ \theta })\,d{\theta }, \quad n-1< {\nu } \le n, \end{aligned}$$
(1)
$$\begin{aligned}& \mathscr{I}_{\mathsf{C}} ^{{\nu }}{{p(t) }} = \frac{1}{\Gamma ( {\nu })} \int _{0}^{t} (t-{\theta })^{ {\nu }-1} {p}({ \theta })\,d{\theta }. \end{aligned}$$
(2)

By taking the Laplace transform from Eq. (1), one achieves

$$\begin{aligned} {\mathcal{L}} \bigl[\mathscr{D}_{\mathsf{C}} ^{{\nu }}{{p(t) }} \bigr] = s^{\nu }P(s)-\sum_{j=0}^{k-1} s^{({\nu }-j-1)} p^{(j)}(0). \end{aligned}$$
(3)

Definition 2

The Caputo–Fabrizio (CF) fractional operators of order ν are respectively defined as [51]

$$\begin{aligned}& \mathscr{D}_{\mathsf{CF}} ^{{\nu }}{{p(t) }} = \frac{ \mathsf{M} ({\nu })}{n-{\nu }} \int _{0}^{t} \frac{d^{n}}{d t^{n}} p({\theta })exp \biggl[- \frac{{\nu }}{n-{\nu }}(t-{\theta }) \biggr]\,d{\theta },\quad n-1< {\nu } \le n, \end{aligned}$$
(4)
$$\begin{aligned}& \mathscr{I}_{\mathsf{CF}} ^{{\nu }}{{p(t) }} = \frac{2(1-{\nu }) }{(2-{\nu }) \mathsf{M}({\nu })}p(t)+ \frac{2{\nu } }{(2-{\nu }) \mathsf{M}({\nu })} \int _{0}^{t} p({\theta })\,d{ \theta }, \end{aligned}$$
(5)

where \(\mathsf{M}({\nu })=\frac{2}{2-{\nu }}\).

By taking the Laplace transform from Eq. (4), the following result is obtained:

$$ \mathscr{L} \bigl[ \mathscr{D}_{\mathsf{CF}} ^{{\nu }}{{p(t) }} \bigr] = \frac{s^{n+1}P(s) -s^{n}p(0)-s^{k-1}p'(0)-\cdots -p^{(n)}(0)}{s+{\nu }(1-s)}. $$
(6)

Definition 3

Taking advantage of the concept, the Mittag-Leffler function \(\mathsf{ML}_{\nu } ({\theta })\) [52]

$$ \mathsf{ML} _{\nu } ({\theta })=\sum _{j=0}^{\infty } \frac{{\theta }^{j}}{\Gamma (1+{\nu } j)}. $$
(7)

The Atangana–Baleanu (AB) fractional derivative and integral of order ν are respectively characterized as [50]

$$\begin{aligned}& \mathscr{D}_{\mathsf{AB}} ^{{\nu }}{{p(t) }} = \frac{{\mathsf{\beta }({\nu })}}{{1 - {\nu }}} \int _{0}^{t} \textbf{ML} \biggl[ {- \frac{{\nu }}{1 - {\nu }}(t - {\theta }) } \biggr]\dot{p}({ \theta })\, d{\theta },\quad { \nu } \in (0,1), \end{aligned}$$
(8)
$$\begin{aligned}& \mathscr{I}_{\mathsf{AB}} ^{{\nu }} p(t) = \frac{1-{\nu }}{\mathsf{\beta }({\nu })}p(t)+ \frac{{\nu }}{\mathsf{\beta }({\nu })\Gamma ({\nu })} \int _{t_{0}}^{t} (t-{ \theta })^{{\nu }-1} p({ \theta })\,d{\theta }, \end{aligned}$$
(9)

where \(\mathsf{\beta }({\nu })=1-{\nu }+\frac{{\nu }}{\Gamma ({\nu })}\).

The following are some of the basic and widely used features of AB fractional operators:

  • Both AB derivative and integral operators have linear properties as follows:

    $$\begin{aligned}& \mathscr{D}_{\mathsf{AB}} ^{{\nu }}{{\bigl(\epsilon _{1}p_{1}(t)+ \epsilon _{2}p_{2}(t)\bigr)}}= \epsilon _{1} \mathscr{D}_{\mathsf{AB}} ^{{\nu }}{{p_{1}(t) }}+ \epsilon _{2} \mathscr{D}_{\mathsf{AB}} ^{{\nu }}{{p_{2}(t) }}, \end{aligned}$$
    (10)
    $$\begin{aligned}& \mathscr{I}_{\mathsf{AB}} ^{{\nu }}{{\bigl(\epsilon _{1}p_{1}(t)+ \epsilon _{2}p_{2}(t)\bigr)}}= \epsilon _{1} \mathscr{I}_{\mathsf{AB}} ^{{\nu }}{{p_{1}(t) }}+ \epsilon _{2} \mathscr{I}_{\mathsf{AB}} ^{{\nu }}{{p_{2}(t) }}. \end{aligned}$$
    (11)

  • The fractional AB derivative operator satisfies the Lipschitz condition

    $$ \bigl\Vert \mathscr{D}_{\mathsf{AB}} ^{{\nu }}{{p_{1}(t) }}- \mathscr{D}_{\mathsf{AB}} ^{{\nu }}{{p_{2}(t) }} \bigr\Vert \leq \mathsf{N} \bigl\Vert p_{1}(t)-p_{2}(t) \bigr\Vert , $$
    (12)

    where \(\Vert p(t) \Vert =\max_{ 0\leq t \leq t_{f}} \vert p(t) \vert \).

  • For a given function \(p(t)\) and the corresponding norm \(\Vert p(t) \Vert =\max_{t_{0}\leq t \leq t_{f}} \vert p(t) \vert \), one achieves

    $$ \bigl\Vert \mathscr{D}_{\mathsf{AB}} ^{{\nu }}{{p(t) }} \bigr\Vert \leq \frac{\mathsf{\beta }({\nu })}{1-{\nu }} \bigl\Vert p(t) \bigr\Vert . $$
    (13)

  • The combination of the integral and derivative of AB-type operators yields

    $$\begin{aligned} {\mathscr{I}}_{\mathsf{AB}} ^{ {\nu }} \bigl( \mathscr{D}_{\mathsf{AB}} ^{{ \nu }}{{p(t) }} \bigr) =p(t)-p(0). \end{aligned}$$
    (14)

  • Utilizing the Laplace transform on fractional definition stated in Eq. (8) yields

    $$ \mathscr{L} \bigl[ \mathscr{D}_{\mathsf{AB}} ^{{\nu }}{{p(t) }} \bigr] = \frac{\mathsf{\beta }({\nu })}{1-{\nu }} \frac{s P(s)-s^{{\nu }-1}p(0)}{s +\frac{{\nu }}{1-{\nu }}}. $$
    (15)

3 Structural expression of the system

To describe the model, let us assume a three-species food-web system of \(\mathbb{X}_{1}(t)\), \(\mathbb{X}_{2}(t)\), and \(\mathbb{X}_{3}(t)\) species. In this model, \(\mathbb{X}_{1}(t)\) and \(\mathbb{X}_{2}(t)\) are the two competing species bearing intrinsic growth rates. Also, it is assumed that their intrinsic growth rates and carrying capacities are \(r_{i}\) and \(K_{i}\) (\(i = 1, 2\)), respectively. Another symbol used in this model is \(\mathbb{X}_{3}(t)\) which preserves the predating over \(\mathbb{X}_{2}(t)\) with Holling type II response. The species \(\mathbb{X}_{1}(t)\) is assumed to benefit from the presence of \(\mathbb{X}_{3}(t)\), as such \(\mathbb{X}_{1}(t)\) is commensal of \(\mathbb{X}_{3}(t)\). The proportion p of species \(\mathbb{X}_{2}(t)\) is refuged from predation. The coefficients \(\alpha _{j}\) (\(i, j = 1, 2\); \(i \neq j\)) stand for the inter-species competition coefficients. Finally, the commensal coefficient is denoted by \(\delta _{13}\).

The following nonlinear differential equation system can be defined to express the interactions and features of this described ecological model:

$$\begin{aligned} \begin{gathered} \frac{d \mathbb{X}_{1}(t)}{d T} =r_{1} \mathbb{X}_{1}(t) \biggl[1- \frac{\mathbb{X}_{1}(t)}{K_{1}}- \frac{\alpha _{12}\mathbb{X}_{2}(t)}{K_{1}} \biggr]+\delta _{13} \mathbb{X}_{1}(t)\mathbb{X}_{3}(t), \\ \frac{d \mathbb{X}_{2}(t)}{d T} = r_{2}\mathbb{X}_{2}(t) \biggl[1- \frac{\mathbb{X}_{2}(t)}{K_{2}}- \frac{\alpha _{21}\mathbb{X}_{1}(t)}{K_{2}} \biggr]- \frac{a(1-p)\mathbb{X}_{2}(t)\mathbb{X}_{3}(t)}{b+(1-p)\mathbb{X}_{2}(t)}, \\ \frac{d\mathbb{X}_{3}(t)}{d T} =\mathbb{X}_{3}(t) \biggl[-e+ \frac{a c (1-p)\mathbb{X}_{2}(t)}{b+(1-p)\mathbb{X}_{2}(t)} \biggr], \end{gathered} \end{aligned}$$
(16)

subject to the following nonnegative initial conditions \(\mathbb{X}_{1}(0),\mathbb{X}_{2}(0),\mathbb{X}_{3}(0)>0 \).

Now, in order to make system (16) dimensionless, we consider the following new variables in the system:

$$\begin{aligned} \begin{gathered} t=r_{1} T,\qquad \mathscr{X}_{1}(t)=\frac{\mathbb{X}_{1}(t)}{K_{1}},\qquad \mathscr{X}_{2}(t)= \frac{\mathbb{X}_{2}(t)}{K_{2}},\qquad \mathscr{X}_{3}(t)= \frac{a\mathbb{X}_{3}(t)}{r_{1}K_{2}},\\ m_{1}=\frac{b}{K_{2}},\qquad m_{2}= \frac{e}{r_{1}},\qquad m_{3}=\frac{a c }{r_{1}}, \qquad r=\frac{r_{2} }{r_{1}}, \\ \beta _{12}= \frac{\alpha _{12}K_{2} }{K_{1}} ,\qquad \beta _{21}= \frac{\alpha _{21}K_{1} }{K_{2}},\qquad \delta _{12}= \frac{\delta _{13}K_{2} }{a}. \end{gathered} \end{aligned}$$
(17)

Taking variables (17) and system (16) into account, the following dimensionless form of the problem is obtained [53]:

$$\begin{aligned} \begin{gathered} \frac{d \mathscr{X}_{1}(t)}{d t} = \mathscr{X}_{1}(t) \bigl[1- \mathscr{X}_{1}(t) -\beta _{12}\mathscr{X}_{2}(t) \bigr]+\delta \mathscr{X}_{1}(t) \mathscr{X}_{3}(t), \\ \frac{d \mathscr{X}_{2}(t)}{d t} = r \mathscr{X}_{2}(t) \bigl[1- { \mathscr{X}_{2}(t)} - {\beta _{21}\mathscr{X}_{1}(t)} \bigr]- \frac{(1-p)\mathscr{X}_{2}(t)\mathscr{X}_{3}(t)}{m_{1}+(1-p)\mathscr{X}_{2}(t)}, \\ \frac{d\mathscr{X}_{3}(t)}{d t} =\mathscr{X}_{3}(t) \biggl[-m_{2}+ \frac{m_{3} (1-p)\mathscr{X}_{2}(t)}{m_{1}+(1-p)\mathscr{X}_{2}(t)} \biggr]. \end{gathered} \end{aligned}$$
(18)

Replacing the standard derivative with the AB fractional derivative \({} \mathscr{D}_{\mathsf{AB}} ^{ {\nu }}\), we modify model (18) as the following fractional-order version:

$$\begin{aligned} \begin{gathered} \mathscr{D}_{\mathsf{AB}} ^{ {\nu }} \mathscr{X}_{1}(t) = \mathscr{X}_{1}(t) \bigl[1- \mathscr{X}_{1}(t) -\beta _{12} \mathscr{X}_{2}(t) \bigr]+\delta \mathscr{X}_{1}(t) \mathscr{X}_{3}(t), \\ \mathscr{D}_{\mathsf{AB}} ^{ {\nu }} \mathscr{X}_{2}(t) =r \mathscr{X}_{2}(t) \bigl[1- {\mathscr{X}_{2}(t)} - {\beta _{21} \mathscr{X}_{1}(t)} \bigr]- \frac{(1-p)\mathscr{X}_{2}(t)\mathscr{X}_{3}(t)}{m_{1}+(1-p)\mathscr{X}_{2}(t)}, \\ \mathscr{D}_{\mathsf{AB}} ^{ {\nu }}\mathscr{X}_{3}(t) = \mathscr{X}_{3}(t) \biggl[-m_{2}+ \frac{m_{3} (1-p)\mathscr{X}_{2}(t)}{m_{1}+(1-p)\mathscr{X}_{2}(t)} \biggr]. \end{gathered} \end{aligned}$$
(19)

4 Some theoretical features of model (19)

Some theoretical features of the fractional model are explored in this section.

4.1 The equilibrium points of the model

Determining the equilibrium points of a dynamic system is of great importance in better analyzing and understanding the behavior of the model. Finding equilibrium points is equivalent to finding positions for which all derivatives in model (19) are equal to zero. By following this procedure, the corresponding equilibrium points of the system are determined as follows:

  • \(\mathcal{B}_{1}= (0,0,0 )\).

  • \(\mathcal{B}_{2}= (1,0,0 )\).

  • \(\mathcal{B}_{3}= (0,1,0 )\).

  • $$\mathcal{B}_{4}= \biggl( 0,{ \frac{{m_{1}} {m_{2}}}{ (-1+p ) ( {m_{2}}- { m_{3}} ) }} ,-{ \frac{r{m_{3}} {m_{1}} ( ( p-{m_{1}}-1 ) { m_{2}}-{m_{3}} ( -1+p ) ) }{ ( -1+p ) ^{2} ( {m_{2}}-{m_{3}} ) ^{2}}} \biggr). $$
  • $$\mathcal{B}_{5}= \biggl( \frac{\beta _{12}-1}{\beta _{12}\beta _{21}-1}, \frac{\beta _{21}-1}{\beta _{12}\beta _{21}-1},0 \biggr). $$
  • $$\mathcal{B}_{6}= \begin{pmatrix} { \frac{ ( -1+p ) ( -{ \delta } { m_{1}} r+p-1 ) {{ m_{3}^{2}}}-2 ( {p}^{2}+ ( -2+ ( -1/2 r{ \delta }-{ \beta _{12}}/2 ) { m_{1}} ) p+1+1/2 r{ \delta } {{ m_{1}^{2}}}+ ( 1/2 r{ \delta }+{ \beta _{12}}/2 ) { m_{1}} ) { m_{2}} { m_{3}}+{{ m_{2}^{2}}} ( -1+p ) ( -{ \beta _{12}} { m_{1}}+p-1 ) }{ ( -1+p ) ( { m_{2}}-{ m_{3}} ) ( ( { \beta _{21}} { \delta } { m_{1}} r-p+1 ) { m_{3}}+ ( -1+p ) { m_{2}} ) }} \\ { \frac{{ m_{1}} { m_{2}}}{ ( -1+p ) ( { m_{2}}-{ m_{3}} ) }} \\ -{ \frac{ ( ( ( { \beta _{21}}-1 ) p-{ \beta _{12}} { \beta _{21}} {m_{1}}+{m_{1}}-{ \beta _{21}}+1 ) {m_{2}}-{m_{3}} ( { \beta _{21}}-1 ) ( -1+p ) ) r{m_{3}} { m_{1}}}{ ( -1+p ) ( {m_{2}}-{m_{3}} ) ( ( -1+p ) {m_{2}}-{m_{3}} ( -{ \beta _{21}} { \delta } {m_{1}} r+p-1 ) ) }} \end{pmatrix} . $$

In order to analyze the local stability of the possible biological equilibrium points, the Jacobian matrix corresponding to model (19) at the equilibrium point \(\mathcal{B}_{i}=({\mathscr{X}_{1,i}^{*}},{\mathscr{X}_{2,i}^{*}},{ \mathscr{X}_{3,i}^{*}})\) is calculated as follows:

$$ \begin{aligned}[b] &J\bigl({\mathscr{X}_{1}^{*}},{ \mathscr{X}_{2}^{*}},{\mathscr{X}_{3}^{*}} \bigr)\\ &\quad = \begin{pmatrix} -\beta _{12} \mathscr{X}_{2}^{*}+\delta \mathscr{X}_{3}^{*}-2 \mathscr{X}_{1}^{*}+1 &-\mathscr{X}_{1}^{*} \beta _{12}&\delta \mathscr{X}_{1}^{*} \\ -r \beta _{21}\mathscr{X}_{2}^{*} &{ \frac{- ( { \beta _{21}} \mathscr{X}_{1}^{*}+2 \mathscr{X}_{2}^{*}-1 ) ( ( -1+p ) \mathscr{X}_{2}^{*}-{ m_{1}} ) ^{2}r+ \mathscr{X}_{3}^{*}{ m_{1}} ( -1+p ) }{ ( ( -1+p ) \mathscr{X}_{2}^{*}-{ m_{1}} ) ^{2}}} &{ \frac{ ( -1+p ) \mathscr{X}_{2}^{*}}{-\mathscr{X}_{2}^{*}p+{ m_{1}}+\mathscr{X}_{2}^{*}}} \\ 0&-{ \frac{ \mathscr{X}_{3}^{*}{ m_{3}} ( -1+p ) { m_{1}}}{ ( -\mathscr{X}_{2}^{*}p+{ m_{1} }+\mathscr{X}_{2}^{*} ) ^{2}}} &-{ m_{2}}+{ \frac{{ m_{3}} ( 1-p ) \mathscr{X}_{2}^{*}}{{ m_{1}}+ ( 1- p ) \mathscr{X}_{2}^{*}}} \end{pmatrix}. \end{aligned} $$
(20)

Necessary and sufficient conditions for the existence and stability of all the equilibrium points of \(\mathcal{B}_{1}-\mathcal{B}_{6}\) have been discussed in reference [53].

4.2 Existence of the solution

In this section, we aim to prove that the understudied fractional model in the paper possesses at least one solution. To this end, we first apply the β integral operator defined in Eq. (9) on system (19) and get

$$ \begin{aligned} \mathscr{X}_{1}(t)- \mathscr{X}_{1}(0)= \frac{1-{\nu }}{ \mathsf{\beta } ({\nu })} \mathscr{K}_{1}( {\Xi } )+ \frac{{\nu }}{ \mathsf{\beta } ({\nu })\Gamma ({\nu })} \int _{0}^{t} (t-{ \theta })^{{\nu }-1} \mathscr{K}_{1}\bigl( {{\Xi }} ({\theta })\bigr)\,d{\theta }, \\ \mathscr{X}_{2}(t)- \mathscr{X}_{2}(0)= \frac{1-{\nu }}{ \mathsf{\beta } ({\nu })} \mathscr{K}_{2}( {\Xi } ) + \frac{{\nu }}{ \mathsf{\beta } ({\nu })\Gamma ({\nu })} \int _{0}^{t} (t-{ \theta })^{{\nu }-1} \mathscr{K}_{2}\bigl( {{\Xi }} ({\theta })\bigr)\,d{\theta }, \\ \mathscr{X}_{3}(t)- \mathscr{X}_{3}(0)= \frac{1-{\nu }}{ \mathsf{\beta } ({\nu })} \mathscr{K}_{3}( {\Xi } ) + \frac{{\nu }}{ \mathsf{\beta } ({\nu })\Gamma ({\nu })} \int _{0}^{t} (t-{ \theta } )^{{\nu }-1} \mathscr{Q}_{3}\bigl( {{\Xi }} ({\theta })\bigr)\,d{\theta }, \end{aligned} $$
(21)

where \({\Pi }(t)= [\mathscr{X}_{1}(t), \mathscr{X}_{2}(t), \mathscr{X}_{3}(t) ] \). Now, let us define the operator \({\Xi } \textbf{(t)} = [\mathscr{Q}_{1}({\Pi }\textbf{(t)}), \mathscr{Q}_{2}({\Pi }\textbf{(t)}), \mathscr{Q}_{3}({\Pi }\textbf{(t)}) ] \), where

$$\begin{aligned} \begin{gathered} \mathscr{Q}_{1}\bigl({\Pi }(t) \bigr)= s \mathscr{X}_{1}(t) \bigl[1- \mathscr{X}_{1}(t) -\beta _{12}\mathscr{X}_{2}(t) \bigr]+\delta \mathscr{X}_{1}(t) \mathscr{X}_{3}(t), \\ \mathscr{Q}_{2}\bigl({\Pi }(t)\bigr)= r \mathscr{X}_{2}(t) \bigl[1- { \mathscr{X}_{2}(t)} - {\beta _{21} \mathscr{X}_{1}(t)} \bigr]- \frac{(1-p)\mathscr{X}_{2}(t)\mathscr{X}_{3}(t)}{m_{1}+(1-p)\mathscr{X}_{2}(t)}, \\ \mathscr{Q}_{3}\bigl( {\Pi }(t)\bigr)= \mathscr{X}_{3}(t) \biggl[-m_{2}+ \frac{m_{3} (1-p)\mathscr{X}_{2}(t)}{m_{1}+(1-p)\mathscr{X}_{2}(t)} \biggr]. \end{gathered} \end{aligned}$$
(22)

Moreover, \({{\Xi }}_{0}= [\mathscr{X}_{1}(0), \mathscr{X}_{2}(0), \mathscr{X}_{3}(0) ] \) is assumed. Using the above notations, Eq. (21) is reformulated as follows:

$$ {{\Xi (t)}}- {{\Xi }}(0)=\frac{1-{\nu }}{ \mathsf{\beta } ({\nu })} \mathscr{\textbf{K}} \bigl( {\Xi }\textbf{(t)}\bigr)+ \frac{{\nu }}{ \mathsf{\beta } ({\nu })\Gamma ({\nu })} \int _{0}^{t} (t-{ \theta })^{{\nu }-1} \mathscr{\textbf{K}}\bigl( {\Xi } {({\theta })}\bigr)\,d{\theta }. $$
(23)

Now, inspired by (23) and starting from \({{\Xi }}_{0}(t)= {{\Xi }}(0)\), we define the following iterative scheme:

$$ {{\Xi }}_{n}(t)- {{\Xi }}(0)=\frac{1-{\nu }}{ \mathsf{\beta } ({\nu })} \mathscr{\textbf{K}}\bigl( {{\Xi }}_{n-1}(t)\bigr)+ \frac{{\nu }}{ \mathsf{\beta } ({\nu })\Gamma ({\nu })} \int _{0}^{t} (t-{ \theta })^{{\nu }-1} \mathscr{\textbf{K}}\bigl( {{\Xi }}_{n-1} {({\theta })}\bigr)\,d{ \theta }. $$
(24)

Subtracting two consecutive terms gives

$$ \begin{aligned}[b] {{\Xi }}_{n}(t)- {{\Xi }}_{n-1}(t)&= \frac{1-{\nu }}{ \mathsf{\beta } ({\nu })} \bigl[ \mathscr{\textbf{K}}\bigl( {{ \Xi }}_{n-1}(t) \bigr)- \mathscr{\textbf{K}}\bigl( {{\Xi }}_{n-2}(t)\bigr) \bigr]\\ &\quad {}+ \frac{{\nu }}{ \mathsf{\beta } ({\nu })\Gamma ({\nu })} \int _{0}^{t} (t-{ \theta })^{{\nu }-1} \bigl[ \mathscr{\textbf{K}}\bigl( {{\Xi }}_{n-1} {({ \theta })}\bigr)-\mathscr{ \textbf{K}}\bigl( {{\Xi }}_{n-2} {({\theta })}\bigr) \bigr]\,d{ \theta }. \end{aligned} $$
(25)

Now, let us define \(\varsigma _{n}(t)={{\Xi }}_{n}(t)- {{\Xi }}_{n-1}(t)\). So, one gets

$$ {{\Xi }}_{n}(t)=\sum_{i=0}^{n} \varsigma _{i}(t). $$
(26)

As a consequence, one obtains

$$\begin{aligned}& \bigl\Vert \varsigma _{n}(t) \bigr\Vert = \bigl\Vert {{\Xi }}_{n}(t)- {{\Xi }}_{n-1}(t) \bigr\Vert , \\& \begin{aligned} \bigl\Vert \varsigma _{n}(t) \bigr\Vert &= \biggl\Vert \frac{1-{\nu }}{ \mathsf{\beta } ({\nu })} \bigl[ \mathscr{\textbf{K}}\bigl( {{ \Xi }}_{n-1}(t) \bigr)- \mathscr{\textbf{K}}\bigl( {{\Xi }}_{n-2}(t)\bigr) \bigr]\\ &\quad {}+ \frac{{\nu }}{ \mathsf{\beta } ({\nu })\Gamma ({\nu })} \int _{0}^{t} (t-{ \theta })^{{\nu }-1} \bigl[ \mathscr{\textbf{K}}\bigl( {{\Xi }}_{n-1} {({ \theta })}\bigr)-\mathscr{ \textbf{K}}\bigl( {{\Xi }}_{n-2} {({\theta })}\bigr) \bigr]\,d{ \theta } \biggr\Vert . \end{aligned} \end{aligned}$$

Hence

$$\begin{aligned} \bigl\Vert \varsigma _{n}(t) \bigr\Vert \leq& \frac{1-{\nu }}{ \mathsf{\beta } ({\nu })}\biggl\Vert \mathscr{\textbf{K}}\bigl( {{ \Xi }}_{n-1}(t) \bigr)- \mathscr{\textbf{K}}\bigl( {{\Xi }}_{n-2}(t)\bigr) \biggr\Vert \\ &{}+ \frac{{\nu }}{ \mathsf{\beta } ({\nu })\Gamma ({\nu })} \int _{0}^{t} (t-{ \theta })^{{\nu }-1} \biggl\Vert \mathscr{\textbf{K}}\bigl( {{\Xi }}_{n-1} {({ \theta })}\bigr)- \mathscr{\textbf{K}}\bigl( {{\Xi }}_{n-2} {({\theta })}\bigr)\biggr\Vert \, d{ \theta } . \end{aligned}$$

Assuming that operator K satisfies the Lipschitz condition, it reads

$$\begin{aligned} \bigl\Vert \varsigma _{n}(t) \bigr\Vert \leq \frac{1-{\nu }}{ \mathsf{\beta } ({\nu })}\textbf{L} \bigl\Vert {{\Xi }}_{n-1}(t)- {{\Xi }}_{n-2}(t) \bigr\Vert + \frac{{\nu }\textbf{L}}{ \mathsf{\beta } ({\nu })\Gamma ({\nu })} \int _{0}^{t} (t-{\theta })^{{\nu }-1} \bigl\Vert {{\Xi }}_{n-1}(t)- {{\Xi }}_{n-2}(t) \bigr\Vert \, d{ \theta } . \end{aligned}$$

Finally, we have the following inequality:

$$\begin{aligned} \bigl\Vert \varsigma _{n}(t) \bigr\Vert \leq \frac{1-{\nu }}{ \mathsf{\beta } ({\nu })}\textbf{L} \bigl\Vert \varsigma _{n-1}(t) \bigr\Vert +\frac{{\nu }\textbf{L}}{ \mathsf{\beta } ({\nu })\Gamma ({\nu })} \int _{0}^{t} (t-{\theta })^{{\nu }-1} \bigl\Vert \varsigma _{n-1}(t) \bigr\Vert \, d{ \theta } . \end{aligned}$$

Now, replacing \(\Vert \varsigma _{n-1}(t)\Vert \) by its value, we get

$$\begin{aligned} \bigl\Vert \varsigma _{n}(t) \bigr\Vert \leq \biggl( \frac{1-{\nu }}{ \mathsf{\beta } ({\nu })}\textbf{L}+ \frac{{\nu }\textbf{L}t^{{\nu }}}{ \mathsf{\beta } ({\nu })\Gamma ({\nu }+1)} \biggr)^{2} \bigl\Vert \varsigma _{n-2}(t) \bigr\Vert . \end{aligned}$$

Also, we have

$$\begin{aligned} \bigl\Vert \varsigma _{n}(t) \bigr\Vert \leq \biggl( \frac{1-{\nu }}{ \mathsf{\beta } ({\nu })}\textbf{L}+ \frac{{\nu }\textbf{L}t^{{\nu }}}{ \mathsf{\beta } ({\nu })\Gamma ({\nu }+1)} \biggr)^{3} \bigl\Vert \varsigma _{n-3}(t) \bigr\Vert . \end{aligned}$$

And finally the following result is derived:

$$\begin{aligned} \bigl\Vert \varsigma _{n}(t) \bigr\Vert \leq & \biggl( \frac{1-{\nu }}{ \mathsf{\beta } ({\nu })}\textbf{L}+ \frac{{\nu }\textbf{L}t^{{\nu }}}{ \mathsf{\beta } ({\nu })\Gamma ({\nu }+1)} \biggr)^{n} \bigl\Vert \varsigma _{0}(t) \bigr\Vert , \\ \leq & \biggl(\frac{1-{\nu }}{ \mathsf{\beta } ({\nu })}+ \frac{{\nu } t^{{\nu }}}{ \mathsf{\beta } ({\nu })\Gamma ({\nu }+1)} \biggr)^{n} \textbf{L}^{n} \max_{t\in [0,T]}{{\Xi }}(0) (t). \end{aligned}$$
(27)

By choosing

$$\begin{aligned} {{\Xi }} (t)=\sum_{i=0}^{n} \varsigma _{i}(t), \end{aligned}$$
(28)

the general structure for \({{\Xi }} (t)\) is suggested:

$$\begin{aligned} {{\Xi }} (t)={{\Xi }}_{n} (t)+\mu _{n} (t), \end{aligned}$$
(29)

where \(\mu _{n} (t)\rightarrow 0\) when \(n (t)\rightarrow \infty \). Thus,

$$\begin{aligned} {{\Xi }} (t)- {{\Xi }_{n}}(t)=\frac{1-{\nu }}{ \mathsf{\beta } ({\nu })} \mathscr{\textbf{K}} \bigl({{\Xi }} (t)- \mu _{n}(t)\bigr)+ \frac{{\nu }}{ \mathsf{\beta } ({\nu })\Gamma ({\nu })} \int _{0}^{t} (t-{ \theta })^{{\nu }-1} \mathscr{\textbf{K}}\bigl( {{\Xi }} ({\theta })- { {\mu }}_{n } {({ \theta })}\bigr)\,d{\theta }. \end{aligned}$$
(30)

Now, we can write

$$\begin{aligned}& {{\Xi }} (t)- {{\Xi }}(0)- \frac{1-{\nu }}{ \mathsf{\beta } ({\nu })} \mathscr{\textbf{K}}\bigl({{\Xi }} (t)- \mu _{n}(t)\bigr)-\frac{{\nu }}{ \mathsf{\beta } ({\nu })\Gamma ({\nu })} \int _{0}^{t} (t-{\theta })^{{\nu }-1} \mathscr{ \textbf{K}}\bigl( {{\Xi }} ({\theta })- { { \mu }}_{n } {({\theta })} \bigr)\,d{\theta } \\& \quad = \mu _{n} (t)+\frac{1-{\nu }}{ \mathsf{\beta } ({\nu })} \bigl[ \mathscr{\textbf{K}} \bigl({{\Xi }} (t)- \mu _{n}(t)\bigr)-\mathscr{\textbf{K}}\bigl({{ \Xi }} (t) \bigr) \bigr]\\& \qquad {}- \frac{{\nu }}{ \mathsf{\beta } ({\nu })\Gamma ({\nu })} \int _{0}^{t} (t-{ \theta })^{{\nu }-1} \bigl[ \mathscr{\textbf{K}}\bigl( {{\Xi }} ({\theta })- { { \mu }}_{n } {({ \theta })}\bigr)-\mathscr{\textbf{K}}\bigl( {{\Xi }} ({\theta }) \bigr) \bigr]\,d{ \theta }. \end{aligned}$$

Utilizing the norm on both sides of the above equation, we obtain

$$\begin{aligned}& \biggl\Vert {{\Xi }} (t)- {{\Xi }}(0) (t)- \frac{1-{\nu }}{ \mathsf{\beta } ({\nu })} \mathscr{\textbf{K}} \bigl({{\Xi }} (t)\bigr)+ \frac{{\nu }}{ \mathsf{\beta } ({\nu })\Gamma ({\nu })} \int _{0}^{t} (t-{ \theta })^{{\nu }-1} \mathscr{\textbf{K}}( {{\Xi }} ({\theta }) \,d{\theta } \biggr\Vert \\& \quad \leq \bigl\Vert \mu _{n} (t) \bigr\Vert + \frac{1-{\nu }}{ \mathsf{\beta } ({\nu })} \bigg\Vert \mathscr{\textbf{K}}\bigl( {{ \Xi }} ({\theta })- { {\mu }}_{n } {({\theta })}\bigr)-\textbf{K}\bigl( {{\Xi }} ({ \theta }) \bigr) \bigg\Vert \\& \qquad {}+\frac{{\nu }}{ \mathsf{\beta } ({\nu })\Gamma ({\nu })} \int _{0}^{t} (t-{ \theta })^{{\nu }-1} \bigg\Vert \mathscr{\textbf{K}}\bigl( {{\Xi }} ({\theta })- { { \mu }}_{n } {({\theta })}\bigr)-\textbf{K}\bigl( {{\Xi }} ({\theta }) \bigr)\bigg\Vert \,d{ \theta } \\& \quad \leq \bigl\Vert \mu _{n} (t) \bigr\Vert + \frac{1-{\nu }}{ \mathsf{\beta } ({\nu })} \textbf{L} \bigl\Vert \mu _{n-1} (t) \bigr\Vert + \frac{ {\nu } t^{{\nu }}}{ \mathsf{\beta } ({\nu })\Gamma ({\nu }+1)} \textbf{L} \bigl\Vert \mu _{n-1} (t) \bigr\Vert . \end{aligned}$$

If \(n\rightarrow \infty \), then right-hand side vanishes, we reach the following conclusion:

$$ {{\Xi (t)}}- {{\Xi }}(0)=\frac{1-{\nu }}{ \mathsf{\beta } ({\nu })} \mathscr{\textbf{K}} \bigl( {\Xi }\textbf{(t)}\bigr)+ \frac{{\nu }}{ \mathsf{\beta } ({\nu })\Gamma ({\nu })} \int _{0}^{t} (t-{ \theta })^{{\nu }-1} \mathscr{\textbf{K}}\bigl( {\Xi } {({\theta })}\bigr)\,d{\theta }. $$
(31)

This result ensures the existence of solution \({{\Xi }} (t)\) in the system.

4.3 The uniqueness of the solution

Now, let us assume that the system possesses two possible distinct solutions of \({{\Xi }} (t)\) and \({\textbf{Y}} (t)\). Hence we must have

$$\begin{aligned} \bigl\Vert {{\Xi }} (t)-{\textbf{Y}} (t) \bigr\Vert \leq & \frac{1-{\nu }}{ \mathsf{\beta } ({\nu })}\textbf{L} \bigl\Vert {{\Xi }} (t)-{ \textbf{Y}} (t) \bigr\Vert + \frac{{\nu }\textbf{L}t^{{\nu }}}{ \mathsf{\beta } ({\nu })\Gamma ({\nu }+1)} \bigl\Vert {{\Xi }} (t)-{\textbf{Y}} (t) \bigr\Vert \\ \leq & \biggl(\frac{1-{\nu }}{ \mathsf{\beta } ({\nu })} \mathscr{\textbf{L}} + \frac{{\nu }\mathscr{\textbf{L}} t^{{\nu }}}{ \mathsf{\beta } (1+{\nu })\Gamma ({\nu })} \biggr) \bigl\Vert {{\Xi }} (t)-{\textbf{Y}} (t) \bigr\Vert \end{aligned}$$
(32)
$$\begin{aligned} \vdots & \end{aligned}$$
(33)
$$\begin{aligned} \leq & \biggl(\frac{1-{\nu }}{ \mathsf{\beta } ({\nu })} \mathscr{\textbf{L}} + \frac{{\nu }\mathscr{\textbf{L}} t^{{\nu }}}{ \mathsf{\beta } (1+{\nu })\Gamma ({\nu })} \biggr)^{n} \bigl\Vert {{\Xi }} (t)-{\textbf{Y}} (t) \bigr\Vert . \end{aligned}$$
(34)

If \(\frac{1-{\nu }}{ \mathsf{\beta } ({\nu })} \mathscr{\textbf{L}} + \frac{{\nu }\mathscr{\textbf{L}} t^{{\nu }}}{ \mathsf{\beta } (1+{\nu })\Gamma ({\nu })} <1\) holds, then for \(n\rightarrow \infty \), we necessarily have \({{\Xi }} (t)={\textbf{Y}} (t)\). This concludes the desired proof.

5 A numerical method

Designing numerical algorithms to solve problems is always one of our most urgent needs in the face of presenting new concepts in differential calculus of fractional order. In addition, these methods can provide an accurate description of the theoretical behaviors of problems, which is important from different aspects.

In this section, with an idea of what has been studied in article [54], an efficient numerical approximation form for solving fractional problems will be presented to the following form:

$$ \mathscr{D}_{\mathsf{AB}} ^{{\nu }}{\mathscr{X} }(t) = \mathscr{W}\bigl(t, \mathscr{X} (t)\bigr). $$
(35)

By applying the integral operator defined in Eq. (9), the Volterra integral equation is resulted as follows:

$$ \mathscr{X} (t) - \mathscr{X} (t_{0}) = \frac{1-{\nu }}{\mathsf{\beta }({\nu })} \mathscr{W}\bigl(t, \mathscr{X} (t)\bigr)+ \frac{{\nu }}{\mathsf{\beta }({\nu })\Gamma ({\nu })} \int _{t_{0}}^{t}(t-{ \theta })^{{\nu }-1} \mathscr{W}\bigl({\theta }, \mathscr{X} ({\theta })\bigr)\,d{ \theta }. $$
(36)

Setting \(t=t_{n}=t_{0}+n\Delta t\) in (36) yields

$$ \mathscr{X} (t_{n}) = \mathscr{X} (t_{0})+ \frac{1-{\nu }}{\mathsf{\beta }({\nu })} \mathscr{W}\bigl(t_{n}, \mathscr{X} (t_{n})\bigr)+ \frac{{\nu }}{\mathsf{\beta }({\nu })\Gamma ({\nu })}\sum_{i=0}^{n-1} \int _{t_{i}}^{t_{i+1}} (t_{n}-{\theta })^{{\nu }-1}\mathscr{W}\bigl({\theta }, \mathscr{X} ({\theta })\bigr)\,d{ \theta }. $$
(37)

In this step, it is necessary to approximate the integrand function \(\mathscr{W}({\theta }, \mathscr{X} ({\theta }))\) using linear interpolation to the following form:

$$ \mathscr{W}\bigl({\theta }, \mathscr{X} ({\theta })\bigr)\approx \mathscr{W}(t_{i+1}, \mathscr{X}_{i+1} )+\frac{{\theta }-t_{i+1}}{\Delta t} \bigl( \mathscr{W}(t_{i+1}, \mathscr{X}_{i+1} )- \mathscr{W}(t_{i}, \mathscr{X}_{2,i} ) \bigr) ,\quad {\theta }\in [t_{i},t_{i+1}], $$
(38)

where \(\mathscr{X}_{i}= \mathscr{X} (t_{i})\).

Furthermore, substituting this linear approximation for \(\mathscr{W}({\theta }, \mathscr{X} ({\theta }))\) in integral (37) along with performing symbolic calculations with the help of computational software, the following structure in calculating the approximate solution of the problem is introduced [41, 55–57]:

$$ \begin{aligned} \mathscr{X} _{n}&= \mathscr{X} _{0} + \frac{{\nu }\Delta t }{\mathsf{\beta }({\nu })} \Biggl(\varsigma _{n} \mathscr{W}(t_{0} , \mathscr{X} _{0} )+\sum _{i=1}^{n}\zeta _{n-i} \mathscr{W}(t_{i}, \mathscr{X} _{i} ) \Biggr),\quad n\ge 1, \end{aligned} $$
(39)

where

$$ \begin{aligned} &\varsigma _{n}= \frac{(-1+n)^{{\nu }+1}-n (-1+n-{\nu })}{\Gamma ({\nu }+2)}, \\ &\zeta _{i}=\textstyle\begin{cases} \frac{1}{{\Gamma ({\nu } + 2)}}+\frac{1-{\nu }}{{\nu } \Delta t },& i = 0, \\ \frac{{{{(i- 1)}^{{\nu }+1} } - 2{i^{{\nu }+1}} + {{(i + 1)}^{{\nu }+1}}}}{{\Gamma ({\nu } + 2)}},& i = 1,2, \ldots ,n-1. \end{cases}\displaystyle \end{aligned} $$
(40)

After determining the general structure for the approximate method in (39) and (40), the method can be employed in solving problem (19). Hence the following iterative structures will provide the desired numerical results for the problem:

$$\begin{aligned}& \begin{aligned} \mathscr{X}_{1, n} &= \mathscr{X}_{1,0} + \frac{{\nu }\Delta t }{\mathsf{\beta }({\nu })} \Biggl[\varsigma _{n} \bigl( \mathscr{X}_{1, 0} [1- \mathscr{X}_{1, 0} -\beta _{12} \mathscr{X}_{2, 0} ]+\delta \mathscr{X}_{1, 0} \mathscr{X}_{3, 0} \bigr) \\ &\quad {}+\sum_{i=0}^{n}\zeta _{n-i} \bigl(\mathscr{X}_{1, i} [1- \mathscr{X}_{1,i} -\beta _{12}\mathscr{X}_{2, i} ]+\delta \mathscr{X}_{1, i} \mathscr{X}_{3, i} \bigr) \Biggr], \end{aligned} \\& \begin{aligned} \mathscr{X}_{2, n} &= \mathscr{X}_{2,0} + \frac{{\nu }\Delta t }{\mathsf{\beta }({\nu })} \Biggl[\varsigma _{n} \biggl(r \mathscr{X}_{2,0} [1- { \mathscr{X}_{2,0} } - {\beta _{21} \mathscr{X}_{1,0} } ]- \frac{(1-p)\mathscr{X}_{2,0} \mathscr{X}_{3,0} }{m_{1}+(1-p)\mathscr{X}_{2,0} } \biggr) \\ &\quad {}+\sum_{i=0}^{n}\zeta _{n-i} \biggl(r \mathscr{X}_{2,0} [1- { \mathscr{X}_{2,0} } - {\beta _{21}\mathscr{X}_{1,0} } ]- \frac{(1-p)\mathscr{X}_{2,0} \mathscr{X}_{3,0} }{m_{1}+(1-p)\mathscr{X}_{2,0} } \biggr) \Biggr], \end{aligned} \\& \begin{aligned} \mathscr{X}_{3, n} &= \mathscr{X}_{3, 0} +\frac{{\nu }\Delta t }{\mathsf{\beta }({\nu })} \Biggl[\varsigma _{n} \biggl(\mathscr{X}_{3, 0} \biggl[-m_{2}+ \frac{m_{3} (1-p)\mathscr{X}_{2, 0}}{m_{1}+(1-p)\mathscr{X}_{2, 0}} \biggr] \biggr) \\ &\quad {}+\sum_{i=0}^{n}\zeta _{n-i} \biggl(\mathscr{X}_{3, i} \biggl[-m_{2}+ \frac{m_{3} (1-p)\mathscr{X}_{2, i}}{m_{1}+(1-p)\mathscr{X}_{2, i}} \biggr] \biggr) \Biggr]. \end{aligned} \end{aligned}$$
(41)

By running these iterative schemes, approximate solutions to the main fractional problem of the paper are obtained. By observing the obtained relations, it can be seen that these relations are implicit formulas in determining approximate solutions to the problem. Therefore, it seems necessary to use an additional method in solving these nonlinear algebraic systems during calculations. One of the main suggestions in this regard is to use Newton’s method [54].

6 Simulation results and discussion

The fractional-order system (19) is solved using the proposed iterative scheme outlined in (41). The model has been solved for different values of ν, which are 0.88, 0.878, 0.906, 0.934, 0.962, and 0.99.

Now, let us consider some different groups for the parameters in the model.

Set 1. If we set

$$ \begin{gathered} r = 0.5,\qquad m_{1} = 0.14,\qquad m_{2} = 0.32009,\qquad m_{3} = 0.5, \\ \delta = 0.04,\qquad \beta _{12} = 0.4,\qquad \beta _{21} = 0.1,\qquad p = 0.4 . \end{gathered} $$
(42)

Figures 1 and 2 show the numerical approximate solution of system (19) using the initial condition \((\mathscr{X}_{1}(0),\mathscr{X}_{2} (0),\mathscr{X}_{3} )= (0.75, 0.5 , 0.25 )\). It is seen that the system is locally asymptotically stable around the interior equilibrium point

$$ \mathcal{B}_{6}= (0.8404396992, 0.4151390510, 0.1623829903 ). $$

At this point we have

$$ J(\mathcal{B}_{6}) = \begin{bmatrix} - 0.8404396988&- 0.3361758797& 0.03361758797 \\ - 0.02075695255&- 0.0472630187&- 0.6401800000 \\ 0& 0.04505099138& 0.0\end{bmatrix} .$$
(43)

Also, the eigenvalues of the Jacobian matrix at the interior equilibrium point are

$$ \begin{bmatrix} - 0.01943365870+ 0.1679727718 i \\ - 0.8488354001 \\ - 0.01943365870- 0.1679727718 i\end{bmatrix} , $$

which all have a negative real part. So, the necessary conditions of local stability are verified.

Figure 1
figure 1

Dynamics for system (19) for different parameters in Set 1

Full size image
Figure 2
figure 2

Dynamics for system (19) for different parameters in Set 1

Full size image

Set 2. If we set

$$ \begin{gathered} p =0.33856,\qquad m_{1} = 0.1815,\qquad m_{2} = 0.30678,\qquad m_{3} = 0.5228,\\ \delta = 0.04,\qquad r = 0.5, \qquad \beta _{12} = 1.5,\qquad \beta _{21} = 0.49. \end{gathered} $$
(44)

Figures 3 and 4 show the numerical approximate solution of system (19) using the initial condition \((\mathscr{X}_{1}(0),\mathscr{X}_{2} (0),\mathscr{X}_{3} )= (0.75, 0.5 , 0.25 )\). The existence of oscillatory behavior of the system for this set of parametric values is observed in these figures. It can be easily investigated that none of the equilibrium points can establish the condition of stability. For example, in the case of the internal equilibrium point

$$ \mathcal{B}_{6}= ( 0.3904532309, 0.4097689817, 0.1276676154 ), $$

we have

$$ J(\mathcal{B}_{6}) = \begin{bmatrix} - 0.3904532298&- 0.5856798464& 0.01561812924 \\ - 0.1003934005&- 0.0771977296&- 0.6401800000 \\ 0& 0.03588385326& 0.0\end{bmatrix} .$$
(45)

This matrix has the following eigenvalues:

$$ \begin{bmatrix} 0.02268655650+ 0.1306853321 i \\ - 0.5130240724 \\ 0.02268655650 - 0.1306853321 i\end{bmatrix} . $$

So, the results observed in the relevant figures are quite reasonable. A similar situation is observed for other equilibrium points.

Figure 3
figure 3

Dynamics for system (19) for different parameters in Set 2

Full size image
Figure 4
figure 4

Dynamics for system (19) for different parameters in Set 2

Full size image

Set 3. To investigate the effect of commensalism on the coexistence of three species, let us take

$$ \begin{gathered} p = 0,\qquad m_{1} = 0.4,\qquad m_{2} = 0.32009,\qquad m_{3} = 0.5,\\ \delta = 0,\qquad r = 0.5,\qquad \beta _{12} = 1.5,\qquad \beta _{21} = 0.4. \end{gathered} $$
(46)

Figures 5 and 6 present the numerical simulations of system (19) with the initial condition \((\mathscr{X}_{1}(0),\mathscr{X}_{2} (0),\mathscr{X}_{3} )= (0.75, 0.5 , 0.25 )\). It is seen that the system is locally asymptotically stable around the \(\mathscr{X}_{1}\)-free equilibrium point

$$ \mathcal{B}_{4}= ( 0, 0.7116669446,0.1602651634 ). $$

The Jacobi matrix at this point becomes

$$ J(\mathcal{B}_{4}) = \begin{bmatrix} - 0.067500417&- 0.0&0 \\ - 0.1423333889&- 0.2635409446&- 0.6401799998 \\ 0& 0.02593700002&- 0.0000000001\end{bmatrix} .$$
(47)

In this case, the eigenvalues for the matrix are

$$ \begin{bmatrix} - 0.06750041700 \\ - 0.1042185452 \\ - 0.1593223995\end{bmatrix} , $$

which all have a negative real part. So, the necessary conditions of local stability are verified.

Figure 5
figure 5

Dynamics for system (19) for different parameters in Set 3

Full size image
Figure 6
figure 6

Dynamics for system (19) for different parameters in Set 3

Full size image

Set 4. To investigate the effect of commensalism on the coexistence of three species, we set

$$ \begin{gathered} m_{1} = 0.28455632,\qquad m_{2} = 0.32009,\qquad m_{3} = 0.5,\\ p = 0,\qquad r = 0.5,\qquad \delta = 0,\qquad \beta _{12} = 1.5,\qquad \beta _{21} = 0.4. \end{gathered} $$
(48)

Figures 7 and 8 show the numerical approximate solution of system (19) using the initial condition \((\mathscr{X}_{1}(0),\mathscr{X}_{2} (0),\mathscr{X}_{3} )= (0.75, 0.5 , 0.25 )\). It is seen that the system is locally asymptotically stable around the interior equilibrium point

$$ \mathcal{B}_{6}= ( 0.3134174655, 0.5062733170, 0.1456548826 ). $$

At this point we have

$$ \begin{bmatrix} - 0.3134174652&- 0.4701261982& 0.1567087328 \\ - 0.1012546634&- 0.1352284031&- 0.6401800000 \\ 0& 0.03313579652& 0.0\end{bmatrix} .$$
(49)

Also, the eigenvalues of this matrix are evaluated

$$ \begin{bmatrix} - 0.000002490262368+ 0.1264559730 i \\ - 0.4486408878 \\ - 0.000002490262368- 0.1264559730 i\end{bmatrix} , $$

which all have a negative real part. So, the necessary conditions of local stability are verified.

Figure 7
figure 7

Dynamics for system (19) for different parameters in Set 4

Full size image
Figure 8
figure 8

Dynamics for system (19) for different parameters in Set 4

Full size image

In this part, we seek to study the sensitivity analysis of some parameters in the model. In this case, we fix all the other parameters, and we examine the effects of changing a parameter on the system behavior.

First, consider the parameters in Set 1, along with \({\nu }=0.95\). We are looking at the role of \(m_{1}\) on results. For this purpose, we consider the set of values 0.08, 0.13, 0.18, 0.23, 0.28, and 0.3300. Numerical simulations corresponding to these assumptions are shown in Figs. 9 and 10.

Figure 9
figure 9

Dynamics for system (19) by taking different values of parameter \(m_{1}\)

Full size image
Figure 10
figure 10

Dynamics for system (19) by taking different values of parameter \(m_{1}\)

Full size image

The results show that for smaller values of \(m_{1}\), the system solution shows more oscillating behaviors and the point equilibrium will be unstable. But as the parameter increases, the system solution quickly converges to its equilibrium point.

Further, let us examine the effect of different values for parameter p on the solution. In this case, we take the parameters in Set 3, along with \({\nu }=0.95\). We consider the set of values 0.15, 0.2, 0.25, 0.30, 0.35, and 0.4. Numerical simulations corresponding to these assumptions are shown in Figs. 11 and 12. The results show that, for smaller values of p, the system solution shows more oscillating behaviors and the point equilibrium is unstable. But as the parameter increases, the system solution quickly converges to its corresponding interior equilibrium point.

Figure 11
figure 11

Dynamics for system (19) by taking different values of parameter \(m_{1}\)

Full size image
Figure 12
figure 12

Dynamics for system (19) by taking different values of parameter p

Full size image

7 Conclusion

Employing mathematical modeling using differential equations helps us better understand the behavior of dynamic biological systems. In addition, the use of fractional order differential equations can address some of the shortcomings observed in the modeling of biological systems related to the concept of memory. This has made the importance of using this tool even more obvious for researchers. From a numerical point of view, the use of fast supercomputers with high processing speeds allows us to provide more advanced numerical methods in solving various types of differential equations, including equations of fractional order. In this paper, the AB fractional derivative is employed to study some computational aspects of a three-species prey–predator model in mathematical biology. One of the most prominent features of taking this fractional derivative operator is using a nonsingular kernel in the structure of the operator. Some theoretical requirements for the model, such as the existence and uniqueness of the solutions, are also discussed in this paper. Preservation of basic information in biological systems is one of the most important features necessary for modeling these phenomena. Since the concept of memory in fractional derivatives can provide us with such facilities, the use of this important tool in biological modeling will be of great importance. In addition to the constant coefficients in the model, the fractional-order parameter ν is a useful tool for calibrating the results with real problem data. As we observe in the numerical simulations, the employed fractional operator provides all the expected theoretical aspects of the studied model. From obtained theoretical and numerical investigations one concludes that the inter-species competition coefficients, prey refuge rate, half saturation constant, the death rate of predator species, and conservation rate of prey species have a significant role in the stability of the bio-diversity of an ecological system.