1 Introduction

The aim of this paper is to investigate the dynamic behaviors of the following predator–prey model with cannibalism for predator:

$$ \begin{gathered} \frac{dx}{dt} = x(b-\alpha x-my), \\ \frac{dy}{dt} = y(-\beta +c_{1}+nx)- \frac{cy^{2}}{y+d}, \end{gathered} $$
(1.1)

where \(c_{1}< c\), x and y are the density of the prey and predator at time t, respectively. b and α denote the intrinsic growth rate and intraspecific competition of the prey, respectively; β is the death rate of the predator; m denotes the strength of intraspecific interaction between prey and predator; n is the conversion efficiency of ingested prey into new predators; \(cy^{2}/(y+d)\) denotes the cannibalism of the predator; \(c_{1}\) is the birth rate from the predator cannibalism. All the coefficients are nonnegative constants.

As was pointed out by Berryman [1], the dynamic relationship between predator and prey has long been and will continue to be one of the dominant themes in both ecology and mathematical ecology due to its universal existence and importance. During the last decade, many scholars investigated the dynamic behaviors of the predator–prey species, see [2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40] and the references therein.

The traditional two species Lotka–Volterra predator–prey model takes the form

$$ \begin{gathered} \frac{dx}{dt} = x(b-\alpha x-my), \\ \frac{dy}{dt} = y(-\beta +nx). \end{gathered} $$
(1.2)

For the dynamic behaviors of (1.2), we summarize it as follows [6, 11].

Theorem A

In system (1.2), there are two boundary equilibria \(O(0,0)\), \(E^{1}( \frac{b}{\alpha }, 0) \). \(O(0,0)\) is a saddle and \(E^{1}( \frac{b}{\alpha }, 0)\) is globally asymptotically stable if \(\beta > \frac{bn}{\alpha }\). Assume that \(\beta < \frac{bn}{\alpha }\), the positive equilibrium \(E^{2} ( \frac{\beta }{n}, \frac{bn-\alpha \beta }{mn} )\) exists, which is globally asymptotically stable.

In researching the dynamic behaviors of the predator–prey model, some scholars [2, 10,11,12, 17,18,19,20] considered the impact of the functional response for the predator–prey. For example, Yu [18] studied the global asymptotic stability of a predator–prey model with modified Leslie–Gower and Holling-type II schemes:

$$ \begin{gathered} \frac{dx}{dt} = x \biggl(r_{1}-b_{1}x- \frac{a_{1}y}{x+k_{1}} \biggr), \\ \frac{dy}{dt} = y \biggl(r_{2}- \frac{a_{2}y}{x+k_{2}} \biggr), \end{gathered} $$
(1.3)

where \(x(t)\), \(y(t)\) stand for the population (the density) of the prey and the predator at time t, respectively. Yu [18] provided two sets of sufficient conditions on the global asymptotic stability of a positive equilibrium. After that, Yue [19] considered the dynamics of a modified Leslie–Gower predator–prey model with Holling-type II schemes and a prey refuge:

$$ \begin{gathered} \dot{x} = x \biggl(r_{1}-b_{1}x- \frac{a_{1}(1-m)y}{(1-m)x+k_{1}} \biggr), \\ \dot{y} = y \biggl(r_{2}- \frac{a_{2}y}{(1-m)x+k_{2}} \biggr), \end{gathered} $$
(1.4)

where mx is part of the refuge protecting of the prey, here \(m\in [0, 1)\). Yue [19] found that increasing the amount of refuge can ensure the coexistence and attractivity of the two species more easily.

In recent years, cannibalism as a special phenomenon in nature which often occurs in plankton [22], fishes [23], spiders [24], and social insect populations [26] attracted the attention of many scholars. It is a behavior that consumes the same species and helps to provide food sources. Obviously, cannibalism has a very important effect on the dynamic behaviors of the populations (see [22,23,24,25,26,27,28,29,30,31]).

Gao [25], Kang et al. [26], and Rodriguez-Rodriguez et al. [27] proposed and studied the single species stage-structure model with cannibalism. Kang et al. [26] and Rodriguez-Rodriguez et al. [27] thought cannibalism had a great significance for evolution. Zhang et al. [28] obtained a set of sufficient conditions for the permanence of the nonautonomous predator–prey system with periodic attacking rate. Recently, Zhang et al. [29] proposed the following stage-structure prey–predator model with cannibalism for predator:

$$ \begin{gathered} \dot{x} = -x+y+\varepsilon xy+xz, \\ \dot{y} = b_{1}x-\sigma y-\beta xy, \\ \dot{z} = (b_{2}-x)z, \end{gathered} $$
(1.5)

where \(x(t)\) and \(y(t)\) are the densities of the adult predator and juvenile predator at time t, respectively, \(z(t)\) is the density of the prey at time t. The term \(\beta xy\) reflecting the intraspecific interaction denotes the cannibalization rate of adult predators to juvenile ones, the term \(\varepsilon xy \) is the rate of the adult predators increase due to being better fed through eating juveniles. Zhang et al. [29] obtained that large cannibalization rate can make the positive equilibrium globally stable although its stability would change with the increase of the cannibalism rate.

Generally speaking, scholars [22,23,24,25,26,27,28,29] used the bilinear function \(\beta x y\) to describe the cannibalism phenomenon. Only recently did scholars [30, 31] adopted the idea of the functional response of predator–prey model and proposed the nonlinear cannibalism model.

In 2016, Basheer et al. [30] proposed the prey–predator model with prey non-linear cannibalism as follows:

$$ \begin{gathered} \frac{du}{dt} = u(1+c_{1}-u)- \frac{uv}{u+\alpha v}-c \frac{u^{2}}{u+d}, \\ \frac{dv}{dt} = \delta v\biggl(\beta - \frac{v}{u}\biggr), \end{gathered} $$
(1.6)

where \(c_{1}< c\), u and v represent the densities of prey and predator at time t, respectively. The parameters \(c_{1}\), α, c, d, δ, and β are nonnegative constants. Different from the previous works [24,25,26,27,28,29], Basheer et al. [30] used the Holling II type functional response to describe cannibalism. Here the generic cannibalism term \(C(u)\) is added in the prey equation and is given by

$$ C(u)=c\times u\times \frac{u}{u+d}, $$

where c is the cannibalism rate. This term is obviously more appropriate with the reality of ecology and has a clear gain of energy to the cannibalistic prey. This gain results in an increase in reproduction in the prey, modeled via adding a \(c_{1}u\) term to the prey equation. Obviously, \(c_{1}< c\), as it takes depredation of a number of prey by the cannibal to produce one new offspring. They obtained that prey cannibalism alters the dynamics of the predator–prey model. System (1.6) is stable with no cannibalism, while it is unstable with prey cannibalism under the same conditions. After that, Basheer et al. [31] studied the predator–prey model with cannibalism in both predator and prey population and obtained more detailed results.

As far as system (1.2) is concerned, if the boundary equilibrium point \(E^{1}\) of system (1.2) is globally asymptotically stable, which means that the predator will eventually become extinct and the prey will survive, then how does cannibalism affect the dynamic behaviors of the system? If the positive equilibrium point \(E^{2}\) of system (1.2) is globally asymptotically stable, then how does cannibalism affect the dynamic behaviors of the system? This motivated us to propose and study system (1.1).

The paper is arranged as follows. In the next section, we investigate the existence and local stability of the equilibria of system (1.1). In Sect. 3, we discuss the global stability of the equilibria. Numeric simulations are presented in Sect. 4 to show the feasibility of the main results. We end this paper with a brief discussion.

2 Existence and local stability of equilibria

In this paper, let \((x(t), y(t))\) be a solution of system (1.1) which satisfies the initial value \(x(0)>0\), \(y(0)>0 \), and we are only interested in the dynamics of system (1.1) in the first quadrant

$$ R_{0}^{+}\times R_{0}^{+}= \bigl\{ (x, y)\in R^{2}|x\geq 0,y\geq 0\bigr\} . $$

2.1 The existence of equilibria

The equilibria of system (1.1) are determined by the system

$$ \begin{gathered} x(b-\alpha x-my) = 0, \\ y(-\beta +c_{1}+nx)- \frac{cy^{2}}{y+d} = 0. \end{gathered} $$
(2.1)

The system always admits the boundary equilibria \(E_{0}(0,0)\), \(E_{1}( b/\alpha ,0)\), while for other possible boundary equilibria and positive equilibria, we need to consider the following cases:

(i) If \(x=0\), \(y\neq0\), we may have the other boundary equilibrium \(E_{2}(0,y_{1})\), where \(y_{1}\) is the root of the following equation:

$$ (-\beta +c_{1})- \frac{cy}{y+d} = 0. $$
(2.2)

After simplifying calculation, we can get \(y= \frac{d(c_{1}-\beta )}{\beta +c-c_{1}}\). The boundary equilibrium \(E_{2} (0, \frac{d(c_{1}-\beta )}{\beta +c-c_{1}} )\) exists if \(c_{1}>\beta \).

(ii) If \(x\neq0\), \(y\neq0\), the interior equilibrium \(E^{*}(x^{*},y^{*})\) is determined by the equations as follows:

$$ \begin{gathered} b-\alpha x-my = 0, \\ -\beta +c_{1}+nx- \frac{cy}{y+d} = 0. \end{gathered} $$
(2.3)

From the first equation of (2.3), we have \(y= \frac{b-\alpha x}{m}\). Substituting y into the second equation of (2.3), we can get the equation as follows:

$$ Ax^{2}-Bx+C=0, $$
(2.4)

where \(A=\alpha n\), \(B=\alpha (\beta +c-c_{1})+bn+dmn\), \(C=b(\beta +c-c_{1})+dm(\beta -c_{1})\). Obviously, \(A>0\), \(B>0\). Let Δ denote the discriminant of Eq. (2.4) and express it as follows:

$$ \begin{aligned}[b] \Delta &= B^{2}-4AC \\ &= \bigl(\alpha (\beta +c-c_{1})-bn-dmn \bigr)^{2}+4 \alpha cdmn>0. \end{aligned} $$
(2.5)

From \(y= \frac{b-\alpha x}{m}>0\), we have

$$ 0< x< \frac{b}{\alpha }. $$
(2.6)

Now, we will discuss the root of Eq. (2.4) under the assumption that inequality (2.6) holds.

  1. (a)

    If \(C\leq 0\), Eq. (2.4) has the unique positive root \(x_{1}= \frac{B+\sqrt{B^{2}-4AC}}{2A}\geq \frac{B}{A}> \frac{b}{\alpha }\). Obviously, \(x_{1}\) does not satisfy the condition of (2.6).

  2. (b)

    If \(C>0\), we have \(\beta > c_{1}\) or \(\beta \leq c_{1}< \beta +\frac{bc}{b+dm}\). Then Eq. (2.4) has two positive roots \(x_{2,3}= \frac{B\pm \sqrt{B^{2}-4AC}}{2A}\).

Defining the function \(f(x)=Ax^{2}-Bx+C\), we have

$$ \begin{aligned}[b] f \biggl( \frac{b}{\alpha } \biggr)&= A\cdot \biggl( \frac{b}{\alpha } \biggr)^{2}+B\cdot \frac{b}{\alpha }+C \\ &= \frac{dm}{\alpha } \bigl[\alpha (\beta -c_{1})-bn \bigr]. \end{aligned} $$
(2.7)

(1) If \(\beta \leq c_{1}<\beta + \frac{bc}{b+dm}\), we have \(f( \frac{b}{\alpha })<0\), then system (1.1) has a positive equilibrium \(E_{3}(x_{2}^{*}, y_{2}^{*})\), where \(x_{2}^{*}= \frac{B-\sqrt{B^{2}-4AC}}{2A}\), \(y_{2}^{*}= \frac{b-\alpha x_{2}^{*}}{m}\).

(2) If \(\beta > c_{1}\), we cannot determine the size of \(f( \frac{b}{\alpha })\). So we will discuss the following:

If \(f( \frac{b}{\alpha })<0\), we have

$$ \beta -c_{1}< \frac{bn}{\alpha }, $$
(2.8)

it is similar to case (1).

If \(f( \frac{b}{\alpha })\geq 0\), we have

$$ \beta -c_{1}\geq \frac{bn}{\alpha }. $$
(2.9)

Consider \(x_{3}\leq \frac{b}{\alpha }\), after simplifying calculation, we have

$$ \beta -c_{1}+c+ \frac{dmn}{\alpha }+ \frac{\sqrt{\Delta }}{\alpha }\leq \frac{bn}{\alpha }. $$
(2.10)

Obviously, it contradicts with (2.9). So system (1.1) has no positive equilibrium.

Summarizing the above discussion, we obtain the following theorem.

Theorem 2.1

For all positive parameters, there are two boundary equilibria \(E_{0}(0, 0)\), \(E_{1}( \frac{b}{\alpha }, 0)\). The boundary equilibrium \(E_{2} (0, \frac{d(c_{1}-\beta )}{\beta +c-c_{1}} )\) exists if \(c_{1}>\beta \). In system (1.1), for the positive equilibrium, we have:

  1. (i)

    If \(0<\beta -c_{1}< \frac{bn}{\alpha }\), then system (1.1) has the unique positive equilibrium \(E^{*}(x_{2}^{*}, y_{2}^{*})\), where \(x_{2}^{*}= \frac{B-\sqrt{B^{2}-4AC}}{2A}\), \(y_{2}^{*}= \frac{b-\alpha x_{2}^{*}}{m}\).

  2. (ii)

    If \(\beta \leq c_{1}<\beta + \frac{bc}{b+dm}\), then system (1.1) has the unique positive equilibrium \(E^{*}(x_{2}^{*}, y_{2}^{*})\), where \(x_{2}^{*}= \frac{B-\sqrt{B^{2}-4AC}}{2A}\), \(y_{2}^{*}= \frac{b-\alpha x_{2}^{*}}{m}\).

2.2 The local stability of equilibria

Theorem 2.2

In system (1.1), for the boundary equilibrium \(E_{0}(0,0)\), we have

  1. (1)

    If \(c_{1}<\beta \), then \(E_{0}(0, 0)\) is a saddle;

  2. (2)

    If \(c_{1}=\beta \), then \(E_{0}(0, 0)\) is a saddle node;

  3. (3)

    If \(c_{1}>\beta \), then \(E_{0}(0, 0)\) is an unstable node.

Proof

The Jacobian matrix of system (1.1) is calculated as follows:

$$ J(x,y)= \begin{pmatrix} -2\alpha x-my+b& -mx \\ ny&nx-\beta +c_{1}- \frac{2cy}{y+d}+ \frac{cy^{2}}{(y+d)^{2}} \end{pmatrix}. $$
(2.11)

Then the Jacobian matrix of system (1.1) about the equilibrium \(E_{0}(0,0)\) is

$$ J \bigl(E_{0}(0,0) \bigr)= \begin{pmatrix} b & 0 \\ 0 & -\beta +c_{1} \end{pmatrix}. $$
(2.12)

The eigenvalues of \(J(E_{0})\) are \(\lambda _{1}=b>0\), \(\lambda _{2}=c _{1}-\beta \). Hence, if \(\lambda _{2}=c_{1}-\beta <0\), i.e., \(\beta >c_{1}\), then \(E_{0}(0, 0)\) is a saddle. If \(\lambda _{2}=c_{1}- \beta >0\), i.e., \(\beta < c_{1}\), then we have

$$ \bigl[\operatorname{Tr} J(E_{0}) \bigr]^{2}-4\operatorname{Det}J(E_{0})=(b+ \beta -c_{1})^{2} \geq 0, $$

so \(E_{0}(0,0)\) is an unstable node. If \(\lambda _{2}=c_{1}-\beta =0\), namely \(\beta =c_{1}\), the eigenvalues are now given by \(\lambda _{1}=b>0\), \(\lambda _{2}=0\). Then Theorem 7.1 in Chap. 2 in [32] is used to determine the stability of the equilibrium \(E_{0}\). Let \(d\tau =bdt\), where τ is a new time variable, which makes the system into the following form:

$$ \begin{gathered} \frac{dx}{d\tau } = x- \frac{m}{b}xy- \frac{\alpha }{b}x^{2}, \\ \frac{dy}{d\tau } = \frac{n}{b}xy- \frac{c}{bd}y^{2}+ \frac{c}{bd^{2}}y^{3}+Q_{1}(x, y), \end{gathered} $$
(2.13)

where \(Q_{1}(x,y)\) is a power series in \((x,y)\) with terms \(x^{i}y ^{j}\) satisfying \(i+j\ge 4\).

By the implicit function theorem, there is a unique function \(x=\phi (y)\) in the first quadrant such that \(\phi (0)=0\) near the origin. From \(\frac{dx}{d\tau }=0 \), we get the implicit function \(x=0\), then

$$ \frac{dy}{d\tau }=- \frac{c}{bd}y^{2}+ \frac{c}{bd^{2}}y^{3}+Q_{1}(x, y). $$

According to Theorem 7.1 in Chap. 2 in [32], we have \(m=2\), \(a_{m}= \frac{c}{bd}>0\), so \(E_{0}(0, 0)\) is a saddle node.

The proof of Theorem 2.2 is finished. □

Theorem 2.3

In system (1.1), for the boundary equilibrium \(E_{1}( \frac{b}{\alpha }, 0)\), we have:

  1. (1)

    If \(c_{1}\geq \beta \), then \(E_{1}( \frac{b}{\alpha }, 0)\) is a saddle;

  2. (2)

    If \(c_{1}<\beta \), then:

    1. (i)

      If \(\beta -c_{1}< \frac{bn}{\alpha }\), then \(E_{1}( \frac{b}{\alpha }, 0)\) is a saddle;

    2. (ii)

      If \(\beta -c_{1}> \frac{bn}{\alpha }\), then \(E_{1}( \frac{b}{\alpha }, 0)\) is a stable node;

    3. (iii)

      If \(\beta -c_{1}= \frac{bn}{\alpha }\), then \(E_{1}( \frac{b}{\alpha }, 0)\) is a saddle node.

Proof

The Jacobian matrix of system (1.1) about the equilibrium \(E_{1}( \frac{b}{\alpha }, 0)\) is given by

$$ J\biggl(E_{1}\biggl( \frac{b}{\alpha }, 0\biggr)\biggr)= \begin{pmatrix} -b & \frac{bm}{\alpha } \\ 0 & \frac{bn}{\alpha }-(\beta -c_{1}) \end{pmatrix}. $$
(2.14)

The eigenvalues of \(J(E_{1})\) are \(\lambda _{1}=-b<0\), \(\lambda _{2}= \frac{bn}{\alpha }-(\beta -c_{1})\).

If \(\beta -c_{1}\leq 0\), i.e., \(c_{1}\geq \beta \), then \(\lambda _{2}= \frac{bn}{\alpha }-(\beta -c_{1})>0\), so \(E_{1}( \frac{b}{\alpha }, 0)\) is a saddle.

If \(c_{1}<\beta \), we have \(\lambda _{2}>0\), if \(\frac{bn}{\alpha }>\beta -c_{1}\), then \(E_{1}( \frac{b}{\alpha }, 0)\) is a saddle.

If \(c_{1}<\beta \) and \(\frac{bn}{\alpha }<\beta -c_{1}\), then \(\lambda _{2}<0\), we have

$$ \bigl[\operatorname{Tr} J(E_{1}) \bigr]^{2}-4\operatorname{Det}J(E_{1})= \biggl(-b- \frac{bn-\alpha (\beta -c_{1})}{\alpha } \biggr)^{2}\geq 0, $$

so \(E_{1}( \frac{b}{\alpha }, 0)\) is a stable node.

If \(c_{1}<\beta \) and \(\frac{bn}{\alpha }=\beta -c_{1}\), then \(\lambda _{2}=0\), the eigenvalues are now given by \(\lambda _{1}=-b<0\), \(\lambda _{2}=0\). Then Theorem 7.1 in Chap. 2 in [32] is used to determine the stability of the equilibrium \(E_{1}\). Now we transform the equilibrium \(E_{1}\) to the origin by translation \((X,Y)=(x- \frac{b}{\alpha },y)\) at first, and then expand in power series up to the forth order around the origin, which makes the system into the following form:

$$ \begin{gathered} \frac{dX}{dt} = -bX- \frac{bm}{\alpha }Y- \alpha X^{2}-mXY, \\ \frac{dY}{dt} = nXY- \frac{cY^{2}}{d}+ \frac{cY^{3}}{d}- \frac{cY^{4}}{d}+Q_{2}(X,Y), \end{gathered} $$
(2.15)

where \(Q_{2}(X,Y)\) is a power series in \((X,Y)\) with terms \(X^{i}Y ^{j}\) satisfying \(i+j\ge 5\).

Let \(x=-bX- \frac{bm}{\alpha }Y\), \(y=Y\), \(d\tau =-bdt\), where τ is a new time variable, then we have

$$ \begin{gathered} \frac{dx}{d\tau } = x- \frac{m}{\alpha } \biggl( \frac{mn}{\alpha }+\frac{c}{d}\biggr)y^{2}- \frac{\alpha }{b^{2}}x^{2} \\ \hphantom{\frac{dx}{d\tau } =} {}- \frac{m}{b}\biggl(1+ \frac{n}{\alpha } \biggr)xy+ \frac{mc}{\alpha d^{2}}y^{3}+P_{1}(x, y), \\ \frac{dy}{d\tau } = \biggl( \frac{mn}{b\alpha }+ \frac{c}{bd} \biggr)y^{2}+ \frac{n}{b^{2}}xy- \frac{c}{bd^{2}}y^{3}+Q_{3}(x, y), \end{gathered} $$
(2.16)

where \(P_{1}(x,y)\) and \(Q_{3}(x,y)\) are the power series in \((x,y)\) with terms \(x^{i}y^{j}\) satisfying \(i+j\ge 4\).

By the implicit function theorem, there is a unique function \(x=\phi (y)\) in the first quadrant such that \(\phi (0)=0\) near the origin. From \(\frac{dx}{d\tau }=0 \), we could obtain the implicit function \(x= \frac{m}{\alpha }( \frac{mn}{\alpha }+ \frac{c}{d})y^{2}+P_{2}(x, y)\), then

$$ \frac{dy}{d\tau }=\biggl( \frac{mn}{b\alpha }+ \frac{c}{bd} \biggr)y^{2}+Q_{4}(x, y), $$

where \(P_{2}(x, y)\) and \(Q_{4}(x, y)\) are the power series in \((x, y)\) with terms \(x^{i}y^{j}\) satisfying \(i+j\ge 3\).

According to Theorem 7.1 in Chap. 2 in [32], we have \(m=2\), \(a_{m}= \frac{mn}{b\alpha }+ \frac{c}{bd}>0\), so \(E_{1}( \frac{b}{\alpha }, 0)\) is a saddle node.

The proof of Theorem 2.3 is finished. □

Theorem 2.4

In system (1.1), when the boundary equilibrium \(E_{2} (0, \frac{d(c_{1}-\beta )}{\beta +c-c_{1}} )\) exists, we have

  1. (1)

    If \(\beta < c_{1}<\beta + \frac{b(\beta +c)+\beta dm}{b+dm}\), then \(E_{2} (0, \frac{d(c_{1}-\beta )}{\beta +c-c_{1}} )\) is a saddle;

  2. (2)

    If \(c_{1}>\beta + \frac{b(\beta +c)+\beta dm}{b+dm}\), then \(E_{2} (0, \frac{d(c_{1}-\beta )}{\beta +c-c_{1}} )\) is a stable node;

  3. (3)

    If \(c_{1}=\beta + \frac{b(\beta +c)+\beta dm}{b+dm}\), then \(E_{2} (0, \frac{d(c_{1}-\beta )}{\beta +c-c_{1}} )\) is a saddle node.

Proof

The Jacobian matrix of system (1.1) about the equilibrium \(E_{2} (0, \frac{d(c_{1}-\beta )}{\beta +c-c_{1}} )\) is

$$ J \biggl(E_{2} \biggl(0, \frac{d(c_{1}-\beta )}{\beta +c-c_{1}} \biggr) \biggr)= \begin{pmatrix} \frac{C}{c+\beta -c_{1}} & 0 \\ ny_{2} & - \frac{cdy_{2}}{(y_{2}+d)^{2}} \end{pmatrix}. $$
(2.17)

The eigenvalues of \(J(E_{2})\) are \(\lambda _{1}= \frac{C}{c+\beta -c_{1}}\), \(\lambda _{2}=- \frac{cdy_{2}}{(y+d)^{2}}<0\).

If \(C>0\), i.e., \(\beta < c_{1}<\beta + \frac{b(\beta +c)+\beta dm}{b+dm}\), we have \(\lambda _{1}= \frac{C}{c+\beta -c_{1}}>0\), so \(E_{2} (0, \frac{d(c_{1}-\beta )}{\beta +c-c_{1}} )\) is a saddle.

If \(c_{1}>\beta + \frac{b(\beta +c)+\beta dm}{b+dm}\), then we have

$$ \bigl[\operatorname{Tr} J(E_{2}) \bigr]^{2}-4\operatorname{Det}J(E_{2}) \geq 0, $$

so \(E_{2} (0, \frac{d(c_{1}-\beta )}{\beta +c-c_{1}} )\) is a stable node.

If \(c_{1}=\beta + \frac{b(\beta +c)+\beta dm}{b+dm}\), then \(E_{2} (0, \frac{d(c_{1}-\beta )}{\beta +c-c_{1}} )\) is a saddle node. The proof is similar to Theorem 2.3, we omitted it.

The proof of Theorem 2.4 is finished. □

Theorem 2.5

In system (1.1), when the equilibrium \(E^{*}(x^{*}, y^{*})\) exists, it is locally asymptotically stable.

Proof

The Jacobian matrix of system (1.1) about the equilibrium \(E^{*}(x^{*}, y^{*})\) is

$$ J \bigl(E^{*}\bigl(x^{*}, y^{*}\bigr) \bigr)= \begin{pmatrix} -\alpha x^{*} & -mx^{*} \\ ny^{*} & - \frac{cdy^{*}}{(y^{*}+d)^{2}} \end{pmatrix}. $$
(2.18)

Then we have

$$ \operatorname{Det} J\bigl(E^{*}\bigr)= \frac{cd\alpha x^{*}y^{*}}{(y^{*}+d)^{2}}+mnx^{*}y^{*}>0, $$

and

$$ \operatorname{Tr} J\bigl(E^{*}\bigr)=-\alpha x^{*}- \frac{cdy^{*}}{(y^{*}+d)^{2}}< 0. $$

So \(E^{*}(x^{*}, y^{*})\) is locally asymptotically stable.

The proof of Theorem 2.5 is finished. □

3 Global stability of equilibria

In this section we consider the global asymptotic stability of the equilibria.

Theorem 3.1

Assume that

$$ \beta -c_{1}> \frac{bn}{\alpha } $$
(3.1)

holds, then \(E_{1}(\frac{b}{\alpha }, 0)\) is globally asymptotically stable.

Proof

We will prove Theorem 3.1 by constructing some suitable Lyapunov function.

Let us define a Lyapunov function

$$ V_{1}(x, y)= \frac{n}{m}\biggl(x-\overline{x}-\overline{x} \ln \frac{x}{\overline{x}}\biggr)+y, $$
(3.2)

where \(\overline{x}=\frac{b}{\alpha }\). Then the time derivative of \(V_{1}\) along the trajectories of (1.1) is

$$ \begin{aligned} D^{+}V_{1}(t) &= \frac{n}{m}(x-\overline{x}) \frac{\dot{x}}{\overline{x}}+y(-\beta +c_{1}+nx)- \frac{cy^{2}}{y+d} \\ &=- \frac{\alpha n}{m}(x-\overline{x})^{2}-ny(x-\overline{x})+y(- \beta +c _{1}+nx)- \frac{cy^{2}}{y+d} \\ &< - \frac{\alpha n}{m}(x-\overline{x})^{2}-ny(x-\overline{x})+y \biggl(- \frac{bn}{\alpha }+nx\biggr)- \frac{cy^{2}}{y+d} \\ &=- \frac{\alpha n}{m}(x-\overline{x})^{2}- \frac{cy^{2}}{y+d} \\ &< 0. \end{aligned} $$

Thus, \(V_{1}(x, y)\) satisfies Lyapunov asymptotic stability theorem, and the boundary equilibrium \(E_{1}( \frac{b}{\alpha }, 0)\) of system (1.1) is globally asymptotically stable.

The proof of Theorem 3.1 is finished. □

Theorem 3.2

Assume that

$$ c_{1}>\beta +\frac{b(\beta +c)+\beta dm}{b+dm} $$
(3.3)

holds, then \(E_{2} (0, \frac{d(c_{1}-\beta )}{\beta +c-c_{1}} )\) is globally asymptotically stable.

Proof

We will prove Theorem 3.2 by constructing some suitable Lyapunov function.

Let us define a Lyapunov function

$$ V_{2}(x, y)=x+ \frac{m}{n}\biggl(y-\overline{y}- \overline{y}\ln \frac{y}{\overline{y}}\biggr), $$
(3.4)

where \(\overline{y}= \frac{d(c_{1}-\beta )}{\beta +c-c_{1}}\). Then the time derivative of \(V_{2}\) along the trajectories of (1.1) is

$$\begin{aligned} D^{+}V_{2}(t) &= x(b-\alpha x-my)+mx(y-\overline{y})- \frac{m}{n}\biggl( \frac{cy}{y+d}- \frac{c\overline{y}}{\overline{y}+d}\biggr) (y-\overline{y}) \\ &=-\alpha x^{2}+(b-my)x+mx(y-\overline{y})- \frac{cdm(y-\overline{y})^{2}}{n(y+d)(\overline{y}+d)} \\ &< -\alpha x^{2}- \frac{cdm(y-\overline{y})^{2}}{(y+d)(\overline{y}+d)} \\ &< 0. \end{aligned}$$

Thus, \(V_{2}(x, y)\) satisfies Lyapunov asymptotic stability theorem, and the boundary equilibrium \(E_{2} (0, \frac{d(c_{1}-\beta )}{\beta +c-c_{1}} )\) of system (1.1) is globally asymptotically stable.

The proof of Theorem 3.2 is finished. □

Theorem 3.3

When the equilibrium \(E_{3}(x^{*}, y^{*})\) exists, it is globally asymptotically stable.

Proof

We will prove Theorem 3.3 by constructing some suitable Lyapunov functions.

Let us define a Lyapunov function

$$ V_{3}(x, y)=\biggl(x-x^{*}-x^{*}\ln \frac{x}{x^{*}}\biggr)+ \frac{m}{n}\biggl(y-y^{*}-y^{*} \ln \frac{y}{y^{*}}\biggr). $$
(3.5)

Then the time derivative of \(V_{3}\) along the trajectories of (1.1) is

$$ \begin{aligned} D^{+}V_{3}(t) &= \bigl[- \alpha \bigl(x-x^{*}\bigr)-m\bigl(y-y^{*}\bigr) \bigr] \bigl(x-x^{*} \bigr) \\ &\quad {}+ \frac{m}{n} \biggl[n\bigl(x-x^{*}\bigr)-\biggl( \frac{cy}{y+d}- \frac{cy^{*}}{y^{*}+d}\biggr) \biggr] \bigl(y-y^{*} \bigr) \\ &=-\alpha \bigl(x-x^{*}\bigr)^{2}- \frac{cdm(y-y^{*})^{2}}{n(y+d)(y^{*}+d)} \\ &< 0. \end{aligned} $$

Thus, \(V_{3}(x, y)\) satisfies Lyapunov asymptotic stability theorem, and the positive equilibrium \(E_{3}(x^{*}, y^{*})\) of system (1.1) is globally asymptotically stable when the equilibrium \(E_{3}(x^{*}, y ^{*})\) exists.

The proof of Theorem 3.3 is finished. □

4 Numerical simulations

In this section we consider the dynamics of systems (1.1) and (1.2) under different parameters.

Let \(b=5\), \(\alpha =3\), \(m=0.6\), \(\beta =2.5\), \(n=1.2\), then system (1.2) is given by

$$ \begin{gathered} \frac{dx}{dt} = x(5-3x-0.6y), \\ \frac{dy}{dt} = y(-2.5+1.2x). \end{gathered} $$
(4.1)

We have \(\frac{bn}{\alpha }=2<\beta =2.5\). From Theorem A, system (4.1) has two boundary equilibria \(O(0,0)\), \(E^{1}(1.67, 0)\), and \(E_{0}(0,0)\) is a saddle, \(E^{1}(1.67, 0)\) is globally asymptotically stable (see Fig. 1).

Figure 1
figure 1

Dynamic behaviors of system (4.1)

Now we consider some cannibalism parameters on the basis of (4.1). Let \(c=8\), \(d=15\), then system (1.1) is given by

$$ \begin{gathered} \frac{dx}{dt} = x(5-3x-0.6y), \\ \frac{dy}{dt} = y(-2.5+c_{1}+1.2x)- \frac{8y^{2}}{y+15}. \end{gathered} $$
(4.2)

We consider \(c_{1}\) as variable. System (4.2) always has two boundary equilibria \(E_{0}(0,0)\), \(E_{1}(1.67, 0)\) from Theorem 2.1. If \(c_{1}=0.4\), from Sect. 2.2, we have \(E_{0}(0,0)\) is a saddle and \(E_{1}(1.67, 0)\) is a stable node (see Fig. 2). If \(c_{1}=2\), the positive equilibrium \(E^{*}(1.26, 2.11)\) exists, which is globally asymptotically stable. \(E_{0}(0, 0)\) and \(E_{1}(1.67, 0)\) are saddle (see Fig. 3). If \(c_{1}=2.5\), system (4.2) has a globally asymptotically stable positive equilibrium \(E^{*}(1.1, 2.97)\), \(E_{0}(0,0)\) is a saddle node, \(E_{1}(1.67, 0)\) is a saddle (see Fig. 4). If \(c_{1}=5\), system (4.2) has a boundary equilibrium \(E_{2}(0, 6.9)\), which is a saddle. Then \(E_{0}(0, 0)\) is an unstable node, \(E_{1}(1.67, 0)\) is a saddle, \(E^{*}(0.163, 7.63)\) is globally asymptotically stable (see Fig. 5). If \(c_{1}=7.86\), the positive equilibria of system (4.2) will disappear, and the boundary equilibrium \(E_{1}(0, 30.4)\) is globally asymptotically stable. \(E_{0}(0, 0)\) is an unstable node, \(E_{1}(1.67, 0)\) is a saddle (see Fig. 6).

Figure 2
figure 2

Dynamic behaviors of system (4.2) if \(c_{1}=0.4\)

Figure 3
figure 3

Dynamic behaviors of system (4.2) if \(c1=2\)

Figure 4
figure 4

Dynamic behaviors of system (4.2) if \(c1=2.5\)

Figure 5
figure 5

Dynamic behaviors of system (4.2) if \(c1=5\)

Figure 6
figure 6

Dynamic behaviors of system (4.2) if \(c1=7.86\)

Now let us consider system (1.2), which has a unique positive equilibrium, let \(b=5\), \(\alpha =3\), \(m=0.6\), \(\beta =2.5\), \(n=1.8\), then system (1.2) is

$$ \begin{gathered} \frac{dx}{dt} = x(5-3x-0.6y), \\ \frac{dy}{dt} = y(-2.5+1.8x). \end{gathered} $$
(4.3)

We have \(\frac{bn}{\alpha }=3>\beta =2.5\). From Theorem A, system (4.3) has two boundary equilibria \(O(0,0)\), \(E^{1}(1.67, 0)\) and a unique positive equilibrium \(E^{2}(1.39, 1.36)\), and \(E_{0}(0,0)\) is a saddle, \(E^{1}(1.67, 0)\) is unstable, and \(E^{2}(1.39, 1.36)\) is globally asymptotically stable (see Fig. 7).

Figure 7
figure 7

Dynamic behaviors of system (4.3)

We consider the predator cannibalism based on system (4.3). Let \(c=8\), \(d=15\), then we have

$$ \begin{gathered} \frac{dx}{dt} = x(5-3x-0.6y), \\ \frac{dy}{dt} = y(-2.5+c_{1}+1.8x)- \frac{8y^{2}}{y+15}. \end{gathered} $$
(4.4)

System (4.4) always has two boundary equilibria \(E_{0}(0,0)\), \(E_{1}(1.67, 0)\) from Theorem 2.1. If \(c_{1}=2\), the positive equilibrium \(E^{*}(1.05, 3.12)\) exists, which is globally asymptotically stable. \(E_{0}(0, 0)\) and \(E_{1}(1.67, 0)\) are saddle (see Fig. 8). If \(c_{1}=5\), system (4.4) has a boundary equilibrium \(E_{2}(0, 6.87)\), which is a saddle. Then \(E_{0}(0, 0)\) is an unstable node, \(E_{1}(1.67, 0)\) is a saddle, \(E^{*}(0.139, 7.84)\) is globally asymptotically stable (see Fig. 9). If \(c_{1}=7.86\), the positive equilibrium will disappear for system (4.4), and the boundary equilibrium \(E_{1}(0, 30.6)\) is globally asymptotically stable. \(E_{0}(0, 0)\) is an unstable node, \(E_{1}(1.67, 0)\) is a saddle (see Fig. 10).

Figure 8
figure 8

Dynamic behaviors of system (4.4) if \(c_{1}=2\)

Figure 9
figure 9

Dynamic behaviors of system (4.4) if \(c_{1}=5\)

Figure 10
figure 10

Dynamic behaviors of system (4.4) if \(c_{1}=7.86\)

5 Conclusion

Based on the traditional Lotka–Volterra predator–prey model, we propose and study a predator–prey model with predator cannibalism in this paper. We have investigated the local and global stability of the possible equilibria of the model. Meanwhile, we can find some interesting phenomenon about the dynamic behaviors of system (1.1). If system (1.2) (no cannibalism, i.e., \(c=0\) and \(c_{1}=0\)) has a boundary equilibrium \(E^{1}( \frac{b}{\alpha }, 0)\), which is globally asymptotically stable (see Fig. 1), a suitable cannibalism rate (\((\beta < c_{1}<\beta + \frac{b(\beta +c)+\beta dm}{b+dm} )\)) leads to system (1.1) admitting a unique positive equilibrium, and it is globally asymptotically stable (see Fig. 3, Fig. 4, and Fig. 5). That is to say, cannibalism within a certain range can make the two species persistent. So in this case, cannibalism in a certain range has a positive effect for the coexistence of the prey and the predator. With the increase of \(c_{1}\), the positive equilibrium will disappear and the boundary equilibrium \(E_{2} (0, \frac{d(c_{1}-\beta )}{c+\beta -c_{1}} )\) will appear (see Fig. 6). That is to say, without other sources of food, predator populations can still survive on cannibalism. For example, salamanders only depend on cannibalism to survive in summer.

If system (1.2) has a positive equilibrium \(E^{2}( \frac{\beta }{n}, \frac{bm-\alpha \beta }{mn})\), which is globally asymptotically stable (see Fig. 7), with the increase of \(c_{1}\), the population density of prey decreases while that of predator increases (see Fig. 8 and Fig. 9). When \(c_{1}\) is large enough, prey populations will be driven to extinction. That is to say, predator cannibalism will make prey extinct (see Fig. 10). Predator cannibalism also changes the type of the equilibria (see Fig. 1, Fig. 5, and Fig. 6; Fig. 7, Fig. 9, and Fig. 10).

That is, by introducing the predator cannibalism, the dynamic behaviors of the system become complicated.